Jacamar CI v0.5.0

  • Release: v0.5.0

  • Date: 04/05/2021

Important

Thanks to the great testing and feedback we’ve had we’ve made a large number of administratively focused changes to both improve performance, deployment, and most importantly security. Please update any test installation of Jacamar CI with the latest version.

Admin Changes

  • Overhaul of supplementary group identification and expand testing to verify desired functionality for use with capabilities as well as all other supported downscoping mechanism (!127).

    • This is an incredibly important change as it seeks to ensure use of capabilities is not only supported but in fact a more secure alternative to relying on root. If any remaining permission issues are observed during testing please report immediately.

  • New downscope option downscope = "sudo" added which invokes a crafted sudo su command that is used to subsequently launch a user-owned jacamar application process (!103).

    • As with much of Jacamar this is still under active development, and we do not advise use in production without extensive testing. Please report any issues you may experience when using this new functionality.

    • sudo -E su <username> -m -s /bin/bash --pty -c ...

    • Through the use of the -E and -m arguments supplied to the associated application we preserve a targeted custom environment. This means that similar to the setuid option we rely on Jacamar to specify safe, downscope environment variables.

  • Support for libseccomp added with configurable blocked system calls, and default established for jacamar-auth (!119).

    • This changes means that going forward the Jacamar CI RPM, as well as build, will required libseccomp.

    • [auth.seccomp]
    disabled = false
    block_calls = ["sethostname", "sendfile"]

    • When downscope = "setuid" is configured we will now block any ioctl system calls by default. Though we can establish future default behaviors if is important to note we will attempt to do so in a limited fashion and only when unexpected behaviors may be observed across a range of machines deployments/configurations.

    • Any “blocked” command will only produce a basic warning message that will appear in the CI job log.

  • Corrected SIGTERM monitoring and command context handling that led to high CPU usage across a number of deployments (!117).

  • Additional testing and error handling for Slurm (!105).

    • We now test against version 20.11.3 as part of CI/CD processes.

    • Output from sacct is better processed to account for a range of potential user environments.

  • Data directory configurations now support environment variables (data_dir = "/$GPFS/.ci") resolved by the user (!118).

    • The variable resolution will occur in the downscoped user jacamar application, as such you may need to specify a source_script to ensure the desired variable is present. We do not wish to default to a login shell when the application is launched as it remains unknown what potential CI users have configured in their login environment which may lead to hard to troubleshoot job failures.

  • Improved Git ASKPASS credential creation and remove helper from local .git/config into $CI_PROJECT_DIR/../.credential folder (!124).

    • This solution conforms to the changes documented in Git commit 2432137

    • If problems remain with Git due to user/system environments we suggest testing more heavy handed approaches (e.g., removing configurations) via the runners pre_clone_script, from there we can work towards a more universal solution.

  • RunAs validation support override of the target gitlab_account used in conjunction with the CI Token Broker (!122).

  • Fully qualify jacamar application when generating downscope command (!101).

    • The path is identified by the privileged users of jacamar-auth only if the jacamar_path is not configured.

  • Improve support for downscope = "none" to allow single user deployment to leverage authorization level features (!110).

    • There is no established user focused guide on this process at this time, we want to ensure the desired workflow is more clear before moving forward.

  • Identify Bash shell in command creation, with optional configuration support (!107).

  • Clarified error message associated with data_dir creation and expanded related testing (!123).

  • Updated Slurm job script with bash --login prior to execution (!131).

    • Since Slurm jobs are executed on potentially remote compute environment, the login environment jacamar provides is not sufficient for accounting for all potential cases, even the source_script is not guaranteed to help in these use cases. This change will align with previously established and testing Slurm executor functionality mistakenly left out of Jacamar CI.

  • Create Slurm output files in advance of job submission as opposed to allowing sbatch to create its own (!132).

    • The job output will be created in a CI job identifiable script folder with the file name slurm-ci-<jobID>.out and provided to the constructed sbatch command via the --output flag.

    • This will align similar workflows between Cobalt and Slurm executors.

Bug & Development Fixes

  • Allow hyphens in username rules (!106).

  • Updated all Pavilion2 tests to leverage new functionality and provide more fine-tuned testing both local and in CI pipelines (!126, !130, !129).

    • All previous make commands related to Pavilion testing have been updated:

      • make pav-docker-auth: Testing jacamar-auth functionality with root.

      • make pav-docker-build: Build a runner binary using the appropriate CentOS 7 environment in the binaries directory.

      • make pav-docker-capabilities: Testing jacamar-auth functionality with a cap_setuid and capt_setgid provided.

      • make pav-docker-jacamar: Testing jacamar as well as limited none downscoping in userspace.

      • make pav-docker-slurm: Slurm test series focused on using the jacamar application as a Slurm executor.

    • The container images referenced are all declared as environment variables in the Makefile.

  • Support for Go 1.16 (!113).

  • Improve bytes buffer for command execution (!104).

  • Removed support + usage of jacamar-plugins (!115).

    • Plugins where previously an undocumented feature that had been included to support testing.

  • Updated OLCF testing pipeline (!120).

  • Removed outdated ALCF testing pipeline and supporting files (!112).

  • Tests added to ensure source_script functionality (!121).

  • Test to verify credentials removal (!111).

  • Check broker token response for expected patterns (!109).

  • Remove translation test files upon completion (!108).