Jacamar CI v0.1.0

Release: v0.1.0
Date: 6/30/2020
Commit: 941dc888

This is the first release of Jacamar, the custom executor driver targeting support for HPC environments. As the release version (0.1.0) would imply we are still very much in active development; however, we’ve reached a point where more facility focused testing/feedback would greatly benefit the effort.

Note

Please avoid deployments of privileged processes relating to Jacamar until you’ve fully vetted the current version in a testing environment.

Due to this being the first release these notes will likely be lacking enough content to provide you a complete understanding of the application. We will be greatly expanding the ECP official documentation over the coming weeks to provide just such information. The goal of supporting those unfamiliar with the application.

General Notes

Jacamar is a completely functional and highly configurable driver for the GitLab custom executor.
- All core features of the forked GitLab runner with ECP enhancements have been migrated to this code base, including all Batch executor functionality. A great number of targeted enhancements and quality improvements have been made as part of this process.
- This includes all developed functionality added as part of past ECP releases.

To assist with any troubleshooting required, clear details will be made available via the job log:

Running with gitlab-runner 13.0.0 (9d1be31e)
  on Custom Executor Test jbHXQozi
Preparing the "custom" executor
  Using Custom executor with driver Jacamar CI 0.1.0...
Preparing environment
  Targeting shell execution
  Running as paulbry UID: 1000 GID: 1000
  Local time: 2020-07-01 16:19:07

With the Cobalt executor it is now possible to summarize errors upon job completion by declaring the COBALT_SUMMARIZE_ERRORS: true CI variables. This can help in cases where the multi-log behaviors with qsub are causing issues distinguishing stderr within the traditional CI job log.
If enabled by the runner administrator it is now possible to specify a CUSTOM_CI_BUILDS_DIR variable in your CI job. This will be used as the root location for the runner generated build directory.
```
ci-job:
  variables:
    CUSTOM_CI_BUILDS_DIR: /test/dir
  script:
    - echo "CI_PROJECT_DIR: ${CI_PROJECT_DIR}"
```
```
CI_PROJECT_DIR: /test/dir/username/builds/runner/0/group/project
```
- It is important to note that uniqueness of the directory structure is still enforced to avoid collisions between jobs.
- You are responsible for ensuring that the directory specified is accessible by any compute resources accessed from the runner.

Admin Notes

Though the custom executor is officially support by GitLab there exists several changes to the runner required to use Jacamar with HPC resources at this time.
- The custom-executor-dev branch can be found on the ECP CI group.
- A RPM has also been posted to the release page.

Though with the release we’ve included compiled binaries, it is recommended however you build your own using the provided make file.

$ make build VERSION=0.1.0
mkdir -p binaries
CGO_ENABLED=1 go build -ldflags "-X gitlab.com/ecp-ci/jacamar-ci/internal/version.version=0.1.0 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.gitCommit=27b3184 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.gitBranch=develop -X gitlab.com/ecp-ci/jacamar-ci/internal/version.goVersion=go1.14.4-linux/amd64 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.buildDate=2020-07-01T15:33:39+0000" -o binaries/jacamar cmd/jacamar/main.go
CGO_ENABLED=1 go build -ldflags "-X gitlab.com/ecp-ci/jacamar-ci/internal/version.version=0.1.0 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.gitCommit=27b3184 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.gitBranch=develop -X gitlab.com/ecp-ci/jacamar-ci/internal/version.goVersion=go1.14.4-linux/amd64 -X gitlab.com/ecp-ci/jacamar-ci/internal/version.buildDate=2020-07-01T15:33:39+0000" -o binaries/jacamar-auth cmd/jacamar-auth/main.go
$ make install PREFIX=/usr/local
install -d /usr/local/bin
install -m 755 binaries/jacamar /usr/local/bin
install -m 755 binaries/jacamar-auth /usr/local/bin

To build your own version ensure that you have installed Go version 1.14+. Cgo will also be leveraged due to requirements of the os/user libraries.
in future release we plan on providing an official RPMs.

Both application (jacamar and jacamar-auth) support command line interaction; however, are currently only designed to be interacted with by the GitLab runner custom executor:

$ jacamar --help
Version: 0.1.0
Git Commit: 27b3184
Git Branch: develop
Go Version: go1.14.4-linux/amd64
Built: 2020-07-01T15:33:39+0000

Usage: jacamar <command> [<args>]

Options:
--help, -h             display this help and exit
--version              display version and exit

Commands:
config                 Execute configuration stage (config_exec)
prepare                Execute preparation stage (prepare_exec)
run                    Execute run stage (run_exec)
cleanup                Execute the cleanup stage (cleanup_exec)

$ jacamar-auth --help
Version: 0.1.0
Git Commit: 27b3184
Git Branch: develop
Go Version: go1.14.4-linux/amd64
Built: 2020-07-01T15:33:39+0000

Usage: jacamar-auth <command> [<args>]

Options:
--help, -h             display this help and exit
--version              display version and exit

Commands:
config                 Execute configuration stage (config_exec)
prepare                Execute preparation stage (prepare_exec)
run                    Execute run stage (run_exec)
cleanup                Execute the cleanup stage (cleanup_exec)

Jacamar relies on it’s own configuration file provided to the config stage via the --configuration command line argument. This can be realized in your runner’s standard config.toml:

[[runners]]
    ...
    [runners.custom]
    config_exec = "jacamar-auth"
    config_args = ["config", "--configuration", "/etc/gitlab-runner/custom-config.toml"]
    prepare_exec = "jacamar-auth"
    prepare_args = ["prepare"]
    run_exec = "jacamar-auth"
    run_args = ["run"]
    cleanup_exec = "jacamar-auth"
    cleanup_args = ["cleanup", "--configuration", "/etc/gitlab-runner/custom-config.toml"]

All key executors previously supported by the ECP enhancements (shell, cobalt, lsf, and slurm) are available.
- There is no longer any batch executor option, instead each scheduler is it’s own potential option.
```
[general]
name = "Example runner"
executor = "slurm"
```
Setuid is still supported but it has been greatly overhauled and separated completely from the authorization process responsible for identifying CI job user context. This means that a GitLab user can be authorized without relying ultimately on a setuid operation.
```
[general]
name = "Example setuid runner"
executor = "shell"
data_dir = "/ecp"

[auth]
enabled = true
run_mechanism = "setuid"
```
- Functionality such as allow/block-lists, federation, and runas have also been realized as part of this new authorization flow.
Privileged operations such as setuid are only capable via the minimal application layer jacamar-auth. This is meant to be referenced in the GitLab runner’s configuration and will in turn invoke jacamar.
- As of this release, both the GitLab runner and in turn jacamar-auth must still be executed with root level privilege to realize setuid.
Due to the extensive changes to Jacamar’s architectures, the system account log will no longer be used for identifying a job’s exit status in the Cobalt executor. Instead the <jobid>.cobaltlog file will be relied upon.
- The accounting_logs option is no longer observed in the configuration.
GitLab upstream does not correctly observe all possible configurations when dealing with cache stored on a local file system. To address this we’ve added the ability to enable a shared cache dir. This means that even though locally stored caches will observe the authorized user’s folder structure it cannot be used across different instances of the runner that share the same data_dir.
```
[general]
shared_cache_dir = false
```
The specified data_dir in the configuration has been greatly improved to diminish the complexity of the folder structure while still enforced expected permissions/ownership.
- Was known as setuid_data_dir in previous models.
- User owned directories (e.g. 700 permissions) will now be generated immediately following the defined data_dir. These can be created using similar standard as one would with a HOME directory. However, if the directory is not found the user will attempt to create their own, following a strict set if requirements to ensure an adequate level of security.