Deployment

Due the design of Jacamar CI and the requirement to adhere to the structure of custom executor the deployment process consists of more than a single application. It is a multiple step process:

Installation: Either from supported packages or from source the process of installing not just Jacamar CI but also a patched version of the GitLab-Runner.
Runner Registration: Registering the runner with the GitLab server and accounting for custom executor requirements and advanced management.
Custom Executor Configuration: Specific steps required in order to configure the executor to correctly interact with Jacamar CI.
System Service: Establishing a systemd service for the runner process.

By engaging feedback and better understanding the desired deployment methods we are currently working to streamline the process. Additional comments are always welcome.

Installation

Before Jacamar can be deployed you must first install the GitLab Runner official release. If questions would arise regarding the runner installation we advise referring to the official installation documentation.

Requirements

Jacamar’s runtime requirements:

Bash
Git versions 2.9+
libc
libseccomp

In addition, if you wish to configure support for any scheduler then you are currently limited:

Packages

You can find supported releases on our project page. We recommend using these releases unless you are attempting to test or deployment specific features.

Note

Release v0.9.0 relocated all RPM installed binaries into a single location, /opt/jacamar/bin. This will offer a better standard moving forward, please be aware of this when upgrading from an older version.

With each release we currently support two potential deployment methods in addition to the supported architectures (x86_64 and ppc64le):

RPM (jacamar-ci-{VER}-1.{ARCH}.rpm) - Standard release with all applications/files deployed to /opt/jacamar with world read/execute permissions.
RPM w/Capabilities (jacamar-ci-caps-{VER}-1.{ARCH}.rpm) - Same directory structures as the standard release but with setuid and setgid capabilities provided to jacamar-auth application. Subsequently access is restricted to only the gitlab-runner user account (500 permissions).

Build from Source

Build requirements for a Linux system:

Go versions 1.17.5+
Make
libc
libseccomp

git clone https://gitlab.com/ecp-ci/jacamar-ci.git
cd jacamar-ci
make build
make install PREFIX=/usr/local

You can explore related Makefile commands with make help. Support has been added to assist in custom packaging as well as binary builds.

There are also containers available in Jacamar’s container registry that can be used for builds as all requirements are already accounted for.

RPM w/Containers

Docker or Podman can be used to build RPMs locally:

$ make rpm-container
echo "Running RPM Build in podman..."
...
+ cd /root/rpmbuild/BUILD
+ cd jacamar-ci-v0.10.0.pre.b91da87
+ /usr/bin/rm -rf /root/rpmbuild/BUILDROOT/jacamar-ci-0.10.0.pre.b91da87-1.el7.x86_64
+ exit 0
8017c7f4a80aa3dabd30af9102c1012b3c40a43577aa2ac95fbe0c2c51d50303  /builds/jacamar-ci/rpms/jacamar-ci-0.10.0.pre.b91da87-1.el7.x86_64.rpm

The resulting RPMs can then be found in the rpms/ directory.

Runner Registration

For general information on the process please see the Configuring GitLab Runners documentation. We advise also referring to the official documentation for complete details on both the interactive as well as command line registration methods. Keeping in mind there are additional potential advanced configuration options that can be be managed via the config.toml after successful registration.

Important

You will need to select the custom executor type in order to use Jacamar CI. However, you can change this setting at any time.

GitLab Runner Auth

In order to simplify the GitLab runner registration we recommend using LLNL/gitlab-runner-auth project. By leveraging this script as part of the ExecStartPre you can more easily manage runners on stateless as well as stateful resources.

Managing the TOML File

By default the configuration for the GitLab Runner can be found under the /etc/gitlab-runner directory. You can specify a different configuration file when launching the runner, for example:

gitlab-runner run --config /fs/gitlab-configuration/config.toml

In this case the runner will use the configuration file specified via the --config flag as opposed to the default location. This may prove beneficial if for instance you have the runner deployed on a stateless node and want to manage the configuration from a mounted directory.

Changes to the file can be made at any time, the runner will detect any alterations and reload the configuration. This will be observed by all subsequent jobs. The action is also logged by the runner:

Configuration loaded                                builds=0

Runner Tokens

As part of the runner registration process you will interact with two distinct but equally sensitive tokens. Depending on the type of runner you will be required to obtain the associated registration token. This token is only used during the registration process; however, the same toke can be utilized until it has been reset from the server. Resetting the registration token will not revoke access of any runner registered with that token.

After a successful registration the runner will be provided with a new authentication token. It is specifically scoped to the registered runner and used to authenticate all runner interactions with the GitLab server. It can be seen in plain text in the config.toml:

[[runners]]
    token = "<authentication token>"

Warning

The server recognizes runners by their unique tokens; however, a token is only tied to a specific runner/configuration by existing in the associated config.toml. There are no limitations in place to prevent a token from being used incorrectly to spoof a runner if illicitly obtained.

The token’s existence in the config.toml is required and as such should be protected to the best of your ability. Only the runner user is required to have read access to this file, as such it is advised to confirm correct file permissions.

Custom Executor Configuration

Additional steps are required for the configuration of the custom executor. Begin by adding the following to the table ([runners.custom]) to your runner config (e.g., /etc/gitlab-runner/config.toml).

[runners.custom]
  config_exec = "/opt/jacamar/bin/jacamar-auth"
  config_args = ["config", "--configuration", "/etc/gitlab-runner/custom-config.toml"]
  prepare_exec = "/opt/jacamar/bin/jacamar-auth"
  prepare_args = ["prepare"]
  run_exec = "/opt/jacamar/bin/jacamar-auth"
  run_args = ["run"]
  cleanup_exec = "/opt/jacamar/bin/jacamar-auth"
  cleanup_args = ["cleanup", "--configuration", "/etc/gitlab-runner/custom-config.toml"]

By including the above configuration you fulfil requirements both of the custom executor and Jacamar CI:

Each stage within a CI job (e.g. config, prepare, run, or cleanup) must have an associated executable/script defined. In our case we are using jacamar-auth application.
Additional arguments provided to jacamar-auth, in this case it requires a sub-command related to the current stage.
Jacamar CI requires its own set of configurations. We include them by specifying the --configuration argument and the location of the file we will be creating next.

Please see our configuration documentation for complete details.

System Service

The runner supports several system services though for the purposes of this documentation we are focusing on systemd exclusively.

Important

Runner management benefits from the configuration of the system service. Jacamar CI is designed to only be invoked by the runner managed through a process such as systemd.

Managing the creation of the service file can be done manually by an administrator using the gitlab-runner install and gitlab-runner uninstall commands respectively.

$ gitlab-runner install --user root --service example
Runtime platform        arch=amd64 os=linux pid=3632 revision=4b30b4d1 version=13.4.0

$ cat /etc/systemd/system/example.service
[Unit]
Description=GitLab Runner
...

$ gitlab-runner uninstall --service example
Runtime platform        arch=amd64 os=linux pid=3674 revision=4b30b4d1 version=13.4.0

Now if we examine an example service file it is possible to highlight several recommended settings that may be worth considering for you deployment:

[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/lib/gitlab-runner/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/path/to/gitlab-runner run --config /path/to/runner/prefix/config.toml --working-directory /example/gitlab-runner --config --service gitlab-runner --syslog --user root
Restart=always
RestartSec=120
StandardOutput=null
StandardError=null

[Install]
WantedBy=multi-user.target

The ExecStart has been configured to use the appropriate available runner parameters
gitlab-runner ... --syslog will correctly integrate with the system logging service.
To avoid duplication of output is recommend to defined both StandardOutput as well as StandardError to null.

Additional details on service deployment for either setuid or sudo can be found in the related guides:

Temporary Files/Directories

There are two major cases where the upstream runner will create temporary files and directories:

As of release 12.0 all job traces will be stored to disk.
When using the custom executor a temporary directory is used to transfer job scripts to the assigned driver.

In both cases the folder permissions are set so only the runner user will have access. In many cases this will be root or some other service user.

$ cd /tmp
$ ll | grep trace
drwx------ 1 root    root     94B Jun 17 10:49 custom-executor026800128
-rw------- 1 root    root     14K Jun 17 10:50 trace036573548
-rw------- 1 root    root     14K Jun 17 10:50 trace184916677

Note

A runner that has crashed or was improperly shutdown will not automatically cleanup any of these temporary files/directories. Else the cleanup will be handled without any administrator intervention.

The temporary directory used for storing files is determined by Go’s os.TempDir() function. In the case of Unix systems the default value will be /tmp unless there is a value set for $TMPDIR, this can be defined in the service file.

Limits on Resources

In the case of setuid downscoping, resource limits normally set through pam are not enforced (i.e., your /etc/security/limits.conf file). However, it is still possible to establish limits via systemd:

[Service]
ExecStart=/path/to/gitlab-runner
...
LimitNOFILE=1024
LimitLOCKS=infinity

Note

If using RHEL 8+ you may need to enable set_stack_size in the Jacamar CI configuration if defining a stack size greater than 8M.

Once defined and the service reset, all jobs will have these limits enforced. For complete details and all possible directives please see the systemd.exec Process Properties documentation or your man systemd.exec locally.