Non-Root Jacamar CI Downscoping via Sudo

Jacamar CI offers a robust series of configurations that seek to support advanced CI/CD workflows on HPC test resources. As part of these workflows the authorization process enables optional downscoping that limits permissions to that of the pipeline trigger user’s local account. This guide details how you can potentially deploy, configure, and run Jacamar CI with the traditional sudo-based permission scheme.

Note

If you’ve never used Jacamar CI or the GitLab Custom Executor before we highly encourage you to first explore the Administrative Tutorial.

Deployment

To begin the deployment process you will require usage of a privileged account.

Obtain the latest release RPMs, depending on the version you may be required to use a patched version of the runner.
Install both RPMs and any necessary dependencies (sudo rpm -i ...).

Note that there is a /opt/jacamar/bin created that maintains the jacamar-auth binary owned by root with 700 permissions. We will address this later in the guide but be warned if you installed from source this protected jacamar-auth deployment is only present in the RPM.

Switch to the desired non-root user account and register the runner with your target GitLab server:

gitlab-runner register

There are non-interactive options as well external tools (e.g., LLNL/gitlab-runner-auth) that can be used to manage the registration process.

Important

You will need a service account that will take responsibility for both the gitlab-runner and jacamar-auth processes as well as configurations. For the purposes of this guide we are using a ci-manager user account we’ve arbitrarily named.

Configurations

We will need to configure both the Runner and Jacamar CI in a mostly traditional manner; however, special care must be given to the location these files are stored. Our ci-manager user must be able to read the files and sensitive tokens/configurations can be found in both. To accomplish this we are going to keep both configuration in the user’s home directory but you are free to store wherever easiest.

Runner

Edit /home/ci-manager/.gitlab-runner/config.toml generated during registration to add the appropriate options to the [runners.custom] table:

concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Jacamar CI Cap Testing"
  url = "https://code.ornl.gov/"
  token = "<RUNNER-TOKEN>"
  executor = "custom"
  [runners.custom]
    config_exec = "/opt/jacamar/bin/jacamar-auth"
    config_args = ["config", "--configuration", "/home/ci-manager/.gitlab-runner/jacamar-config.toml"]
    prepare_exec = "/opt/jacamar/bin/jacamar-auth"
    prepare_args = ["prepare"]
    run_exec = "/opt/jacamar/bin/jacamar-auth"
    run_args = ["run"]
    cleanup_exec = "/opt/jacamar/bin/jacamar-auth"
    cleanup_args = ["cleanup", "--configuration", "/home/ci-manager/.gitlab-runner/jacamar-config.toml"]

Jacamar CI

Create the /home/ci-manager/.gitlab-runner/jacamar-config.toml we referenced in our Runner configuration:

[general]
  executor = "shell"
  data_dir = "$HOME"

[auth]
  downscope = "sudo"

Please note we’ve demonstrated a very minimal configuration for the purposes of managing jobs under a non-root user. Please reference Jacamar CI’s configuration documentation for a complete list of potential options. Since the entirety of the CI job (including scheduler interactions) all non [auth*] configurations will function in a non-root deployment.

System Service

Returning to a privileged account, execute the following:

sudo gitlab-runner install -n jacamar-ci-runner -u ci-manager –syslog -c /home/ci-manager/.gitlab-runner/config.toml
- This will install a system service for us to use.

Edit the /etc/systemd/system/jacamar-ci-runner.service file

we just created:
[Unit]
Description=GitLab Runner using Jacamar CI with sudo
After=syslog.target network.target
ConditionFileIsExecutable=/usr/bin/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStartPre=+bash -c '/usr/bin/chown -R ci-manager:ci-manager /opt/jacamar && /usr/bin/chmod -R 770 /opt/jacamar'
ExecStart=/usr/bin/gitlab-runner "run" "--syslog"  "--working-directory" "/opt/jacamar" "--config" "/home/ci-manager/.gitlab-runner/config.toml" "--service" "jacamar-ci-runner" "--user" "ci-manager"
Restart=always
RestartSec=120
User=ci-manager
Group=ci-manager

[Install]
WantedBy=multi-user.target
It is important to target the ci-manager account (using User and Group) as both the gitlab-runner and jacamar-auth process will execute under the same user.

By declaring the =+ in our ExecStartPre it will run the subsequent command as root as opposed to the ci-manager. This allows for privileged actions (e.g., setting file ownership).

Note

If you are running a versions of systemd older than 240 you will need to use the PermissionsStartOnly flag as opposed to =+

[Service]
...
PermissionsStartOnly=true
ExecStartPre=bash -c '...

Add ci-manager user and group permissions
- Use visudo or edit /etc/sudoers to give the ci-manager permission to use sudo to execute programs without a password. Our ci-manager will expect to impersonate a user of it’s own group, ALL=(%ci-manager). The command should allow the environment to be maintained, SETENV: and to be run without a password prompt, NOPASSWD:. The line in the sudoers file would look like this, if the user was given permissions for ALL executables:
```
ci-manager ALL=(%ci-manager)  NOPASSWD:SETENV: ALL
```
  - For the strictest control, the user should only be given permissions to execute the Jacamar executable:
    ci-manager ALL=(%ci-manager) NOPASSWD:SETENV: /usr/bin/jacamar
Add known users to ci-manager groups, so the ci-manager user can impersonate them during execution.
- ```
sudo usermod -G ci-manager <user1>
sudo usermod -G ci-manager <user2>
```
Restart and verify service is running:

sudo systemctl disable jacamar-ci-runner.service
sudo systemctl enable jacamar-ci-runner.service
sudo systemctl restart jacamar-ci-runner.service
sudo systemctl status jacamar-ci-runner.service

Examine Process

Finally, with the runner polling for jobs we advise creating test cases (.gitlab-ci.yml)to verify desired functionality.

ci-job:
  script:
    - id
    - /sbin/capsh --print
    - sleep 120

Our example job is just a simple look that demonstrates the user executing the CI job locally, ensuring the capabilities were not improperly set (inherited), and providing sufficient time during which you can more closely examine the locally running processes:

$ ps -aef --forest
UID          PID    PPID  C STIME TTY          TIME CMD
...
ci-mana+    1053       1  0 15:40 ?        00:00:06 /usr/bin/gitlab-runner run --syslog --working-directory /opt/jacamar --config /home/ci-manager/.gitlab-runner/config.toml
ci-mana+    2517    1053  0 17:20 ?        00:00:00  \_ /opt/jacamar/bin/jacamar-auth -u run /tmp/custom-executor440226574/script347499535/script. build_script
usr         2523    2517  0 17:20 ?        00:00:00      \_ /usr/bin/jacamar --no-auth run env-script build_script
usr         2528    2523  0 17:20 ?        00:00:00          \_ /usr/bin/bash --login
usr         2553    2528  0 17:20 ?        00:00:00              \_ bash /home/usr/.jacamar-ci/scripts/short/000/group/project/981625/build_script.bash
usr         2555    2553  0 17:20 ?        00:00:00                  \_ bash /home/usr/.jacamar-ci/scripts/short/000/group/project/981625/build_script.bash
usr         2559    2555  0 17:20 ?        00:00:00                      \_ sleep 120

We encourage you to take whatever steps you feel sufficient to ensure your deployment meets expectations beyond even what we show here. However, at this point you are able to expand utilization or explore additional configuration options.