Non-Root Jacamar CI Downscoping (with Capabilities) via SetUID

Jacamar CI offers a robust series of configurations that seek to support advanced CI/CD workflows on HPC test resources. As part of these workflows the authorization process enables optional downscoping that limits permissions to that of the pipeline trigger user’s local account. Traditionally this would require the usage of root; however, with recent improvements we now support downscoping by leveraging Linux capabilities. This guide details how you can potentially deploy, configure, and run Jacamar CI in such a manner.

Note

If you’ve never used Jacamar CI or the GitLab Custom Executor before we highly encourage you to first explore the Administrative Tutorial.

Deployment

To begin the deployment process you will require usage of a privileged account.

  1. Obtain the latest Jacamar CI RPMs and runner release.

  • With Jacamar CI release packages there is an optional deployment with configuration already enabled for use with the gitlab-runner account (“RPM - w/Capabilities”). This guide details configuring both.

  1. Install both RPMs and any necessary dependencies (sudo rpm -i ...).

Note

Release v0.9.0 relocated all RPM installed binaries into a single location, /opt/jacamar/bin. This will offer a better standard moving forward, please be aware of this when upgrading from an older version.

  1. Switch to the non-root user account (e.g., gitlab-runner) and register the runner with your target GitLab server:

Important

You will need a service account that will take responsibility for both the gitlab-runner and jacamar-auth processes as well as configurations. For the purposes of this guide we are using a gitlab-runner user account, if you’ve deployed the runner with the official release this service account likely already exists.

Configurations

Configure both the Runner and Jacamar CI in a mostly traditional manner; however, special care must be given to the location these files are stored. Our gitlab-runner user must be able to read the files and sensitive tokens/configurations can be found in both. To accomplish this we are going to keep both configuration in the user’s home directory (/home/gitlab-runner/.gitlab-runner) but you are free to store wherever easiest.

Runner

  1. Edit /home/gitlab-runner/.gitlab-runner/config.toml generated during registration to add the appropriate options to the [runners.custom] table:

    • concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Jacamar CI Cap Testing"
  url = "https://gitlab.example.com/"
  token = "<RUNNER-TOKEN>"
  executor = "custom"
  [runners.custom]
    config_exec = "/opt/jacamar/bin/jacamar-auth"
    config_args = ["config", "--configuration", "/home/gitlab-runner/.gitlab-runner/jacamar-config.toml"]
    prepare_exec = "/opt/jacamar/bin/jacamar-auth"
    prepare_args = ["prepare"]
    run_exec = "/opt/jacamar/bin/jacamar-auth"
    run_args = ["run"]
    cleanup_exec = "/opt/jacamar/bin/jacamar-auth"
    cleanup_args = ["cleanup", "--configuration", "/home/gitlab-runner/.gitlab-runner/jacamar-config.toml"]

Jacamar CI

  1. Create the /home/gitlab-runner/.gitlab-runner/jacamar-config.toml we referenced in our Runner configuration:

  • [general]
  executor = "shell"
  data_dir = "$HOME"

[auth]
  downscope = "setuid"

Please note we’ve demonstrated a very minimal configuration for the purposes of managing jobs under a non-root user. Please reference Jacamar CI’s configuration documentation for a complete list of potential options.

System Service

Returning to a privileged account:

  1. Identify if the default runner process is installed, sudo systemctl status gitlab-runner

  • If missing use the runner to install a system service.

  • Important, avoid having multiple runner services active on the same machine, this can lead to unexpected results.

  1. Edit the /etc/systemd/system/gitlab-runner.service file:

  • [Unit]
Description=GitLab Runner using Jacamar CI with Capabilities
After=syslog.target network.target
ConditionFileIsExecutable=/usr/bin/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--syslog"  "--working-directory" "/opt/jacamar" "--config" "/home/gitlab-runner/.gitlab-runner/config.toml" "--service" "gitlab--runner" "--user" "gitlab-runner"
Restart=always
RestartSec=120
User=gitlab-runner
Group=gitlab-runner

[Install]
WantedBy=multi-user.target

  • It is important to target the gitlab-runner account (using User and Group) as both the gitlab-runner and jacamar-auth applications will execute under the same user.

  • $ getcap /opt/jacamar/bin/jacamar-auth
/opt/jacamar/bin/jacamar-auth cap_setgid,cap_setuid=ep

  1. Edit the /etc/systemd/system/gitlab-runner.service file:

  • [Unit]
Description=GitLab Runner using Jacamar CI with Capabilities
After=syslog.target network.target
ConditionFileIsExecutable=/usr/bin/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStartPre=+bash -c '/usr/bin/chown gitlab-runner:gitlab-runner /opt/jacamar/bin/jacamar-auth && /usr/bin/chmod 700 /opt/jacamar/bin/jacamar-auth && /usr/sbin/setcap cap_setuid,cap_setgid=ep /opt/jacamar/bin/jacamar-auth'
ExecStart=/usr/bin/gitlab-runner "run" "--syslog"  "--working-directory" "/opt/jacamar" "--config" "/home/gitlab-runner/.gitlab-runner/config.toml" "--service" "gitlab--runner" "--user" "gitlab-runner"
Restart=always
RestartSec=120
User=gitlab-runner
Group=gitlab-runner

[Install]
WantedBy=multi-user.target

  • It is important to target the gitlab-runner account (using User and Group) as both the gitlab-runner and jacamar-auth applications will execute under the same user.

  • By declaring the =+ in our ExecStartPre it will run the subsequent command as root as opposed to the gitlab-runner user. This allows for privileged actions (e.g., setting file ownership and capabilities).

Note

If you are running a versions of systemd older than 240 you will need to use the PermissionsStartOnly flag as opposed to =+

[Service]
...
PermissionsStartOnly=true
ExecStartPre=bash -c '...

  1. Restart and verify service is running:

  • sudo systemctl disable gitlab-runner.service
sudo systemctl enable gitlab-runner.service
sudo systemctl restart gitlab-runner.service
sudo systemctl status gitlab-runner.service

Examine Process

Finally, with the runner polling for jobs we advise creating test cases (.gitlab-ci.yml)to verify desired functionality.

ci-job:
  script:
    - id
    - /sbin/capsh --print
    - sleep 120

Our example job is just a simple look that demonstrates the user executing the CI job locally, ensuring the capabilities were not improperly set (inherited), and providing sufficient time during which you can more closely examine the locally running processes:

$ ps -aef --forest
UID          PID    PPID  C STIME TTY          TIME CMD
...
gitlab-+    1053       1  0 15:40 ?        00:00:06 /usr/bin/gitlab-runner run --syslog --working-directory /opt/jacamar --config /home/gitlab-runner/.gitlab-runner/config.toml
gitlab-+    2517    1053  0 17:20 ?        00:00:00  \_ /opt/jacamar/bin/jacamar-auth -u run /tmp/custom-executor440226574/script347499535/script. build_script
usr         2523    2517  0 17:20 ?        00:00:00      \_ /usr/bin/jacamar --no-auth run env-script build_script
usr         2528    2523  0 17:20 ?        00:00:00          \_ /usr/bin/bash --login
usr         2553    2528  0 17:20 ?        00:00:00              \_ bash /home/usr/.jacamar-ci/scripts/short/000/group/project/981625/build_script.bash
usr         2555    2553  0 17:20 ?        00:00:00                  \_ bash /home/usr/.jacamar-ci/scripts/short/000/group/project/981625/build_script.bash
usr         2559    2555  0 17:20 ?        00:00:00                      \_ sleep 120

We encourage you to take whatever steps you feel sufficient to ensure your deployment meets expectations beyond even what we show here. However, at this point you are able to expand utilization or explore additional configuration options.