Migrating to new id_tokens from CI_JOB_JWT

With the upcoming release of GitLab server 16.5 (currently planned for October 22, 2023) the old versions of the JWT are being fully deprecated. If you are un-familiar with the CI_JOB_JWT in Jacamar CI, these are utilized to consistently and securely identify key information about the CI job and most importantly the user responsible for triggering it.

Important

Once you deploy server version 16.5 any job run by Jacamar CI will no longer function without the user providing the id_tokens in their respective job. This guide coupled with the recent v0.15 release of Jacamar CI can help with the migration.

For additional details regarding the changes and the new id_tokens please see the official Secure GitLab CI/CD workflows using OIDC JWT on a DevSecOps platform blog post. It is an excellent resource and if you’re unfamiliar with JSON Web Tokens in GitLab it can help immensely.

Migrating with Jacamar CI v0.15+

Before proceeding with the migration verify that your server is v15.7+. Previous versions will not have the id_tokens functionality available.

Install the v0.15+ release of Jacamar CI on your system. Then you’ll need to modify your Jacamar CI configuration to include:

[general]
jwt_env_variable = "SITE_ID_TOKEN"

Warning

The jwt_env_variable defines the expected environment variable Jacamar CI will look for, if not present the default CI_JOB_JWT will be used. The value can be anything you want; however, we advise not using CI_JOB_JWT else Jacamar CI cannot tell the difference and will be unable to warn the user correctly of any required job modification.

With just that modification anyone utilizing this runner will now encounter the following warning if a job does not contain the defined jwt_env_variable:

Preparing the "custom" executor
No id_token found on SITE_ID_TOKEN variable. Please update your CI job to include the following:

  id_tokens:
    SITE_ID_TOKEN:
      aud: https://gitlab.example.com

Prior to server release v16.0 the default CI_JOB_JWT will remain available. For additional details see: https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens
Using Custom executor with driver Jacamar CI 0.15.0...

Once the server is upgraded to v16.0+ the CI_JOB_JWT will no longer be provided by default and thus the job will break; however, the warning message will stay. Please keep in mind that even though users can request id_tokens they remain signed by the GitLab server and will be strictly verified against the public keys.

Once the new id_tokens field has been added the warning message will no longer appear. You can further verify the desired functionality by limiting JWT access in the project settings which completely removes the default CI_JOB_JWT from any and all jobs.

Migrating with Previous Jacamar CI Versions

You are not required to upgrade your deployment of Jacamar CI to migrate to id_tokens, only that the server is v15.7+. Have users add the following to any CI job using Jacamar CI:

example_job:
  id_tokens:
    CI_JOB_JWT:
      aud: https://gitlab.example.com

To preserve compatability Jacamar CI will always attempt to check the CI_JOB_JWT variable for a valid token. The audience (aud) can be set to anything you want, though we recommend using a consistent value as you can later enforce a required audience via configuration.

Leveraging Include/Extends

There are ways that you can assist users in migrating jobs, while at the same time provide a degree of future proofing with:

  • include - “Import configuration from other YAML files”

  • extends - “Configuration entries that this job inherits from”

The goal is to establish a documented practice that encourages the use of administratively defined configurations users can import as opposed to defining specific tags/id_tokens.

For example, let’s say we have a project called defaults that exists within our ci/ sub-group that is made available to users as an internal project with maintenance/development permissions restricted to appropriate administrators. Here we can create a runners.yml file for the purpose of providing runner specific configurations to our CI users:

.site-id-token: &site-id-token
  id_tokens:
    CI_JOB_JWT:
      aud: https://gitlab.example.com

.example-shell-runner:
  <<: *site-id-token
  tags:
    - example-system
    - shell

.example-slurm-runner:
  <<: *site-id-token
  tags:
    - example-system
    - slurm

.dev-shell-runner:
  <<: *site-id-token
  tags:
    - dev-system
    - shell

Now when migrating a .gitlab-ci.yml to prepare for the v16.0+ server release it would look like:

include:
  - project: 'ci/defaults'
    ref: main
    file:
      - '/runners.yml'

build:
  extends:
    # The tags/id_token will be inherited as defined in our runners.yml file.
    # Any previously existing tags should be removed.
    - .example-shell-runner
  script:
    - make build

Note

Any keywords established in our default configurations can be overwritten by users at the job level.

Users will still be responsible for updating any and all .gitlab-ci.yml files to leverage these new configurations. However, by choosing to use this include + extends workflow we hope that any future requirements or runner modifications become easier to realize without the need for manual intervention.