Migrating to new id_tokens from CI_JOB_JWT

Important

Once you deploy server version 17.0 releases of Jacamar CI prior to v0.18.0 will no longer function.

With the release of GitLab server 17.0 there are several keys JWT related features being deprecated that you should be aware of:

Old versions of the JWT are being fully deprecated in favor of id_tokens. If you are un-familiar with the CI_JOB_JWT in Jacamar CI, it is utilized to consistently and securely identify key information about the CI job and most importantly the user responsible for triggering it.
- For additional details regarding the changes and the new id_tokens please see the official Secure GitLab CI/CD workflows using OIDC JWT on a DevSecOps platform blog post. It is an excellent resource and if you’re unfamiliar with JSON Web Tokens in GitLab it can help immensely.
JWKS instance endpoint (/-/jwks) is being removed in favor of the /oauth/discovery/keys. Upgrading to Jacamar CI is the only fix for this.

Migrating with Jacamar CI v0.18+

Before proceeding with the migration verify that your server is v15.7+. Previous versions will not have the id_tokens functionality available.

Install the v0.18+ release of Jacamar CI on your system. Then you’ll need to modify your Jacamar CI configuration to include:

[general]
jwt_env_variable = "SITE_ID_TOKEN"

Warning

The jwt_env_variable defines the expected environment variable Jacamar CI will look for, if not present the default CI_JOB_JWT will be used. The value can be anything you want; however, we advise not using CI_JOB_JWT else Jacamar CI cannot tell the difference and will be unable to warn the user correctly of any required job modification.

With just that modification anyone utilizing this runner will now encounter the following warning if a job does not contain the defined jwt_env_variable:

Preparing the "custom" executor
No id_token found on SITE_ID_TOKEN variable. Please update your CI job to include the following:

  id_tokens:
    SITE_ID_TOKEN:
      aud: https://gitlab.example.com

Prior to server release v16.0 the default CI_JOB_JWT will remain available. For additional details see: https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens
Using Custom executor with driver Jacamar CI 0.18.0...

Once the server is upgraded to v17.0+ the CI_JOB_JWT will no longer be provided by default and the job will fail with an obfuscated authorization error; however, this warning message will remain. Please keep in mind that even though users can request id_tokens they remain signed by the GitLab server and will be strictly verified against the public keys.

Once the new id_tokens field has been added the warning message will no longer appear. You can further verify the desired functionality by limiting JWT access in the project settings which completely removes the default CI_JOB_JWT from any and all jobs.

Leveraging Include/Extends

There are ways that you can assist users in migrating jobs, while at the same time provide a degree of future proofing with:

include - “Import configuration from other YAML files”
extends - “Configuration entries that this job inherits from”

The goal is to establish a documented practice that encourages the use of administratively defined configurations users can import as opposed to defining specific tags/id_tokens.

For example, let’s say we have a project called defaults that exists within our ci/ sub-group that is made available to users as an internal project with maintenance/development permissions restricted to appropriate administrators. Here we can create a runners.yml file for the purpose of providing runner specific configurations to our CI users:

.site-id-token: &site-id-token
  id_tokens:
    CI_JOB_JWT:
      aud: https://gitlab.example.com

.example-shell-runner:
  <<: *site-id-token
  tags:
    - example-system
    - shell

.example-slurm-runner:
  <<: *site-id-token
  tags:
    - example-system
    - slurm

.dev-shell-runner:
  <<: *site-id-token
  tags:
    - dev-system
    - shell

Now when migrating a .gitlab-ci.yml to prepare for the v17.0+ server release it would look like:

include:
  - project: 'ci/defaults'
    ref: main
    file:
      - '/runners.yml'

build:
  extends:
    # The tags/id_token will be inherited as defined in our runners.yml file.
    # Any previously existing tags should be removed.
    - .example-shell-runner
  script:
    - make build

Note

Any keywords established in our default configurations can be overwritten by users at the job level.

Users will still be responsible for updating any and all .gitlab-ci.yml files to leverage these new configurations. However, by choosing to use this include + extends workflow we hope that any future requirements or runner modifications become easier to realize without the need for manual intervention.

Beginning Migration with Earlier Jacamar CI Versions

Note

Though it is is possible to begin migration to id_tokens with this method you will need to upgrade to Jacamar CI to v0.18+ in order to correctly handle other feature being deprecated in the server.

You are not required to upgrade your deployment of Jacamar CI to begin migrating to id_tokens, only that the server is v15.7+. Simply have users add the following to any CI job using Jacamar CI:

example_job:
  id_tokens:
    CI_JOB_JWT:
      aud: https://gitlab.example.com

To preserve compatibility Jacamar CI will always attempt to check the CI_JOB_JWT variable for a valid token. The audience (aud) can be set to anything you want, though we recommend using a consistent value as you can later enforce a required audience via configuration.

Another option introduced in GitLab server v16.4 is defining id_tokens at the default level:

default:
  id_tokens:
    CI_JOB_JWT:
      aud: https://gitlab.example.com

Doing this will cause all jobs within your pipeline to to have the CI_JOB_JWT generated and included automatically. For projects exclusively using Jacamar CI this will likely present the easiest migration option.