Jacamar-Auth

Authorization Flow

For administrators, jacamar-auth is meant to provide an increased level of accountability to CI jobs that would otherwise execute under a single responsible service account. To support this goal a comprehensive yet flexible approach has been implemented that tightly couples; CI job context, local configurations, and administrator defined policies. The culmination of this authorization workflow is a trusted local account than can be targeted for downscoping process and execute the CI job itself.

Important

An important assumption is that server accounts are managed using the same underlying systems as those found on the target CI system. Meaning userA on GitLab is the same as userA on the system. Of equal importance is that they are unable to influence the potential username on the GitLab server. For additional details see the Security Considerations in the server documentation.

Core to this entire workflow is the job context provided that allows for us to establish not only definitive information regarding the job itself but the GitLab user who triggered it. This is accomplished by leveraging several key trusted CI variables provided to the custom executor along with the CI job JWT. Together, along with Jacamar’s config file, we are able to establish the verifiable context for the entire job.

There are several optional steps in the workflow:

• RunAs User: Custom script that is provided known user information and JWT established context to allow additional validation and limited overrides.

• Allow/Block Lists: Observed lists that can be used to prevent specific actions based upon verified user context.

• Downscoping Mechanisms: Leverages authorized user context to drop all current permissions and run the associated CI job as that user.

RunAs User

The RunAs configuration supports non-standard requirements in the authorization flow realized through an administrative defined script. Known jobs and current user context is provided and select overrides can be observed.

[auth.runas]
  validation_script = "/custom/run-validate.py"
  user_variable = "TARGET_SERVICE_USER"
  sha256 = "e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317"

The complete flow for this process:

Validation

The validation_script is a custom script/application, required in order in order to use any RunAs User functionality. It will be invoked as part of the authorization flow in one of two ways along with a standard set of environment variables for context:

1. $ script username - If no RunAs User Variable can be identified.

2. $ script verifier username - The user has provided a target user by the RunAs User Variable (verifier).

In either case the username provided is the assumed local username and the current target for any potential downscope operation. This is likely based upon the user login as established in the JWT.

In addition to command line arguments, as of Jacamar 0.3.0 additional context derived from the GitLab Job JWT is provided to the script via environment variables:

Key	Description
JWT_USER_LOGIN	Login username of the user who started the job. Can be user changed depending on server deployment (always verify before trusting this value).
JWT_USER_EMAIL	Primary email of the user who started the job.
JWT_USER_ID	Identification number of the user who started the job.
JWT_NAMESPACE_ID	Unique ID given to a username or group name that the current project belongs to.
JWT_PROJECT_ID	Unique ID of the current project.
JWT_PROJECT_PATH	Human readable namespace for the project.
JWT_JOB_ID	Unique ID of the current job that GitLab uses internally.
JWT_PIPELINE_ID	Unique ID of the current CI pipeline.
JWT_PIPELINE_SOURCE	The server identified CI pipeline source.
JWT_ISS	The JWT issuer’s GitLab domain.
JWT_IDENTITY_<name>	The user’s identities are optionally included, the server admin defined name is normalized to capital letters and any chanter not match A-Z0-9_ is simply removed.
RUNAS_TARGET_USER	User proposed account that can be the target for downscoping if approved (same as command line argument verifier).
RUNAS_CURRENT_USER	Currently identified local user account of the CI trigger user. This can differ from the JWT’s UserLogin depending on configuration of the authorization flow (same as command line argument username).

Simple test example Bash validation_script.

#!/bin/bash

if [ "$#" -eq 1 ]; then
    user="$1"
    if [ "$user" == "gitlabUser" ] ; then
        echo '{"username": "newUser"}'
    exit 0
fi
elif [ "$#" -eq 2 ]; then
    user="$2"
    service_account="$1"
fi
if [ "$service_account" == "passTest" ] ; then
    exit 0
fi

exit 1

JSON Overrides

It is possible to override select details via a JSON payload returned using stdout from your RunAs script:

{
    username: "<string>",
    data_dir: "<string>",
    user_message: "<string - sent to user job log>",
    error_message: "<string - sent to admin only log>"

}

Note

In cases where an error_message is not found the output from the script will be logged for administrative review. By default no output from the validation script will be conveyed to the user directly.

RunAs User Variable

The user_variable defines a potential CI environment variable that users/teams can leverage to request access to a target service account. It does not guarantee that the requested account will be observed, only that once the user provided value has been validated to a minimal level it will then be supplied to your Validation process.

Important

The value of this variable should not be trusted, instead it should be viewed only as a request that can then be further examined within the context of the rest of the user details.

This configuration is optional both for you the administrator as well as the CI user (if defined). The existence of an associated value will directly influence the Validation script execution.

Including User Identities

Important

Identities are not guaranteed to be provided to the script as their presence in the JWT currently must be enabled by the user. Please ensure that any script relying on these variables handle this case.

Authentication methods generate an entry in the identities table that links the subject of a succesfull external login to GitLab’s user.

To enable, either the user or a server administrator will need to turn on the pass_user_identities_to_ci_jwt via the API:

curl \
    --request PUT \
    --header "Private-Token: <token>" \
    --data "pass_user_identities_to_ci_jwt=true" \
    <gitlab-instance>/api/v4/user/preferences

Once enabled the JWT will be expanded with additional details which in turn will be added to your RunAs validation script’s environment. For instance, if the JWT has the following details in the payload:

"user_identities": [{
    "provider": "GITHUB",
    "extern_uid": "123456789"
  },{
    "provider": "oidc.example.com",
    "extern_uid": "user"
}]

After normalization the environment provided to your RunAs script will include:

• JWT_IDENTITY_GITHUB=123456789

• JWT_IDENTITY_OIDCEXAMPLECOM=user

Allow/Block Lists

The user/group lists provide direct control over who has access to the runner. At several points during authorization the currently identified target user’s local account will be evaluated against these rules. If a failure state is encountered the job will end before any user influenced code can be executed. An error message for this failure will be logged; however, the user facing information is generalized to avoid exposing specifics of the configuration.

• user_allowlist - An authoritative list of users who can execute Gitlab SetUID Runners.

• user_blocklist - A list of usernames that are not allowed to run CI jobs. More authoritative than group allowlist / blocklist, but can be overridden by user allowlist.

• groups_blocklist - A list of groups that are not allowed to run CI jobs.

• groups_allowlist - A list of groups that are allowed to run CI jobs. Least authoritative.

All lists can be configured within Jacamar’s Auth table by defining an array of local user/group names.

[auth]
  user_allowlist = ["usr1"]
  user_blocklist = ["usr2", "usr3"]
  groups_allowlist = ["grp1", "grp2"]
  groups_blocklist = ["grp3"]

Any usernames or groups provided are assumed to relate to Linux user/groups on the local system. The runner does not ensure the lists configured in are in fact valid and mistakes can allow undesirable job results.

Shell Allowlist

If defined, provides an authoritative list of acceptable shells. Any deviations from this list will result in job failure. The user database is consulted to identify the target CI user’s local shell.

[auth]
  shell_allowlist = ["/bin/bash", "/bin/zsh"]

In the above example, only user’s whose default shell has been defined as either /bin/bash or /bin/zsh will be allowed to execute jobs.

Note

Regardless of the user’s default all jobs will be executed in a non-interactive Bash login shell.

Downscoping Mechanisms

After the Authorization Flow has successfully completed we are able to act upon the validated user by downscoping permissions to this target and beginning the CI execution process.

[auth]
  downscope = "setuid"

Value	Description
setuid	Levering underlying system calls to setuid/setgid when creating a child process for actual job execution.
sudo	Construct a targeted command (sudo -E -u <username> -- /usr/bin/jacamar <stage>), relying on the sudo application to enforce downscoping of permissions.
none	Indicates you wish to leverage the authorization capabilities of jacamar-auth but the job will be run without downscoping permissions.
Default	There is no default value to downscope and not specifying one when using jacamar-auth will result in a failed job.

For our core mechanism, setuid, we are relying on Go’s implementation of setuid for spawning a child process owned by the user via credential package. This child process will be tightly controlled and a user owned shell that will launch Jacamar with all the necessary context to execute the CI job in the administrator expected manner. Only stdout/stderr will be piped back to the runner, thus preserving the custom executor model while still realizing desired Security Model.

Note

If you are familiar with previous iterations of the ECP runner enhancements then you might have used setuid = true. Though this has changed to downscope = "setuid" we currently only support the setuid related operation. There are plans to expand this in the future.

Security Model

The authorization functionality relies on the CI job permissions model that was introduced in GitLab 8.12. In particular, the following points from the CI job permission spec are key to understand:

1. Job permissions are tightly integrated with the user who triggered the job

2. GitLab is already aware of who triggers a job/pipeline

3. Your CI job can access everything that you as a user can access

4. We (runner host machines) already know know/enforce user permissions

5. Jobs are also limited by the server to the responsible user’s permissions

The downscoping feature extends these principles onto the runner’s host system by executing CI jobs as their local account. Since CI jobs now run as an unprivileged user processes, job permissions carry over to the OS, and they can access files and directories in the host filesystem that their user has access to.

Job Token Handling

Important

With server version 14.5+ it is now easy to limit the CI job token scope at a project level. We highly recommend utilizing this feature

For each CI job a unique job token is generated. This is scoped to user’s account, provides read access to all their projects, and remains valid for the duration of the job. In the traditional model the runner generated job scripts would use this token via the command line. In multi-tenant environments however, users can potentially inspect all commands of running processes.

Jacamar takes steps to hide these tokens from appearing as command line arguments in any runner generated commands. This includes uploading/downloading artifacts, and all Git interactions by carefully leveraging the GIT_ASKPASS env variable.