Welcome to the ECP CI Documentation

The Exascale Compute Project CI effort is focused on empowering software projects with new models for testing by enabling access to development HPC resources within their CI/CD workflows. While also preserving a team’s existing development processes and collaborating with facilities to ensure both the security, as well as reliability, of all testing capabilities.

Contents

If you are wondering “What is Continuous Integration (CI) testing”, then we encourage you to explore the related articles from Better Scientific Software (BSSw).

Enhancements

Supporting continuous integration on HPC resources necessitated a number of targeted software enhancements, new services, and the establishment of improved workflow capabilities:

  • Advanced Workflows: An ongoing collaborative effort between not just facility administrators but project teams to established documented workflows and best practices.

  • Batch Executors: Run user defined scripts at the job level by submitting to the underlying scheduling system.

  • Custom Executor Compatibility: Upstream GitLab has added support for the custom executor which allows for the implementation of administratively defined application to realize runner functionality. All future realizations of runner side enhancements will be completed within this exciting model.

  • Downscoping Permissions: Targeting the validated user responsible for triggering the CI pipeline and dropping permissions prior to job execution to that of their system local account.

The documentation found on this site elaborates on these enhancements for both individuals who wish to leverage CI resources as well as administrators planning to deploy these tools. As many of the enhancements are still under development, we welcome and appreciate any feedback/questions you may have.

CI Use Models

Facilities, as well as users, can leverage the ECP CI enhancements realized in the runner via the custom executor in a manner that supports the evolving needs of projects. Due to the flexibility afforded by GitLab both in source code management, as well as CI, via a range of supported executor and project level customization.

Sites can deploy an environment consisting of a local server, ECP enhanced executor, along with any additional resources they wish to support. The local model can be used to leverage resources unavailable outside of the facility while still benefiting from publicly available development, documentation, and pre-established workflows.

We are not intending the ECP CI effort to completely replace any existing CI workflows. Instead, we want these tools to benefit from flexibility offered to CI/CD pipelines both in configuration and runner deployment. Ideally these CI models will create added value when leveraged as part of any comprehensive testing strategy.

Open Source Projects

In order to realize the goals of the ECP CI effort we maintain several open source projects: