Integrating CI/CD pipelines with your workload manager with Gitlab runners and Jacamar CI

With the increasing usage of CI/CD pipelines for project development, testing, and deployment, HPC system users who have to navigate through the command line might be overwhelmed and find themselves having to deal with the overhead of integrating and running their projects in classical HPC architectures.

Running a Gitlab runner on your head node gives your users the option and flexibility to run pipelines using the built-in SSH or Shell executors in order to submit their jobs. However, full integration with the local workload manager is never achieved. Quite often, they have to rely on some scripting to plan and orchestrate their executions.

Jacamar CI comes to save the day by providing two mechanisms on top of the GitLab runner that allow full integration with the workload manager of your choice, whether it is SLURM, PBS, Cobalt or LSF. First, Jacamar CI integrates a series of custom executors that automatically interface with all the workload managers mentioned. Secondly, it also features downscoping mechanisms that allow direct mapping from a GitLab instance user to the local user of choice. This can be achieved via two different procedures, setuid procedure and sudo procedure, which are thoroughly documented on the project’s webpage. Additionally, it offers the possibility to create access control lists and validation methods to decide the target users that will be in charge of running the jobs on the HPC platforms.

As for the installation, no high-level wizardry is required: the Gitlab runner is distributed as a precompiled binary, and you only need to choose the one compatible with your architecture. In regard to the compilation of Jacamar CI, building the project is as easy as running make on the source code via the provided makefile. Just make sure that all the required packages are installed (bash, Go, libc, git and libsecomp with devel package), and it should be straightforward.

With this setup in place, users will be able to run their pipelines directly from their Gitlab projects knowing that their job will be automatically queued on the target system’s workload manager just by incorporating a single variable containing their job specification to the pipeline.

System administrators can also benefit greatly from having a dedicated Gitlab runner + Jacamar CI environment installed and running on their head node/deployment node by integrating a potential CI/CD pipeline in their deployments, configurations or even to perform scientific software installations. This will prove to be a useful approach to keep track of changes, accountability of all the actions performed on the system and more, all centralised in a Gitlab instance.

Links: