Articles > September 14, 2023 Diego Lasa

Integrating CI/CD pipelines with your workload manager with Gitlab runners and Jacamar CI

With the increasing usage of CI/CD pipelines for project development, testing, and deployment, HPC system users who have to navigate through the command line might be overwhelmed and find themselves having to deal with the overhead of integrating and running their projects in classical HPC architectures.

Running a Gitlab runner on your head node gives your users the option and flexibility to run pipelines using the built-in SSH or Shell executors in order to submit their jobs. However, full integration with the local workload manager is never achieved. Quite often, they have to rely on some scripting to plan and orchestrate their executions.

Jacamar CI comes to save the day by providing two mechanisms on top of the GitLab runner that allow full integration with the workload manager of your choice, whether it is SLURM, PBS, Cobalt or LSF. First, Jacamar CI integrates a series of custom executors that automatically interface with all the workload managers mentioned. Secondly, it also features downscoping mechanisms that allow direct mapping from a GitLab instance user to the local user of choice. This can be achieved via two different procedures, setuid procedure and sudo procedure, which are thoroughly documented on the project’s webpage. Additionally, it offers the possibility to create access control lists and validation methods to decide the target users that will be in charge of running the jobs on the HPC platforms. 

As for the installation, no high-level wizardry is required: the Gitlab runner is distributed as a precompiled binary, and you only need to choose the one compatible with your architecture. In regard to the compilation of Jacamar CI, building the project is as easy as running make on the source code via the provided makefile. Just make sure that all the required packages are installed (bash, Go, libc, git and libsecomp with devel package), and it should be straightforward.

With this setup in place, users will be able to run their pipelines directly from their Gitlab projects knowing that their job will be automatically queued on the target system’s workload manager just by incorporating a single variable containing their job specification to the pipeline.

System administrators can also benefit greatly from having a dedicated Gitlab runner + Jacamar CI environment installed and running on their head node/deployment node by integrating a potential CI/CD pipeline in their deployments, configurations or even to perform scientific software installations. This will prove to be a useful approach to keep track of changes, accountability of all the actions performed on the system and more, all centralised in a Gitlab instance.



Diego Lasa

Related Articles

Jordi Blasco

Enabling Abaqus Checkpointing & Restart with Slurm Workload Manager

How to enable this feature can be used to minimise the impact of a hardware issue by restarting the job from the last checkpoint.

Jordi Blasco

How to use the Slurm simulator as a development and testing environment

The Slurm simulator represents an extremely powerful. Benefits of adopting that technology, and how you can use it for your daily operations.

Elisabeth Ortega, Ph. D

Improving efficiency in HPC clusters using monitoring tools

A n efficient HPC cluster is one that has resources ready to be used on demand. Users are happy when they get access to the resources they need immediately.

Visit our forum

One of the main goals of this project is to motivate new initiatives and collaborations in the HPC field. Visit our forum to share your knowledge and discuss with other HPC experts!

About us

HPCKP (High-Performance Computing Knowledge Portal) is an Open Knowledge project focused on technology transfer and knowledge sharing in the HPC, AI and Quantum Science fields.

Promo HPCNow