More and more researchers are expanding their workflows to include multiple applications that would normally require an administrator to install, as they traditionally install into all parts of the Operating system. This causes delays to research and increases the administrator’s workload both with the install and upkeep of each new application. Containers have been a great way to solve this but what happens when running on a HPC cluster without containers?
This talk is part of a wider data centric set of activities looking at workflows, that presents a way to build and use contained applications without requiring an administrator to install additional components. By utilizing techniques usually used in High Availability clusters, researchers are able to install applications to assist with Machine Learning and Big Data jobs in a single location. While still making them available to the whole HPC cluster if needed via the job scheduling tool utilised in their environment, such as IBM Spectrum LSF, Slurm or Flux.
