Models of cloud and high performance computing (HPC) integration have typically focused on bursted or coordinated models, where clusters of machines work alongside one another to combine service-oriented with simulation-based work.
As novel workloads emerge that combine heterogeneous components for machine learning, services, databases, and simulation, often requiring complex resources and orchestration, traditional workload managers and models for integration are no longer enough.
An approach is needed that seamlessly blends the two traditionally disparate technology spaces into one.
In this talk, we share work to run workload ensembles in user-space Kubernetes, a means to run the leading cloud-native orchestrator, Kubernetes, entirely in user-space, and with orchestration controlled by Flux Framework, the system scheduler deployed on El Capitan, the number 1 system in the Top500 that is also deployed on 5 other systems in the list.
We will showcase features such as elasticity, modularity, and that performance in user-space Kubernetes is comparable to that of bare metal.
This novel paradigm that enables the combination of simulation and service orchestration, done in a reproducible, declarative way on-premises, offers promise as a possible future for the HPC community, and should be of interest to administrators, developers, and scientists alike.
