Building software for HPC environment is now easy with tools like Easybuild or Spack. Testing these applications can be done through vendor scripts such as make test or ctest that are part of the software. Most of these tests are done on binaries in the build directory instead of the install path where binaries will reside. We can reuse the vendor tests to point to the install path but this is very tricky such as recursive Makefiles, or CMakeLists.txt.
Furthermore, there is no universal HPC test toolkit that can be used by the entire HPC community to conduct tests using one medium. A project called buildtest aims to help share test scripts among the HPC community. buildtest is an python-based automatic test generating framework that makes use of YAML configurations to generate shell-scripts (.sh) that can run independently or work in CTest framework. buildtest is compatible with apps built with easybuild. buildtest can write binary, compilation, and scripting (R, Perl, Python, Ruby) tests. buildtest can also be used for testing system packages.
The goal of this project is to provide HPC sites to have access to a single test toolkit that can build tests according to their software collection. The toolkit will provide HPC engineers a means to check the software and quickly detect broken functionalities. buildtest can be used for educational purposes to help users learn about the application through testscripts.
