Lars Bilke^1*, Thomas Fischer¹, Tobias Meisel¹, Dmitri Naumov¹

1 Helmholtz Centre for Environmental Research – UFZ, Leipzig, Germany

* Corresponding authors emails: lars.bilke@ufz.de

DOI10.24435/materialscloud:b4-ex [version v1]

Publication date: Dec 17, 2024

How to cite this record

Lars Bilke, Thomas Fischer, Tobias Meisel, Dmitri Naumov, Reproducible HPC software deployments, simulations and workflows, Materials Cloud Archive 2024.202 (2024), https://doi.org/10.24435/materialscloud:b4-ex

Description

Reproducibility in running scientific simulations on high-performance computing (HPC) environments is a persistent challenge due to variations in software and hardware stacks. Differences in software versions or hardware-specific optimizations often lead to discrepancies in simulation outputs. While Linux containers are commonly used to standardize software environments, tools like Docker lack reproducibility in image creation, requiring archiving of binary image blobs for future use. This method turns containers into black boxes, preventing verification of how the contained software was built. In the linked paper, we demonstrate how we use GNU Guix to create our software stack bit-by-bit reproducible from a source bootstrap. Our approach incorporates a portable OpenMPI implementation, optimized software builds, and deployment via Apptainer images across three HPC environments. We show that our reproducible software stack facilitates consistent multi-physics simulations and complex workflows on diverse HPC platforms, exemplified by the OpenGeoSys software project. To ensure provenance of our findings, we utilized the AiiDA workflow manager. This dataset includes the complete AiiDA provenance database underlying the results presented in the paper. The AiiDA workflow itself is defined in and can be reproduced with this repository: https://gitlab.opengeosys.org/bilke/hpc-container-study.

Materials Cloud sections using this data

Files

File name	Size	Description
hpc-container-study-1.aiida MD5md5:3fc102a23a330522dd01dca25ec1ec01 Open this AiiDA archive on renkulab.io (https://renkulab.io/)	2.5 GiB	AiiDA archive

License

Files and data are licensed under the terms of the following license: Creative Commons Attribution Share Alike 4.0 International.
Metadata, except for email addresses, are licensed under the Creative Commons Attribution Share-Alike 4.0 International license.

External references

Keywords

OpenGeoSys AiiDA HPC workflows reproducibility GNU Guix