Spack

Spack
Name	Spack
Developer	Lawrence Livermore National Laboratory, Sandia National Laboratories, University of California
Initial release	2013
Written in	Python
Operating system	Linux, macOS
License	BSD-like

Spack is an open-source package management tool designed for building, installing, and managing scientific software stacks on high-performance computing systems. Created to address complex dependency resolution and configuration challenges on clusters and supercomputers, it automates compilation, variant selection, and environment stacking for scientific workflows. Spack emphasizes reproducibility, portability, and fine-grained control over build options to support research projects across laboratories, universities, national laboratories, and industry research centers.

Overview

Spack originated at Lawrence Livermore National Laboratory and grew through collaborations with Sandia National Laboratories, the University of California, and other institutions involved in computational science. It targets environments where users need multiple simultaneous versions of libraries such as MPI implementations, numerical libraries, and toolchains used by projects at institutions like Argonne National Laboratory, Oak Ridge National Laboratory, Los Alamos National Laboratory, CERN, and research groups at Massachusetts Institute of Technology and Stanford University. Typical users include researchers working with applications from organizations like NASA, NOAA, and national research facilities funded by agencies such as the U.S. Department of Energy and the National Science Foundation.

Features

Spack provides features tailored to scientific computing: fine-grained variant management, dependency graph resolution, compiler and toolchain abstractions, and binary caching. It supports multiple flavors of MPI such as Open MPI, MPICH, and vendor implementations like Intel MPI and Cray MPICH. Spack can manage build systems used by projects including CMake, Autoconf, Meson, and SCons and interacts with compilers such as GCC, Clang, and Intel C Compiler. It offers environment modules integration with systems such as Lmod and Environment Modules to facilitate deployment on clusters managed by resource managers like Slurm Workload Manager, PBS Professional, and Torque.

Architecture and Design

Spack's architecture centers on a Python-based package description layer, a concretizer that resolves dependency graphs, and a package build and install engine. Package recipes are authored in Python and reference external build tools and libraries commonly used in computational projects from institutions like Argonne National Laboratory and software foundations such as the Apache Software Foundation. The concretizer computes full, hashed build specifications that capture versions, variants, compilers, and dependencies, enabling reproducible installations across systems like those at NERSC and Fermilab. The design accommodates binary caches and mirrors used by repositories hosted on platforms including GitHub, GitLab, and organizational artifact servers.

Usage and Workflow

Users interact with Spack via a command-line interface to fetch, configure, build, and install packages. Typical workflows involve selecting compilers such as GCC or Clang, choosing MPI stacks like Open MPI or MPICH, and specifying build variants for libraries including HDF5, NetCDF, PETSc, and Trilinos. Reproducible workflows are achieved by concretizing specifications, creating environment modules, and leveraging binary caches to distribute prebuilt artifacts across clusters at facilities like Lawrence Berkeley National Laboratory and universities such as University of Cambridge and University of Oxford. Integration with continuous integration systems from providers like Jenkins, GitHub Actions, and GitLab CI/CD supports automated testing and packaging for software developed by groups affiliated with organizations like Siemens and IBM Research.

Package Management and Repositories

Spack organizes packages into repositories of Python package recipes. The central community repository is maintained by contributors from laboratories, universities, and corporations and is distributed via code hosting services such as GitHub and GitLab. Organizations operate private Spack repositories to manage internal stacks used by projects at Boeing, Lockheed Martin, and research collaborations with Schrödinger. Binary mirrors allow distribution of prebuilt bundles for clusters at centers like NERSC and ALCF; mirrors can be hosted on object stores provided by vendors such as Amazon Web Services and Microsoft Azure or on institutional storage systems managed by organizations like XSEDE.

Development and Community

Development is community-driven with contributions from national laboratories, academic groups, and companies. The project follows collaborative workflows common to projects at The Linux Foundation-hosted initiatives and open-source foundations. Governance and project coordination involve maintainers and contributors who often participate in conferences and workshops hosted by organizations such as SC Conference (Supercomputing), USENIX, and domain-specific meetings at AGU and SIAM. The community maintains mailing lists, issue trackers, and pull request processes on platforms like GitHub to coordinate work on package recipes, the concretizer, and integrations with HPC ecosystem components like Lmod and Spack's binary cache tooling.

Adoption and Use Cases

Spack is used widely in scientific computing for building complex software stacks required by computational chemistry, climate modeling, astrophysics, and materials science. Notable application stacks installed via Spack include frameworks and libraries such as GROMACS, LAMMPS, WRF, CESM, Quantum ESPRESSO, SIESTA, VASP (where licensed), and analysis tools used in pipelines at institutions like Los Alamos National Laboratory and Oak Ridge National Laboratory. It supports reproducible research workflows for collaborations involving universities like Princeton University, Harvard University, and University of California, Berkeley as well as industry research groups at Google Research and Microsoft Research.

Category:Package management systems