Singularity (container)

Singularity (container)
Name	Singularity
Developer	Sylabs
Released	2016
Programming language	Go
Operating system	Linux
License	BSD-3-Clause

Singularity (container) is a containerization platform designed to enable reproducible, portable, and secure execution of applications on high-performance computing systems. It emphasizes compatibility with scientific workloads, integration with cluster schedulers, and non-root execution to meet the needs of researchers at institutions such as Lawrence Berkeley National Laboratory, Argonne National Laboratory, Oak Ridge National Laboratory, National Energy Research Scientific Computing Center, and universities participating in the Open Science Grid. Singularity was developed to work alongside container standards like Docker (software), orchestration systems such as Kubernetes, and package ecosystems including Conda (package manager) and Spack.

Overview

Singularity emerged from research and development conducted at Lawrence Berkeley National Laboratory and was later commercialized by Sylabs. It targets scientific computing environments managed by schedulers such as Slurm Workload Manager, PBS Professional, Torque (software), and LSF (software), and interoperates with federation initiatives like XSEDE and projects hosted by CERN. Singularity images are typically single-file containers that can be built from recipes referencing base distributions like Ubuntu, CentOS, Debian, or specialized stacks such as TensorFlow and PyTorch. The design philosophy prioritizes reproducibility promoted by initiatives including the Reproducible Builds movement and workflows used in collaborations like Human Genome Project and Large Hadron Collider analyses.

Architecture and Components

Singularity's architecture centers on a runtime binary implemented in Go (programming language), an image format that encapsulates a filesystem, and integration layers for resource managers such as Slurm Workload Manager and Apache Mesos. Core components include the Singularity runtime, image builder utilities, and a client CLI that interacts with registries like Singularity Hub and OCI-compliant registries used by Docker Hub and GitLab. The image format supports squashfs and ext3/4 backends, and can encapsulate environments built from base images derived from distributions like CentOS Stream and Alpine Linux. For orchestration, Singularity integrates with container networking projects such as CNI (Container Network Interface) and storage technologies like Ceph and Lustre used at national supercomputing centers like NERSC. Security features map to Linux kernel facilities including namespaces, cgroups, and SELinux profiles used on systems at Los Alamos National Laboratory.

Usage and Workflow

Typical workflows involve building an image from a definition file referencing package sources such as Debian, CRAN (R Project), Bioconductor, or PyPI for Python packages, then running it on compute nodes scheduled by Slurm Workload Manager, launched via portals like Open OnDemand or workflow systems like Nextflow and Snakemake. Scientists integrate Singularity containers with continuous integration services such as GitHub Actions, GitLab CI/CD, and Jenkins (software) to automate builds for projects hosted on platforms like GitHub and GitLab. For cloud deployments, Singularity can operate on instances provisioned through providers like Amazon Web Services, Google Cloud Platform, and Microsoft Azure, and interoperate with identity and access systems such as CILogon and federation frameworks used by InCommon.

Security and Isolation

Singularity emphasizes non-privileged execution: containers typically run with the invoking user's UID and GID, aligning with policies at organizations like NERSC and Oak Ridge National Laboratory. The model mitigates risks present in privileged container runtimes used with Docker (software) by reducing need for setuid daemons and by leveraging kernel features maintained by projects such as the Linux kernel. Additional controls include support for AppArmor, SELinux, and integration with site-specific authentication systems like LDAP and Kerberos (protocol). Singularity's image signing and provenance mechanisms interoperate with trust frameworks such as The Update Framework and supply-chain initiatives advanced by institutions like GitHub and Google.

Performance and Compatibility

Designed for HPC workloads, Singularity minimizes overhead by using native kernel features rather than nested virtualization used by platforms like VMware ESXi or KVM (kernel virtual machine). Benchmarks on infrastructures at Argonne National Laboratory, Oak Ridge National Laboratory, and National Energy Research Scientific Computing Center demonstrate near-native I/O and compute performance for MPI-enabled applications using libraries such as OpenMPI, MPICH, and accelerators managed by NVIDIA drivers and CUDA. Compatibility layers enable use of container images from Docker (software), OCI registries, and packaging systems like RPM and Debian package repositories, facilitating reproducible pipelines in collaborations akin to ENCODE and TCGA.

Development, Community, and Governance

Development of Singularity has involved academic labs, national labs, and commercial entities such as Sylabs and community contributors from projects hosted on GitHub. Governance and community engagement occur through forums, working groups, and conferences like SC (conference), XSEDE Annual Conference, OpenRepro workshops, and meetings of organizations including The Linux Foundation and Open Container Initiative. Upstream contributions and interoperability efforts connect Singularity to ecosystems including OCI (Open Container Initiative), CNI (Container Network Interface), and packaging efforts like Conda (package manager) and Spack, while training and documentation are disseminated at institutions such as University of California, Berkeley, Stanford University, and Massachusetts Institute of Technology.

Category:Containerization