HDF5

HDF5
Name	HDF5
Developer	National Center for Supercomputing Applications, The HDF Group
Released	1998
Operating system	Unix-like, Microsoft Windows, macOS
License	BSD license

HDF5 HDF5 is a file format and set of tools for storing and managing large, complex scientific data. It provides a hierarchical data model, compression, metadata, and parallel I/O capabilities used across research and industry. Prominent in high-performance computing, remote sensing, and astronomy, HDF5 interoperates with many software ecosystems and standards.

Overview

HDF5 organizes data into a tree-like structure of groups and datasets akin to the directory models used by UNIX, Windows NT, macOS file systems and draws conceptual parallels to Extensible Markup Language and NetCDF. Developers from institutions such as National Center for Supercomputing Applications and organizations including NASA and European Space Agency employ HDF5 alongside tools like MATLAB, Python, R, and Julia. Research projects at CERN, Los Alamos National Laboratory, and Lawrence Berkeley National Laboratory use HDF5 for simulation output, while industry adopters such as IBM, Intel, and NVIDIA integrate HDF5 into workflows involving MPI and CUDA. Standards and formats related to HDF5 include Open Geospatial Consortium specifications, OGC data models, and links to ISO 19115 metadata practices.

History and Development

HDF5 evolved from earlier efforts at the National Center for Supercomputing Applications in the 1990s as a successor to HDF4, influenced by community needs articulated in forums tied to DOE laboratories and collaborations with NASA missions like Terra (satellite) and Aqua (satellite). The HDF Group later assumed stewardship, collaborating with universities such as University of Illinois Urbana–Champaign and research centers like Argonne National Laboratory to develop APIs and parallel features compatible with MPI-IO and high-performance file systems such as Lustre and GPFS. Funding and contributions have come from agencies including the National Science Foundation and programs linked to European Commission initiatives.

File Format and Internal Structure

The HDF5 format stores metadata and raw arrays in a directed graph of objects: groups, datasets, datatypes, and attributes. Internally it uses B-tree-like structures similar to indexing strategies found in Berkeley DB and file-system techniques from XFS, employs chunked storage and compression filters comparable to approaches in zlib, and supports user-defined datatypes akin to constructs in POSIX. HDF5 files can include extensible metadata modeled after Dublin Core conventions and integrate with standards used by ISO datasets. The format supports features required by projects like Square Kilometre Array and observatories such as Hubble Space Telescope where efficient access to multi-dimensional arrays is critical.

APIs, Libraries, and Language Bindings

Official HDF5 libraries are written in C with [parallel] extensions for MPI and adapters for Fortran, while community bindings exist for Python via h5py and PyTables, for C++ via an official C++ API, for Java through native wrappers, and for MATLAB via built-in reader/writer support. Commercial and open-source projects such as Anaconda, SciPy, NumPy, and pandas integrate HDF5 through these bindings. Interoperability layers connect HDF5 to formats and tools like NetCDF, HDF-EOS, GDAL, and data portals used by NOAA and USGS.

Features and Use Cases

HDF5 supports large-scale multi-dimensional arrays, chunking, compression filters including LZF, zlib, and plugin filters, along with parallel I/O for workflows in HPC centers such as Oak Ridge National Laboratory and Lawrence Livermore National Laboratory. Use cases span observational data from MODIS, simulation outputs from GROMACS and LAMMPS, machine learning model checkpoints used with TensorFlow and PyTorch, and archival of datasets in repositories like Zenodo and institutional data archives at European Organization for Nuclear Research. HDF5 is applied in disciplines working with instruments from NOAA Hurricane Hunters to particle experiments at Fermilab, and in engineering contexts at companies like Boeing and Siemens.

Performance, Scalability, and Limitations

HDF5 scales effectively on parallel file systems when used with MPI-IO and tuned chunking strategies; performance considerations echo best practices from High Performance Computing communities and workshops at SC and ISC High Performance. Limitations include complexity of concurrent metadata writes, issues with small-file workloads reminiscent of challenges documented by Amazon Web Services and Google Cloud Platform users, and portability caveats across different versions and library builds similar to versioning concerns found in OpenSSL and SQLite. Community efforts and tools from The HDF Group and collaborators at Argonne National Laboratory aim to mitigate these constraints.

Adoption, Ecosystem, and Tools

A wide ecosystem surrounds HDF5: visualization tools like ParaView and VisIt, converters such as GDAL, data management systems used by NASA Earth Observing System Data and Information System and NOAA National Centers for Environmental Information, metadata editors in the spirit of Dublin Core implementations, and package distributions by Conda (package manager) and Debian. Training and documentation are provided by organizations including The HDF Group, workshops at AGU, and courses at universities like Massachusetts Institute of Technology and Stanford University. The community engages through conferences and collaborations with institutions such as IEEE, ACM, and funding agencies including NSF and DOE.

Category:File formats Category:Scientific data formats