This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| The HDF Group | |
|---|---|
| Name | The HDF Group |
| Type | Nonprofit organization |
| Founded | 1998 |
| Headquarters | Albuquerque, New Mexico |
| Area served | Global |
| Products | HDF5, HDF4, HDFView, VOL, HSDS |
| Focus | Scientific data management |
The HDF Group The HDF Group is a nonprofit organization that develops and maintains data management technologies used in NASA, European Space Agency, CERN, Los Alamos National Laboratory, and Lawrence Berkeley National Laboratory. It provides core libraries and tools adopted by projects such as NOAA, USGS, DOE facilities, Google, and IBM for large-scale scientific data. The organization collaborates with institutions including University of Illinois Urbana–Champaign, Stanford University, MIT, and Oak Ridge National Laboratory to support interoperable file formats and community standards.
The organization was founded in 1998 to steward the HDF5 and HDF4 formats originally developed at National Center for Supercomputing Applications and University of Illinois Urbana–Champaign, during an era of growth in projects like Human Genome Project, Large Hadron Collider, Hubble Space Telescope, and Chandra X-ray Observatory. Early collaborations involved agencies such as NASA Goddard Space Flight Center, NOAA National Centers for Environmental Information, and US Geological Survey, while later work intersected with initiatives at Argonne National Laboratory, Sandia National Laboratories, and Pacific Northwest National Laboratory. Over time the group expanded tool support across platforms used by Microsoft Research, Apple Inc., Oracle Corporation, and Amazon Web Services.
The mission centers on sustaining the Hierarchical Data Format family for scientific computing communities including researchers at Harvard University, Yale University, Caltech, Princeton University, and engineers at Intel Corporation, NVIDIA, AMD, and Cray Inc.. Organizational governance has involved board members and contributors from institutions such as University of California, Berkeley, University of Washington, Imperial College London, and ETH Zurich. Operational units coordinate software engineering, user support, outreach to projects like Square Kilometre Array, James Webb Space Telescope, ALMA, and educational activities with National Science Foundation-funded programs and academic consortia.
Core technologies include the HDF5 file format and the earlier HDF4, supported by libraries and tools such as HDFView, hsds (HDF Server), and plugins for MATLAB, Python (programming language), R (programming language), and Fortran. Integration and interfaces exist for platforms like TensorFlow, PyTorch, Apache Hadoop, Apache Spark, and Kubernetes clusters used by European Organization for Nuclear Research and National Center for Atmospheric Research. The group maintains compatibility with community efforts including NetCDF, FIT S, GeoTIFF, and data models from Open Geospatial Consortium and DynamoDB-backed services provided by Amazon Web Services.
Projects leveraging HDF technologies span disciplines: climate science via IPCC modeling and CMIP datasets, astrophysics within Sloan Digital Sky Survey and Gaia (spacecraft), genomics in 1000 Genomes Project pipelines, and earth observation by Landsat, Sentinel-2, and MODIS. High-performance computing applications appear in simulations at Blue Waters, Titan (supercomputer), and Summit (supercomputer), while visualization and analysis use tools from ParaView, VisIt, Matplotlib, and Enthought. Industrial adopters include General Electric, Boeing, Ford Motor Company, and Siemens for engineering data workflows.
Community engagement includes workshops at venues such as International Supercomputing Conference, AGU Fall Meeting, American Geophysical Union, and SciPy conferences, with code contributions from organizations like GitHub, Bitbucket, and academic labs at University of Chicago. Collaboration partners and standards organizations include Open Source Initiative, Linux Foundation, World Wide Web Consortium, and research alliances such as ELIXIR and European Grid Infrastructure. Training and documentation efforts are delivered in partnership with Coursera, IEEE, ACM, and university extension programs.
Funding sources combine grants and contracts from agencies like National Science Foundation, Department of Energy, NASA, and European Commission programs including Horizon 2020, alongside corporate partnerships with Microsoft, Google, Amazon, and consortia such as CERN OpenLab. Cooperative agreements involve national laboratories including Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Oak Ridge National Laboratory, as well as research institutes like Scripps Institution of Oceanography and Max Planck Society.
The group's software and stewardship have influenced standards and received recognition in community citation and adoption metrics across projects like CMIP6, LIGO Scientific Collaboration, Square Kilometre Array Organisation, and multinational programs such as Intergovernmental Panel on Climate Change. Its technologies underpin datasets used by World Health Organization, United Nations Environment Programme, International Monetary Fund, and major publishers including Nature (journal), Science (journal), and IEEE Spectrum in reproducible research and data preservation efforts.
Category:Non-profit organizations based in New Mexico Category:Scientific software Category:Data management