NetCDF

NetCDF
Name	NetCDF
Caption	Data model and archive format for array-oriented scientific data
Developer	Unidata
Released	1989
Programming language	C, Fortran, Java, Python
Operating system	Cross-platform
License	UCAR

NetCDF is a software library and self-describing, machine-independent data format designed for array-oriented scientific data. It provides a data model, a set of file formats, and language APIs that enable storage, access, and sharing of multi-dimensional variables such as time series, gridded fields, and profiles. NetCDF is widely used in atmospheric science, oceanography, climate research, and remote sensing, and is integrated into many research workflows, data centers, and software stacks.

Overview

NetCDF defines a concise data model for typed, multi-dimensional arrays and associated metadata, enabling interoperability across computing platforms such as IBM, Intel-based systems, Apple hardware, Microsoft-powered clusters, and high-performance computing centers including Oak Ridge National Laboratory, Los Alamos National Laboratory, and National Center for Atmospheric Research. The NetCDF ecosystem includes libraries and tools maintained by organizations such as Unidata and UCAR that support languages like C, Fortran, Java, and Python. The format facilitates integration with community standards and infrastructures like CF conventions, OPeNDAP, THREDDS, and archives maintained by agencies such as NASA and NOAA.

History and development

NetCDF originated in 1989 at Unidata within UCAR to address needs at research centers including NOAA and university groups participating in programs like the World Climate Research Programme and the Global Ocean Data Assimilation Experiment. Early implementations targeted array access patterns used by projects such as Community Earth System Model development and observational programs at Scripps Institution of Oceanography. Over time, the format evolved through contributions from institutions including NCAR, ECMWF, NASA Goddard, and private firms. Major milestones include the introduction of the netCDF-4/HDF5-backed format that integrated capabilities from HDF5 and collaborations with developers of HDF Group. Governance and specification work have been influenced by community processes involving research programs such as CMIP.

Data model and file formats

NetCDF exposes a data model that separates dimensions, variables, and attributes to represent arrays, coordinates, and metadata used in observational campaigns like Argo and satellite missions such as MODIS aboard Terra and Aqua. The classic format (netCDF classic) and the 64-bit offset variant provide backward-compatible binary layouts; the netCDF-4 format reuses HDF5’s container to add features such as groups, compression, and user-defined types. NetCDF supports typed primitives (e.g., 32-bit float, 64-bit integer) and conventions for attributes used by communities including IPCC modelers and agencies like ESA. File variants are often controlled by metadata conventions such as CF and data packaging schemes used by projects like ESGF.

APIs and software implementations

Official NetCDF APIs exist for C, Fortran, Java, and Python through bindings and libraries maintained by Unidata and collaborating institutions. Implementations and related libraries include tools from HDF Group, integrations with scientific ecosystems like SciPy, xarray, and visualization systems such as NCL and Panoply. Data servers and middleware—examples being OPeNDAP servers and THREDDS—provide remote access to NetCDF datasets for users at organizations like JPL and ECMWF. Build systems and package managers such as Conda and Debian packaging support redistribution in research computing environments.

Applications and adoption

NetCDF is heavily adopted in climate and weather research communities for model output from systems like CESM and WRF, observational assimilation at facilities including NOAA/NCEP, and reanalysis projects such as ERA-Interim and ERA5. Oceanographic programs—GOOS, Argo, and marine biogeochemistry networks—use NetCDF for profile and gridded products. Remote sensing pipelines at institutions like NASA Goddard and ESA distribute sensor data in NetCDF or converted forms for downstream analysis by groups at MIT, Princeton University, and Imperial College London. Scientific publishing and data citation infrastructures, including archives at PANGAEA and national data centers, also archive NetCDF packages for reproducible research.

Performance and interoperability

NetCDF performance depends on format choice, I/O libraries, and storage systems deployed at supercomputing centers such as Argonne National Laboratory, Oak Ridge National Laboratory, and cloud platforms like Amazon Web Services and Google Cloud Platform. The HDF5-backed netCDF-4 enables chunking, compression (e.g., deflate), and parallel I/O via MPI-IO used in workflows run on clusters at NERSC and XSEDE resources. Interoperability is supported through conventions like CF and protocols such as OPeNDAP, enabling tools from GDAL to Matplotlib and visualization systems like ParaView and VisIt to consume NetCDF data. Community efforts—coordinated by organizations like Unidata and standards bodies within the WMO—continue to refine metadata practices and API evolution to meet demands from initiatives including CMIP and large observational campaigns.

Category:Scientific file formats