This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| Zarr (file format) | |
|---|---|
| Name | Zarr |
| Extension | .zarr |
| Developer | NumFOCUS, Open Data community |
| Released | 2016 |
| Latest release | (varies by implementation) |
| License | BSD-like, permissive |
| Website | (project pages) |
Zarr (file format) Zarr is a chunked, compressed, N-dimensional array storage format designed for large-scale scientific data management and analysis. It targets workflows in domains such as atmospheric science, oceanography, astronomy, and genomics, enabling efficient storage, sharing, and parallel access across local filesystems, object stores, and cloud platforms. Zarr complements tools for data analysis and visualization by providing a flexible container for numerical arrays and associated metadata.
Zarr organizes numerical arrays into chunked blocks with compression metadata and attributes to support interoperable data exchange among projects such as NumPy, Dask (software), SciPy, xarray, Pandas, NetCDF, and HDF5. The format emphasizes simplicity and openness, aligning with principles advocated by OpenAI, Mozilla Foundation, NumFOCUS, The Linux Foundation, and communities around Python (programming language), R (programming language), and Julia (programming language). Zarr has been adopted by initiatives including European Space Agency, NASA, NOAA, Copernicus Programme, CEOS, and research groups at institutions like MIT, Stanford University, University of Oxford, and University of California, Berkeley.
The Zarr data model stores N-dimensional arrays with a schema for shape, dtype, order, and chunk layout, similar in concept to array storage used by HDF Group, OGC, and formats discussed at conferences such as SciPy Conference and AGU Fall Meeting. Arrays can include user attributes and support variable-length and complex datatypes used by projects like Bioinformatics groups at Broad Institute and instrumentation teams at CERN. Features include per-chunk compression codecs supported by libraries like Blosc, Zlib, LZ4, and Brotli, and metadata encoded in JSON with semantics compatible with CF (Climate and Forecast) metadata convention and semantic standards promoted by World Meteorological Organization and GEOSS members. Zarr’s model enables compatibility with analytical platforms such as Jupyter, Apache Spark, Hadoop, Kubernetes, and scientific gateways hosted by Amazon Web Services, Google Cloud Platform, and Microsoft Azure research programs.
Zarr defines a logical layout that maps array metadata and chunk files to a storage backend; implementations provide backends for POSIX, Amazon S3, Google Cloud Storage, Azure Blob Storage, OpenStack Swift, and networked filesystems used at facilities like CERN Data Centre and Oak Ridge National Laboratory. The storage abstraction allows use with object-store interfaces pioneered by S3 API adopters and with archival systems used by National Center for Atmospheric Research and European Organisation for Astronomical Research in the Southern Hemisphere. Zarr variants include consolidated metadata representations and support for hierarchical grouping analogous to groups in HDF5 (file format), while enabling simpler mapping to versioned storage systems used by GitHub, Dataverse, and data registries operated by Dryad and Zenodo.
Reference and third-party implementations exist across ecosystems: the reference Python library used by NumPy and SciPy users, community ports for Java, C++, Rust, and bindings integrated into projects such as xarray, Dask, rasterio, GDAL, Zarr-JS and visualization tools like Matplotlib, Bokeh, Holoviews, and Kepler.gl. Data platforms and archives such as Pangeo, Open Data Cube, SDSC, NCAR, NOAA Big Data Program and research hubs at Google Earth Engine and Amazon Open Data Registry provide tooling and examples. Governance and community collaboration occur through channels including NumFOCUS sponsorship, open standards discussions at W3C, Open Geospatial Consortium, and working groups aligned with Research Data Alliance.
Zarr’s chunked layout supports parallel reads and writes enabling scalable workflows on clusters managed by Slurm, Kubernetes, and high-performance computing centers like NERSC and Argonne National Laboratory. Combined with compression codecs such as Blosc and I/O schedulers used in xarray+Dask (software) stacks, Zarr can approach or exceed throughput comparable to HDF5 (file format) in distributed object-store scenarios. Performance considerations include chunk sizing informed by instrumentation teams at NOAA and ESA, caching strategies published by NIST and tuning for cloud egress models used by AWS, GCP, and Azure.
Zarr is used in diverse domains: climate and weather modeling at ECMWF and Met Office, satellite data processing for Copernicus Programme and NASA, radio astronomy pipelines at ALMA and SKA Organization, genomics workflows at Broad Institute and Wellcome Sanger Institute, and machine learning datasets consumed by projects at Google Research, DeepMind, and OpenAI. It supports reproducible research practices promoted by Center for Open Science, FAIR principles advocates, and archives like Zenodo and Figshare for dataset publication.
Zarr originated from community efforts to provide a simple, cloud-friendly array format in 2016, with stewardship by open-source contributors associated with NumFOCUS and collaborations among researchers from institutions like University of Washington, UC Berkeley, and industry partners including Anaconda, Inc. and cloud providers. Development progressed through community-driven proposals, implementations, and adoption in consortia such as Pangeo and standards discussions influenced by projects at HDF Group and the Open Geospatial Consortium. Ongoing evolution includes specification updates, expanded codec support, and ecosystem growth driven by scientific users and infrastructure organizations worldwide.
Category:File formats