European DataGrid

European DataGrid
Name	European DataGrid
Established	2001
Dissolved	2004
Location	Geneva, Switzerland
Funding	European Commission
Discipline	High Energy Physics; Bioinformatics; Astronomy

European DataGrid was a European Commission–funded research initiative to design and deploy a distributed computing infrastructure for large-scale data analysis. It aimed to integrate heterogeneous resources across research institutions to support projects in CERN, European Space Agency, and biomedical research centers. The project influenced subsequent initiatives such as EGEE, EUDAT, and the Worldwide LHC Computing Grid.

Overview

The project created a prototype of a production-quality distributed computing grid computing testbed bringing together institutions like CERN, European Organization for Nuclear Research (CERN), Fermi National Accelerator Laboratory, Lawrence Berkeley National Laboratory, and universities including University of Cambridge, University of Oxford, and Imperial College London. It addressed challenges encountered by large collaborations such as those in ATLAS, CMS, LHCb, and ALICE. Key goals included resource brokering, data management, authentication via X.509, and workload scheduling compatible with middleware standards developed by Global Grid Forum and collaborators like Globus Toolkit.

History and Development

The initiative was launched in 2001 following proposals from consortia involving institutions such as CERN, INFN, CNRS, NIKHEF, and industrial partners including IBM, HP, and Sun Microsystems. Early milestones included deployment of testbeds across sites in Geneva, Zurich, London, Paris, Milan, and Amsterdam. The project evolved through phases of prototype, integration, and validation, informing successor projects like Enabling Grids for E-sciencE (EGEE), gLite, and national initiatives in France, Italy, and United Kingdom. Influential workshops and conferences included meetings at CERN, the International Conference on High Energy Physics, and the Supercomputing Conference.

Architecture and Components

The architecture combined middleware components for data management, job submission, and information services; contributors included teams from INFN, CNRS, CERN IT, and commercial partners such as Sun Microsystems and HP. Core components included a Resource Broker influenced by the Globus Toolkit Resource Allocation Manager, a Replica Catalogue inspired by concepts used at Fermilab and Brookhaven National Laboratory, and a Logging and Bookkeeping service similar to systems at Lawrence Livermore National Laboratory. Security relied on X.509 certificates from Certification Authorities associated with organizations like TERENA and national research and education networks such as JANET, GARR, and RENATER. Monitoring and information services used protocols and tools comparable to Nagios and Ganglia, integrated with directory services like LDAP maintained by partner institutions.

Applications and Use Cases

Primary use cases were driven by high-energy physics experiments including ATLAS, CMS, ALICE, and LHCb for large-scale event reconstruction, simulation, and analysis. Other scientific domains that used the testbed included astronomy groups at European Southern Observatory, bioinformatics teams at European Molecular Biology Laboratory, and climate research groups collaborating with ECMWF and Met Office. Application examples involved distributed Monte Carlo production comparable to workflows at Fermilab and data-intensive pipelines similar to those used by Hubble Space Telescope archival science projects. The infrastructure supported collaborations spanning institutions like University of Edinburgh, Max Planck Society, University of Munich, and ETH Zurich.

Collaborations and Projects

The project assembled a broad partnership including national laboratories, academic institutions, and industry: CERN, INFN, CNRS, NIKHEF, CINECA, Fachhochschule, IBM, HP, and Sun Microsystems. It interfaced with standards and consortia such as the Global Grid Forum, OpenGrid Forum, and later contributed to EGEE and Open Science Grid dialogues. The DataGrid effort collaborated with domain projects like ATLAS, CMS, ESA science missions, Human Genome Project collaborators at European centers, and climate modelling groups associated with IPCC-linked researchers.

Impact and Legacy

Although the project formally ended in 2004, its technical results and organizational lessons shaped successor infrastructures including EGEE, EUDAT, and the Worldwide LHC Computing Grid. Concepts pioneered in the project—resource brokering, replica management, and certificate-based authentication—were adopted by institutions like CERN, Brookhaven National Laboratory, Fermilab, and national e-infrastructure providers across Spain, Germany, and Italy. The initiative influenced standards work in the Open Grid Forum and informed middleware such as gLite and commercial cloud offerings from vendors like Amazon Web Services and Google Cloud Platform through transfer of operational practice into production computing environments. The legacy persists in contemporary research infrastructures at CERN, national laboratories, and university computing centers.

Category:Grid computing Category:European Commission projects Category:CERN projects