EDG (European DataGrid)

EDG (European DataGrid)
Name	European DataGrid
Abbreviation	EDG
Founded	2001
Dissolved	2004
Region	Europe
Type	Research project

EDG (European DataGrid) was a multi-institutional research initiative to develop a distributed computing infrastructure for large-scale scientific data processing. It aimed to provide a production-quality computing grid middleware and services to support data-intensive projects in high-energy physics, astronomy, and bioinformatics. The project brought together national laboratories, universities, and industry partners across Europe to prototype and deploy grid technologies for experiments such as those at CERN, the European Organization for Nuclear Research.

Introduction

The project sought to integrate heterogeneous resources across institutions like CERN, European Space Agency, IN2P3, RAL, INFN, and CNAF into a coherent fabric for data analysis. EDG focused on interoperability with initiatives such as Globus Toolkit, Open Grid Services Architecture, and early work that influenced OGSA specifications. Its remit included resource brokering, data management, secure authentication, and job scheduling to enable collaborations on experiments such as Large Hadron Collider efforts and observatory programs like ESO projects.

History and Development

EDG began in 2001 under funding and coordination mechanisms involving the European Commission and national research agencies. Core partners included CERN, CCLRC, INFN, NIKHEF, CERN IT Division, Fermilab (collaboration links), and universities like University of Oxford, University of Cambridge, Università di Napoli Federico II, and Imperial College London. During its lifecycle EDG produced numbered software releases, testbeds, and demonstrations at international events such as the Supercomputing Conference and meetings convened by eScience initiatives. EDG's later phases interoperated with successor programs including the EU DataGrid follow-ons and projects that contributed to the Worldwide LHC Computing Grid.

Architecture and Components

EDG's architecture comprised modular services for workload management, data replication, information services, and security. Principal elements included a Resource Broker influenced by the Globus Resource Allocation Manager, a Replica Catalog inspired by Berkeley DB-style indexing, a Grid Information Service interoperable with LDAP schemas adopted by IHEP and other institutes, and an Authentication and Authorization infrastructure using X.509 certificates issued by recognized Certification Authorities such as those used by ESRIN and national labs. Storage management interoperated with mass storage systems at CERN and Tier-1 centers, and job submission pathways integrated local batch systems like PBS, LSF, and Condor instances maintained at partner sites. The middleware stack was tested on production-class hardware from vendors including IBM, Sun Microsystems, and HP used by computing centers like SARA and CNAF.

Applications and Use Cases

EDG supported data-intensive analysis pipelines for collaborations in particle physics experiments feeding into ATLAS and CMS prototyping, astrophysics workflows associated with XMM-Newton and INTEGRAL datasets, and bioinformatics tasks within distributed collaborations involving institutes like EMBL and EBI. Use cases included large-scale Monte Carlo production runs leveraging computing farms at RAL, distributed event reconstruction pipelines coordinated with Tier-2 centers, and cross-border data replication strategies with partners such as PIC (Port d'Informació Científica) and GridPP. EDG was demonstrated in disaster recovery scenarios with data reallocation similar to protocols used by European Space Agency ground segments and in collaborative analyses akin to those performed in LIGO working groups.

Collaboration and Partnerships

The consortium model united research councils, national laboratories, and university groups across countries including the United Kingdom, France, Italy, Netherlands, Spain, and Switzerland. Industry partners provided commercial middleware stacks and hardware, with companies such as Microsoft (in adjacent interoperability work), Oracle Corporation for database services, and Cisco Systems for networking demonstrations participating in deployment dialogues. EDG coordinated with parallel initiatives like Grid Ireland, NorduGrid, INFN Grid Project, and international programs including Grid Canada and Open Science Grid to align standards. Training and outreach were conducted through workshops at institutions like CERN and conferences organized by IEEE Computer Society and ACM.

Legacy and Impact on Grid Computing

EDG contributed foundational software components, operational practices, and collaborative frameworks that influenced subsequent large-scale infrastructures such as the Worldwide LHC Computing Grid and European initiatives like EGI. Its work on resource brokerage, data replication, and security models fed into standards discussed at OASIS and influenced early cloud interoperability thinking relevant to later platforms from Amazon Web Services and enterprise grids. Technical personnel and organizational lessons from EDG migrated to successor projects including LCG and numerous national grid deployments; partner institutions carried forward middleware and operational experience into campus clusters and regional computing centers like GridPP and NorduGrid. The project remains cited in studies of distributed computing evolution alongside milestones such as the Globus Toolkit and the emergence of OpenStack-era designs.

Category:Grid computing projects Category:European research projects Category:Distributed computing