European Middleware Initiative

European Middleware Initiative
Name	European Middleware Initiative
Type	Research collaboration
Founded	2010
Dissolved	2014
Headquarters	Europe
Region served	European Union
Focus	Distributed computing middleware for scientific computing

European Middleware Initiative

The European Middleware Initiative coordinated development and support of middleware for large-scale scientific computing across CERN, European Space Agency, European Commission, Deutsches Elektronen-Synchrotron, and major national laboratories. It provided integration, support, and maintenance for middleware stacks used by projects such as Worldwide LHC Computing Grid, Enabling Grids for E-sciencE, EGI-InSPIRE, XSEDE, and other research infrastructures, while engaging with standards bodies like Open Grid Forum and initiatives such as Horizon 2020. The initiative bridged software from developer communities including gLite, UNICORE, and ARC into a coordinated sustainable service for science collaborations such as ATLAS (particle detector), CMS (detector), and LIGO Scientific Collaboration.

Overview

The project aggregated middleware contributions from organisations including CERN, European Southern Observatory, National Institute for Nuclear Physics (Italy), Centre National de la Recherche Scientifique, INFN, CNRS, and CSIC. It targeted high-throughput computing users from experiments like ALICE (A Large Ion Collider Experiment), LHCb, and infrastructures such as EGI and PRACE. The stack encompassed job management components used by HTCondor, data services compatible with dCache and iRODS, security stacks interoperating with VOMS and Shibboleth, and information systems aligned to GLUE Schema and BDII. Partners included grid middleware projects gLite, ARC (Advanced Resource Connector), UNICORE, along with software produced by Fermi National Accelerator Laboratory, Brookhaven National Laboratory, Lawrence Berkeley National Laboratory, and academic groups from University of Edinburgh, Vrije Universiteit Amsterdam, and University of Manchester.

History and Development

The initiative was launched to consolidate outcomes from European projects such as EGEE (Enabling Grids for E-sciencE), EGI-InSPIRE, and regional efforts tied to NorduGrid. Its formation followed deliverables produced under FP7 (Framework Programme 7) and engagement with funding instruments like CIP (Competitiveness and Innovation Framework Programme). Development cycles coordinated releases influenced by collaborations with Ensemble, Géant, PRACE technical teams, and research experiments at CERN during Large Hadron Collider operation phases. Project milestones referenced community roadmaps negotiated at events such as TNC (TERENA Networking Conference), GridKa School, and workshops hosted by European Grid Infrastructure and national research networks including SURFnet and GARR (Italian research and academic network).

Architecture and Components

The middleware architecture integrated compute, storage, security, and information services. Compute components interfaced with batch systems like SLURM, PBS Professional, and Torque (software), and higher-level schedulers such as HTCondor and ARC Run-time Environment. Storage services supported dCache, StoRM, and federated file systems like Ceph and GPFS. Security and identity management relied on X.509, VOMS Role Management, and federated identity solutions like Shibboleth and eduGAIN. Information and monitoring used GLUE Schema, BDII, Nagios, and Ganglia. Data transfer and replication utilized protocols and tools such as GridFTP, FTS (File Transfer Service), Globus Toolkit, and Rucio; databases and metadata services employed MySQL, PostgreSQL, and Elasticsearch. Packaging and distribution integrated with systems like RPM (file format), Debian package, Open Build Service, and CERN CVMFS.

Governance and Funding

Governance involved consortium partners from national research centres, universities, and infrastructures with oversight from advisory boards composed of representatives from CERN, European Commission, INFN, DFG, and user experiments including ATLAS (particle detector) and CMS (detector). Funding combined European Commission grants under FP7 and partner in-kind contributions from institutions such as CNRS, INFN, CSC – IT Center for Science (Finland), KIT (Karlsruhe Institute of Technology), and national ministries like Ministry of Education and Research (Germany). Technical steering was coordinated through working groups patterned after governance in Open Grid Forum, with operational support from regional centres like GridPP and Nordugrid.

Deployment and Use Cases

Deployments supported production grids for experiments at CERN and astronomy projects including LOFAR and Square Kilometre Array pathfinders; bioinformatics pipelines used by European Bioinformatics Institute and climate simulations from ECMWF. Use cases included massive data processing for Large Hadron Collider, distributed analysis for Astrophysics Virtual Observatory, and federated data access for Human Brain Project simulations. The middleware enabled workflows orchestrated by tools such as Pegasus Workflow Management System, Taverna, and Nextflow, and was integrated into portals using Liferay and WS-PGRADE. Commercial and industry collaborations included partnerships with IBM, HP, Red Hat, and Oracle for enterprise-grade packaging and support.

Interoperability and Standards

Interoperability work aligned middleware with standards from Open Grid Forum, OASIS, IETF, and identity federations like eduGAIN. Schema compatibility adhered to GLUE Schema specifications; security conformance referenced X.509 and OAuth 2.0 where applicable. Data formats and exchange leveraged HDF5, NetCDF, and FITS for astronomy, with metadata standards from Dublin Core and domain repositories such as EMBL-EBI. Collaborations with cloud initiatives included compatibility testing against OpenStack, Amazon Web Services, and orchestration standards like OCCI and TOSCA.

Legacy and Successor Projects

After formal sunset, maintenance and components transitioned into successor activities within EGI, Open Science Grid, EUDAT, and efforts under Horizon 2020 and Horizon Europe. Technologies influenced modern research infrastructures including EOSC (European Open Science Cloud), Rucio-based data management in ATLAS (particle detector), and containerised deployments via Docker and Kubernetes. The community and codebases seeded initiatives such as INDIGO-DataCloud, NextGRID, and national projects like Grid Ireland and French National Grid Initiative (IFB); training and documentation practices persisted in schools like GridKa School and PRACE Training Centre.

Category:Middleware Category:Scientific computing