ARC (Advanced Resource Connector)

ARC (Advanced Resource Connector)
Name	ARC (Advanced Resource Connector)
Author	NorduGrid
Developer	NorduGrid, European Grid Infrastructure
Released	2004
Latest release	ARC 5
Operating system	Linux, Unix-like
Programming language	C++, Python
License	Apache License

ARC (Advanced Resource Connector) is a distributed computing middleware system designed to enable heterogeneous high-throughput computing across clusters, grids, and clouds. It provides job submission, resource discovery, data management, and security integrating with scientific infrastructures such as European Grid Infrastructure, Open Grid Forum, Globus Toolkit, gLite, and HTCondor. ARC has been used in projects involving CERN, Nordic e-Infrastructure Collaboration, ELIXIR, EGI, and PRACE.

Overview

ARC is middleware that connects compute resources and data services across administrative domains, enabling workflows for projects like Large Hadron Collider, ALICE (A Large Ion Collider Experiment), ATLAS experiment, CMS experiment, LHCb experiment. It interoperates with standards from Open Grid Forum, authentication systems like X.509, and identity federations such as eduGAIN, supporting access models employed by CERN OpenStack, Amazon Web Services, and Google Cloud Platform in hybrid scenarios. The project has been spearheaded by institutions including NorduGrid, University of Oslo, Uppsala University, and KTH Royal Institute of Technology with funding from programmes like European Commission Horizon 2020, FP7, and Nordic e-Infrastructure Collaboration.

Architecture and Components

ARC's architecture comprises a set of modular services and clients including a meta-scheduler, information system, data management tools, and execution service. Core components integrate with systems such as LDAP, HTCondor, SLURM Workload Manager, PBS Professional, Sun Grid Engine, and Torque (software), enabling interoperability with resource managers used at sites like CERN, Fermilab, DESY, Lawrence Livermore National Laboratory, and Oak Ridge National Laboratory. Data movement uses protocols and services compatible with GridFTP, HTTP, WebDAV, and SRM while security leverages PKI, X.509 Certificates, and services like VOMS and Globus Security Infrastructure. The ARC client stack includes command-line tools, a RESTful interface, and APIs in Python and C++, facilitating integration with science platforms such as Jupyter, Galaxy (bioinformatics platform), Nextflow, and Apache Airflow.

Supported Platforms and Standards

ARC supports Unix-like operating systems deployed on hardware from vendors including Dell Technologies, Hewlett Packard Enterprise, IBM, and cloud providers like Amazon Web Services, Google Cloud Platform, Microsoft Azure. It implements standards from Open Grid Forum such as the Job Submission Description Language, information models like GLUE Schema, and storage interfaces referenced by European Data Infrastructure initiatives. Interoperability extends to identity and access frameworks like eduGAIN, Shibboleth, and certificate authorities listed by TERENA. ARC can interoperate with middleware stacks including Globus Toolkit, UNICORE, gLite, and workload managers like Slurm used by PRACE sites.

Use Cases and Deployment

ARC has been applied in high-energy physics workflows for CERN experiments, bioinformatics pipelines at ELIXIR nodes, climate modelling projects involving ECMWF, and astronomy collaborations with ESO and SKA. Deployments range from campus clusters at University of Bergen to national e-infrastructure sites like NorduGrid partners, and pan-European infrastructures under EGI. Use cases include large-scale Monte Carlo campaigns, data-intensive analysis for ATLAS experiment, batch processing for Human Genome Project-related workflows, and containerized workloads with Docker and Singularity. Integration with workflow systems has enabled pipelines referenced by projects such as Gaia, LOFAR, and Copernicus Programme.

Development, Governance, and Community

The project is coordinated by organizations including NorduGrid, European Grid Infrastructure, and research groups at Uppsala University and University of Oslo. Development follows open-source practices under the Apache License, with contributions from academic and national research organizations such as CSC — IT Center for Science, Forschungszentrum Jülich, CESNET, and SURFsara. Governance involves steering from stakeholders in initiatives like Horizon 2020, collaborations with standards bodies including Open Grid Forum, and partnerships with operational teams at CERN and EGI. The community engages via mailing lists, code repositories, and events such as Supercomputing Conference, EGI Conference, CHEP (Computing in High Energy Physics), and regional workshops.

Security and Performance

ARC emphasizes secure execution using mechanisms from X.509, VOMS, and Public Key Infrastructure deployed at sites like CERN and national CA networks linked to eduGAIN. Data integrity and confidentiality are enforced with transfer protocols like GridFTP and authentication backends including Kerberos in some integrations. Performance tuning has addressed scalability across federated topologies involving thousands of cores at centres such as CSC, PRACE Tier-0 facilities, and national centres like NorduGrid partners. Monitoring and accounting tie into systems such as Nagios, Prometheus, and Elastic Stack for observability in production grids.

History and Releases

ARC originated from the NorduGrid project in the early 2000s to serve Large Hadron Collider computing needs, with early deployments coordinated with CERN and national e-infrastructures. Major releases aligned with interoperability milestones, including ARC 1–5 and migration efforts to support modern cloud and container technologies influenced by initiatives like OpenNebula and Kubernetes. Releases have been announced at conferences such as CHEP and Supercomputing Conference, and coordinated with pan-European rollouts under EGI and research funding from European Commission programmes.

Category:Grid computing software