DIRAC (program)
This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| DIRAC (program) | |
|---|---|
.png) | |
| Name | DIRAC |
| Developer | Science and Technology Facilities Council; CELIA (Laboratory) contributors |
| Released | 2000s |
| Programming language | Python (programming language), C++ |
| Operating system | Linux, Windows, macOS |
| Genre | Grid computing, High-performance computing |
| License | BSD license |
DIRAC (program) DIRAC is a distributed computing workload management system designed to coordinate large-scale computing cluster resources, data management services, and user authentication for scientific collaborations. It connects research institute infrastructures, facilitates resource allocation across heterogeneous sites, and integrates with high energy physics experiments, astronomy facilities, and multi-institutional projects. DIRAC provides job scheduling, data replication, monitoring, and workflow orchestration for distributed research efforts.
Overview
DIRAC implements a pilot-based workload management model influenced by Condor (software), Globus Toolkit, ARC (middleware), gLite, and UNICORE. It targets collaborations such as LHCb, Belle II, CTA (observatory), IceCube Neutrino Observatory, and federations of university clusters. DIRAC components include Workload Management System agents, Data Management System services, and a central Catalog coordinated by middleware brokers and authentication modules. It interoperates with OpenStack, Kubernetes, HTCondor, and cloud providers including Amazon Web Services, Google Cloud Platform, and Microsoft Azure.
History and Development
DIRAC originated within the High Energy Physics community to meet demands from the Large Hadron Collider era, drawing on experiences from CERN collaborations and regional grid projects. Early development teams included engineers from Institute of High Energy Physics (IHEP), Birmingham University, CNRS, and the European Organization for Nuclear Research. Funding and oversight came from national and supranational bodies such as the European Commission, Science and Technology Facilities Council, and national research councils. DIRAC evolved through versions aligning with milestones like LHC Run 1, LHC Run 2, and upgrades associated with Belle II commissioning, incorporating lessons from GridPP and EGI. Contributions and code reviews were coordinated via platforms used by Apache Software Foundation projects and by collaboration with Open Science Grid maintainers.
Features and Capabilities
DIRAC offers job brokerage, pilot submission, dynamic resource matching, and data lifecycle management integrated with Replica Catalog concepts and Storage Resource Manager interfaces. It supports authenticated access through X.509, OAuth 2.0, and federated identity via eduGAIN, enabling integration with European Open Science Cloud initiatives. Monitoring dashboards provide metrics interoperable with Prometheus (software), Grafana, and ELK Stack observability tools. Data transfer capabilities use protocols like GridFTP, FTS (File Transfer Service), HTTP, and S3 (storage service), and it implements throttling strategies inspired by Best Effort scheduling and Backfilling (scheduling). The system supports metadata indexing compatible with AMGA and Rucio catalogs and can enforce quotas, ticketing, and quality-of-service policies.
Architecture and Implementation
The DIRAC architecture is modular, comprising Services, Agents, and Clients linked by RPC and message queues similar to patterns in ZeroMQ and RabbitMQ. Core services include the Workload Management System, Data Management System, and Configuration services, while user-facing clients expose RESTful APIs and command-line tools compatible with Python Software Foundation packaging and Conda (package manager). Backend storage integrates with Ceph, dCache, and EOS (CERN) systems. Security components rely on TLS and federated credential translation approaches akin to those used by Shibboleth and Keycloak. The codebase incorporates test-driven development practices promoted by GitHub and GitLab workflows with continuous integration using Jenkins or GitLab CI pipelines.
Applications and Use Cases
Primary deployments include support for particle physics collaborations such as LHCb for Large Hadron Collider analyses, and for astrophysics instruments including Cherenkov Telescope Array prototype arrays and Vera C. Rubin Observatory preparatory studies. DIRAC enables Monte Carlo production campaigns, real-time reconstruction pipelines, and large-scale skimming and ntuplization operations for teams at CERN, KEK, DESY, and national laboratories like Fermilab and SLAC National Accelerator Laboratory. Beyond physics, DIRAC has been adapted for bioinformatics workflows in projects associated with EMBL-EBI and climate modeling initiatives linked to ECMWF and Met Office research groups. It also supports training and outreach through partnerships with Compute Canada, PRACE, and regional e-infrastructures.
Licensing and Distribution
DIRAC is distributed under permissive open-source terms compatible with BSD license and fosters contributions from academic institutions including Imperial College London, University of Barcelona, and National and Kapodistrian University of Athens. Packaging is available via PyPI and container images are published for registries compatible with Docker Hub and Harbor. Deployment blueprints use Ansible, Terraform, and Helm charts to target cloud and on-premises infrastructures maintained by organizations such as CERN OpenStack and national research networks like GÉANT.
Reception and Impact
DIRAC has been cited in technical reports and adopted by multiple international collaborations, influencing standards in distributed workload management alongside systems such as PanDA, ARC Compute Element, and HTCondor Federations. Its flexible pilot paradigm contributed to operational models used by Open Science Grid and informed design discussions at workshops organized by WLCG and IEEE conferences. Evaluations by projects at CERN and KEK highlight strengths in scalability and modularity, while comparative studies with Rucio and other data management frameworks underscore trade-offs in feature sets and integration costs. The ecosystem surrounding DIRAC continues to engage communities from high energy physics to astronomy through collaborative development and interoperability efforts.
Category:Grid computing software