HEPcloud

HEPcloud
Name	HEPcloud
Established	2016
Primary site	Fermilab
Field	High-energy physics, distributed computing
Related projects	Open Science Grid, Worldwide LHC Computing Grid, CERN

HEPcloud HEPcloud is a Fermilab-led computing facility and paradigm designed to integrate heterogeneous computing resources for high-energy physics experiments and collaborations. The project connects on-premises clusters, national laboratories, cloud providers, and high-performance computing centers to support data-intensive workflows from experiments such as ATLAS, CMS, NOvA, and DUNE. HEPcloud enables dynamic provisioning, elastic scaling, and federated access to compute and storage resources across a broad ecosystem including commercial clouds and academic supercomputing centers.

Overview

HEPcloud was initiated at Fermilab with collaboration from Argonne National Laboratory, Brookhaven National Laboratory, Lawrence Berkeley National Laboratory, and SLAC National Accelerator Laboratory. The initiative aligns with strategies from Department of Energy programs and complements infrastructures like the Open Science Grid, Worldwide LHC Computing Grid, and XSEDE. Early pilots involved partnerships with Amazon Web Services, Google Cloud Platform, Microsoft Azure, and supercomputers such as NERSC and Oak Ridge Leadership Computing Facility. Use cases span experiments including ATLAS (experiment), CMS (experiment), NOvA, DUNE (experiment), MicroBooNE, and MINERvA. Governance and funding intersect with programs at DOE Office of Science and collaborations with institutions like University of Chicago, Rutgers University, and University of Wisconsin–Madison.

Architecture and Components

The HEPcloud architecture integrates workload management systems such as HTCondor, PanDA (Production and Distributed Analysis), and GlideinWMS with cloud APIs from Amazon EC2, Google Compute Engine, and Microsoft Azure Virtual Machines. Storage interfaces include CERN EOS, dCache, and Ceph, while data transfer relies on tools like Globus and FTS (File Transfer Service). Authentication and identity management are implemented with standards from CILogon and OAuth 2.0 alongside certificate infrastructures used at CERN and FNAL. Monitoring and telemetry incorporate solutions such as Prometheus, Grafana, and Elastic Stack (Elasticsearch, Logstash, Kibana). Containerization and software distribution leverage Docker (software), Singularity (software), and CernVM-FS, while workflow orchestration connects to platforms like Kubernetes and Apache Airflow. Interoperability with federated projects involves the Science DMZ model and network services like ESnet and Internet2.

Operations and Resource Provisioning

Operational practice in HEPcloud emphasizes dynamic resource provisioning from providers including Amazon Web Services, Google Cloud Platform, Microsoft Azure, national centers such as NERSC, ALCF, and OLCF, and institutional clusters at Fermilab and Brookhaven National Laboratory. Resource scheduling integrates spot and preemptible instances from commercial vendors and backfills on supercomputers via batch systems like Slurm and PBS Professional. Allocation and accounting interact with policies from DOE Office of Science and collaborative frameworks like Open Science Grid allocations. Orchestration tools implement autoscaling, job submission routing, and pilot factories using systems such as HTCondor and GlideinWMS. Operational security and incident response coordinate with CERT Coordination Center practices and institutional Computer Security Incident Response Teams at Fermilab and partner labs.

Use Cases and Impact

HEPcloud broadens accessible computing for experiments including ATLAS (experiment), CMS (experiment), LHCb experiment workflows, neutrino programs like DUNE (experiment) and NOvA, and astrophysics projects such as LSST simulations. It has enabled peak-demand processing for large-scale productions, Monte Carlo campaigns, and reconstruction tasks tied to releases from CERN and detector upgrades at Fermilab. Collaborations with industry partners such as Google and Amazon (company) have accelerated adoption of cloud-native practices. The model influenced planning at international facilities including CERN, DESY, TRIUMF, and KEK, and informed grant proposals to agencies like National Science Foundation and European Research Council.

Performance and Cost Management

Performance tuning in HEPcloud addresses throughput, latency, and data locality across providers such as Amazon EC2, Google Compute Engine, and Microsoft Azure Virtual Machines as well as supercomputers at Oak Ridge National Laboratory and Lawrence Livermore National Laboratory. Cost management employs spot-instance strategies, reserved capacity comparisons, and optimization tools inspired by commercial practices at Amazon Web Services and Google Cloud Platform. Benchmarking uses standard workloads from ATLAS (experiment) and CMS (experiment) and profiling via Prometheus and Grafana. Economic analysis ties into funding cycles at Department of Energy and cost-recovery models used by centers like NERSC and ALCF.

Security and Compliance

Security for HEPcloud aligns with practices at Fermilab, DOE Office of Science, and international partners such as CERN and European Organization for Nuclear Research. Identity federation uses CILogon and certificate systems compatible with Grid Security Infrastructure, while data governance follows policies influenced by DOE O 205.1B and institutional directives at Brookhaven National Laboratory and Argonne National Laboratory. Compliance and auditing incorporate standards from NIST, incident coordination with US-CERT, and data protection measures consistent with partner agreements at Amazon (company) and Google. Operational security testing and vulnerability management mirror procedures from CERT Coordination Center and national laboratory cybersecurity programs.

Category:High-energy physics Category:Distributed computing systems