FTS3

FTS3
Name	FTS3
Developed by	Unknown
Initial release	Unknown
Latest release	Unknown
Repository	Unknown

FTS3 is a software system for reliable data transfer and workload scheduling used in high-energy physics and distributed computing environments. It integrates with middleware stacks and storage systems to coordinate bulk transfers and manage transfer queues between sites in research collaborations and grid infrastructures. The project interfaces with monitoring tools, database backends, and authentication services to support large-scale science initiatives.

Overview

FTS3 originated to support large collaborations such as CERN, European Organization for Nuclear Research, Large Hadron Collider, ATLAS (particle detector), CMS (detector), and wider Worldwide LHC Computing Grid deployments. It operates alongside services from Open Science Grid, National Science Foundation, Deutsche Forschungsgemeinschaft, Fermilab, and Brookhaven National Laboratory to move datasets for experiments like ALICE (A Large Ion Collider Experiment), LHCb, and multi-institution projects. The system often appears in workflows coordinated with HTCondor, Globus Toolkit, Apache Hadoop, Swift (openstack), and Ceph. FTS3 collaborations typically involve institutions such as University of Oxford, Princeton University, Lawrence Berkeley National Laboratory, Rutherford Appleton Laboratory, and regional infrastructures like GridPP and EGI.

Architecture

The architecture of FTS3 uses components typical of distributed service designs deployed on platforms like Red Hat Enterprise Linux, CentOS, Debian, and Ubuntu (operating system). Core elements interact with relational databases like PostgreSQL, MySQL, and SQLite and use messaging paradigms exemplified by RabbitMQ and ZeroMQ. Networking integrates with transfer protocols such as GridFTP, HTTP, and FTPS and storage systems including dCache, XRootD, Lustre (file system), and GPFS. Management and orchestration tools like Kubernetes, Docker, Ansible, and Puppet (software) are commonly employed for deployment and scaling across sites such as CERN Data Centre and national computing centers.

Features and Functionality

FTS3 provides features for queue management, retry logic, bandwidth throttling, and fault tolerance used by collaborations like ATLAS (particle detector), CMS (detector), and ALICE (A Large Ion Collider Experiment). It exposes APIs consumed by portals from Rucio, FTS (file transfer service), and Globus and integrates with monitoring stacks like Prometheus, Grafana, ELK Stack, and Nagios. Transfer auditing and logging connect to systems such as Splunk, Graylog, and CERN MONIT while job-scheduling policies echo designs in SLURM, Oracle Grid Engine, and HTCondor. Administrative interfaces are accessed through web frontends similar to OpenStack Horizon and often integrate with ticketing platforms like JIRA and ServiceNow.

Deployment and Use Cases

Typical deployments occur at scientific laboratories including CERN, Fermilab, Lawrence Berkeley National Laboratory, DESY, and regional centers such as National Center for Supercomputing Applications and STFC Rutherford Appleton Laboratory. Use cases cover dataset replication for experiments like Large Synoptic Survey Telescope workflows, multi-site analysis for LIGO Scientific Collaboration, distribution for space missions like Gaia (spacecraft), and archival transfers for projects at European Space Agency and NASA. Integration examples include data lifecycle tools like Rucio, provenance systems used by International Virtual Observatory Alliance, and workflows orchestrated with Pegasus (workflow management).

Performance and Scalability

Performance tuning in FTS3 deployments draws on techniques used by Apache Kafka, Ceph, HAProxy, and Nginx to maximize throughput and reduce latency in networks such as GÉANT, Internet2, and national research and education networks. Scalability strategies mirror architectures from Amazon Web Services, Google Cloud Platform, Microsoft Azure, and distributed compute fabrics like OpenStack. Large-scale demonstrations have been reported by collaborations including Worldwide LHC Computing Grid, Open Science Grid, and consortia of universities like University of Cambridge, University of Pennsylvania, and Imperial College London.

Security and Authentication

FTS3 supports authentication and authorization mechanisms compatible with infrastructures like X.509, Kerberos, and federated identity systems such as eduGAIN, OAuth 2.0, and OpenID Connect. Security integrations align with practices from European Union Agency for Cybersecurity, NIST, and institutional policies at centers like CERN and Brookhaven National Laboratory. Transport-layer protections rely on TLS suites deployed similarly to Let's Encrypt practices and certificate management workflows used by HashiCorp Vault and EJBCA in scientific grids.

Development and Community

Development and community contributions come from teams at CERN, Rutherford Appleton Laboratory, INFN, KIT (university), National Institute for Nuclear Physics, and collaborations across EGI, GridPP, and Open Science Grid. The project coordinates with standards bodies and projects including OGF (Open Grid Forum), WLCG, and software ecosystems seen in Debian, Red Hat, and OpenStack Foundation. Documentation, bug reports, and feature requests are typically tracked through platforms such as GitLab, GitHub, and issue trackers used by research computing groups at University of Chicago and Yale University.

Category:Distributed computing Category:High energy physics software