Apache Airavata

Apache Airavata
Name	Apache Airavata
Developer	Apache Software Foundation
Released	2011
Programming language	Java, Python, JavaScript
Operating system	Cross-platform
License	Apache License 2.0

Apache Airavata is an open-source orchestration framework for composing, managing, executing, and monitoring large-scale computational workflows on distributed resources. It provides services and APIs that enable researchers, institutions, and platforms to run science gateways, integrate high-performance computing resources, and coordinate complex pipelines across cloud, cluster, and supercomputing environments. Airavata is used by projects spanning computational chemistry, bioinformatics, climate modeling, and astronomy.

Overview

Airavata originated from collaborations among research organizations and was contributed to the Apache Software Foundation ecosystem to provide a community-governed platform for workflow orchestration. The project targets use cases requiring integration with XSEDE, European Grid Infrastructure, Open Science Grid, and cloud providers such as Amazon Web Services and Google Cloud Platform. It interoperates with workflow systems and resource managers including HTCondor, SLURM, Globus Toolkit, and PBS Professional. Airavata complements science gateway projects like Science Gateways Community Institute, Galaxy (project), and WS-PGRADE.

Architecture

The architecture separates presentation, orchestration, and execution layers using service-oriented principles common to projects like Apache Hadoop and Apache Kafka. Core components include a registry and metadata store, an orchestration engine, a messaging layer, and an execution manager that dispatches tasks to compute resources such as TeraGrid allocations, national lab supercomputers like Argonne National Laboratory, and cloud clusters hosted on Microsoft Azure. It uses middleware patterns familiar from Globus and incorporates job submission adapters similar to integrations with UNICORE and HTCondor. Authentication and authorization can integrate with identity providers including OAuth 2.0, Shibboleth, and Kerberos.

Features and Components

Airavata provides an SDK and APIs for workflow composition comparable to interfaces in Pegasus (workflow management), Kepler (software), and Apache Airflow, while targeting high-performance science workloads like those run by Los Alamos National Laboratory and Lawrence Berkeley National Laboratory. Components include the Registry, Workflow Engine, Experiment Catalog, and Application Descriptor framework that define executable tasks similar to entries in Common Workflow Language. The platform supports language bindings including Python (programming language), Java (programming language), and JavaScript for web clients, and integrates with data transfer services such as GridFTP and RSync. Monitoring and provenance follow practices used by PROV (W3C), Research Data Alliance, and DataONE.

Deployment and Integration

Deployments of Airavata have been performed at academic institutions like University of Chicago, Indiana University, and University of Southern California, and integrated into national infrastructure projects funded by agencies such as the National Science Foundation and Department of Energy (United States). Integration patterns often link Airavata with portal frameworks such as Liferay and Django-based gateways, and with scheduling systems including Torque and SLURM. Containers and orchestration tools like Docker and Kubernetes are increasingly used to package Airavata services, while CI/CD pipelines rely on systems like Jenkins and Travis CI in research software deployments.

Use Cases and Applications

Airavata is applied to domains that require coordinated multi-step computation, including computational chemistry workflows used in projects with Cambridge Crystallographic Data Centre, genomics pipelines aligned with National Center for Biotechnology Information, climate simulation suites similar to those run by NOAA, and astronomy data processing akin to work at European Southern Observatory. It supports interactive science gateways for citizen science initiatives like Zooniverse and collaborative platforms used by research consortia such as ELIXIR and EuroHPC. Projects integrate Airavata to provide reproducible experiments for journals and repositories affiliated with PLOS and arXiv.

Development and Community

The project development is coordinated through the Apache Software Foundation governance model and involves contributors from universities, national laboratories, and industry partners including IBM and Microsoft Research. Development discussions occur on mailing lists and issue trackers similar to workflows in other ASF projects like Apache Spark and Apache Cassandra. The community engages at conferences and workshops such as Supercomputing Conference (SC), PEARC, and EGI Conference, and collaborates with standards bodies like the Open Grid Forum and the W3C.

Security and Performance

Airavata supports secure credential management and delegation mechanisms that align with practices in Globus Auth and integrate with identity federations such as InCommon and eduGAIN. It implements role-based access control patterns used by projects like OpenStack and can leverage encryption and secure data transport methods endorsed by NIST. Performance optimization strategies follow profiling and scalability techniques from MPI-based applications and large-scale systems research exemplified by work at Oak Ridge National Laboratory and Lawrence Livermore National Laboratory, including horizontal scaling with Apache Cassandra-style registries and asynchronous messaging patterns akin to Apache Kafka.

Category:Workflow management systems