Condor Project

Condor Project
Name	Condor Project
Developer	University of Wisconsin–Madison; HTCondor Project contributors
Released	1988
Programming language	C (programming language), C++, Python (programming language)
Operating system	Unix-like, Linux, Microsoft Windows
License	GNU General Public License

Condor Project The Condor Project is a distributed computing system for high-throughput computing developed to harness idle computing resources across heterogeneous environments. It enables researchers and institutions to perform large-scale batch processing by aggregating desktops, servers, and clusters into a coherent compute pool. The project bridges resource providers and consumers across academic, industrial, and government environments, integrating with technologies and institutions such as Open Science Grid, National Science Foundation, CERN, NASA, and Los Alamos National Laboratory.

History

Condor originated at University of Wisconsin–Madison in the late 1980s as a response to idle cycles on workstation farms, influenced by earlier distributed systems research at Massachusetts Institute of Technology and Carnegie Mellon University. Early adopters included research centers like Lawrence Berkeley National Laboratory and Argonne National Laboratory, and the project evolved alongside initiatives such as TeraGrid and XSEDE. Over decades the project incorporated support for standards and middleware from Globus Toolkit, Sun Grid Engine, and Portable Batch System communities, while collaborating with organizations such as National Center for Supercomputing Applications and Oak Ridge National Laboratory.

Architecture and Components

Condor's architecture comprises daemons and services that implement resource brokering, matchmaking, and job lifecycle management. Core components include the central Scheduler (matchmaker), the Startd/Start daemon on execute machines, and the Collector/Negotiator pair for status aggregation and matchmaking; these interact similarly to systems like SLURM and Torque (software). The system supports integration with file transfer tools and data services such as GridFTP and Globus Online, and can interface with virtualization and container platforms including Docker (software), Singularity (software), and hypervisors supported by VMware, Inc..

Job Submission and Scheduling

Users submit jobs via command-line tools and submission description files that specify executable, arguments, and resource requirements, paralleling submission models from Portable Batch System and LSF (software). The negotiator matches jobs to resources using policies and ranking expressions, incorporating considerations similar to schedulers used by Google (company) and Amazon Web Services batch systems. Support for DAGMan enables dependency graphs comparable to workflow engines like Apache Airflow and HTCondor DAGMan-style orchestration, while integration with scientific workflow platforms such as Kepler (software) and Pegasus (workflow management) facilitates complex pipelines.

Resource Management and Policies

Condor implements fine-grained resource management including slot configuration, preemption, checkpointing, and backfill strategies used in environments like Lawrence Livermore National Laboratory and Fermilab. Policies for prioritization, fair-share, and accounting can be aligned with institutional frameworks from European Grid Infrastructure and funding agency requirements from Department of Energy (United States). Support for advance reservations and retirement policies allows coexistence with interactive use on desktops in deployments at universities such as Stanford University and University of California, Berkeley.

Security and Authentication

Security in Condor relies on mutual authentication, delegation, and encryption mechanisms compatible with tools like Kerberos, GSSAPI, and X.509. Integration with identity providers including LDAP, Active Directory, and federated services used by InCommon enables single sign-on and attribute-based access in multi-institution collaborations. The project has incorporated sandboxing techniques and support for secure containers to address concerns raised by organizations such as CERT and NIST.

Use Cases and Applications

Condor has been used for high-throughput workflows in domains including high-energy physics at CERN, bioinformatics at Broad Institute, climate modeling at National Center for Atmospheric Research, and astrophysics at Space Telescope Science Institute. It supports embarrassingly parallel tasks, parameter sweeps, Monte Carlo simulations used by groups at Brookhaven National Laboratory and SLAC National Accelerator Laboratory, and production pipelines in media rendering and financial analytics in institutions such as Pixar and Goldman Sachs where batch processing and opportunistic use of resources are essential.

Development and Community

Development is driven by academic labs and a global contributor community, with governance involving collaborators from institutions including University of Chicago, Columbia University, Purdue University, and companies participating in open-source HPC ecosystems. The project maintains documentation, mailing lists, and workshops that intersect conferences like Supercomputing (conference), Usenix, and International Conference for High Performance Computing, Networking, Storage and Analysis. Community efforts include integration plugins for cloud providers such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure, and partnerships with research infrastructures like Open Science Grid and European Grid Infrastructure.

Category:Distributed computing Category:High-throughput computing Category:Open-source software