Grid Engine

Grid Engine
Name	Grid Engine
Developer	Sun Microsystems; Oracle Corporation; Univa; open-source communities
Released	1990s
Latest release	varies by fork
Programming language	C, C++
Operating system	Unix-like, Linux
License	originally proprietary; later open-source and commercial variants

Grid Engine is a distributed resource management system originally developed to coordinate batch job scheduling across clusters of computers. It provides mechanisms for job submission, queuing, resource allocation, and usage accounting to enable high-throughput computing across heterogeneous nodes. The software evolved through corporate stewardship and community forks, becoming a foundational technology for academic research centers, enterprise datacenters, and cloud-integrated HPC environments.

History

Development began in the early 1990s at a commercial research organization and academic collaborations influenced by projects such as Sun Microsystems research groups and university computing centers. The product was commercialized and acquired by Sun Microsystems in the early 2000s, integrating with initiatives like the N1 Grid and aligning with enterprise strategies including Solaris and networked computing. After a change in corporate ownership linked to Oracle Corporation, stewardship shifted and spurred the emergence of independent projects and companies such as Univa and open-source communities. These forks and continuations intersected with work at institutions like Lawrence Livermore National Laboratory, Argonne National Laboratory, and various European research centers. The changing governance produced both proprietary releases and permissively licensed derivatives, prompting adoption by organizations anchored by European Grid Infrastructure and national research infrastructures.

Architecture and Components

The system implements a master/worker architecture influenced by distributed computing paradigms used at IBM research labs and in projects like Condor and Portable Batch System. Core components include the master daemon that maintains job queues, execution daemons on compute nodes, and a scheduler that enforces policy and priorities. Authentication and authorization commonly integrate with standards and services from Kerberos deployments, LDAP directories, and host-level access controls found in Red Hat Enterprise Linux or Ubuntu. Storage integration leverages networked filesystems such as NFS and parallel filesystems used at Oak Ridge National Laboratory installations. Monitoring and accounting interfaces can be tied to enterprise suites from Nagios or Prometheus and reporting systems used by agencies like National Science Foundation-funded facilities.

Job Submission and Scheduling

Users submit batch and array jobs using command-line clients and resource description syntax influenced by POSIX and shell environments popularized by GNU Project tools and Bash. The scheduler implements backfilling, priority, fair-share, and reservation policies similar to those described in literature from ACM and IEEE conferences on high-performance computing. Job dependencies and complex workflows often integrate with workflow managers from projects like Apache Airflow, Snakemake, or scientific workflow systems used at CERN. Resource attributes such as CPU, memory, GPU, and network topology are expressed to match hardware profiles deployed at centers like Lawrence Berkeley National Laboratory and Fermilab.

Administration and Configuration

Administrators manage queues, host groups, user limits, and scheduling policies through command-line tools and configuration files analogous to system administration practices at Red Hat and Debian-based sites. Security and auditing reference guidelines from National Institute of Standards and Technology and operational playbooks used by European Organization for Nuclear Research support teams. Integration with provisioning systems such as Ansible, Puppet, and Chef is common for cluster lifecycle management; orchestration can be combined with virtualization platforms from VMware or container technologies like Docker and Kubernetes for hybrid clouds.

Implementations and Forks

Multiple commercial and open-source implementations emerged after corporate transitions, including offerings from Univa, community distributions maintained by contributors with ties to projects hosted on platforms like GitHub, and commercial support packages from vendors serving national labs such as Sandia National Laboratories partners. These implementations differ in licensing, features for GPU scheduling targeting NVIDIA accelerators, and integrations with cloud providers such as Amazon Web Services and Google Cloud Platform.

Use Cases and Applications

Typical deployments serve computational workloads in domains championed by institutions like Massachusetts Institute of Technology and Stanford University: bioinformatics pipelines, finite-element analysis in engineering groups at Siemens and Boeing, seismic processing in energy companies like Schlumberger, and Monte Carlo simulations used at financial firms such as Goldman Sachs. Scientific collaborations at observatories like European Southern Observatory and particle physics experiments at CERN have used batch scheduling tools in large compute farms. Enterprise analytic workloads and rendering tasks at studios influenced by workflows from Pixar also map to job arrays and resource controls offered by these systems.

Performance and Scalability

Performance characteristics depend on scheduler algorithms, database backends, and network fabrics such as InfiniBand or Ethernet fabrics deployed in clusters at Argonne National Laboratory and Oak Ridge National Laboratory. Scalability studies often reference benchmarks and analyses presented at SC (conference) and IEEE International Parallel and Distributed Processing Symposium. High-throughput use cases require tuning of job dispatch rates, queue hierarchy, and accounting to sustain tens of thousands of concurrent tasks across tens of thousands of cores, mirroring deployments at national supercomputing centers. Integrations with parallel filesystems like Lustre and job-centric telemetry systems improve I/O-bound workload performance in production environments.

Category:Batch scheduling systems