LSF (software)

LSF (software)
Name	LSF
Title	LSF (software)
Developer	IBM Spectrum LSF
Released	1992
Latest release version	IBM Spectrum LSF 10.x
Operating system	Unix-like, Linux, Microsoft Windows
License	Proprietary, commercial

LSF (software) LSF is a commercial workload management and job scheduling system for high-performance computing, high-throughput computing, and cluster management. It coordinates resource allocation, job queuing, and job execution across compute clusters and data centers, integrating with orchestration, storage, and authentication platforms. LSF is widely deployed in industries such as life sciences, finance, and oil and gas for batch processing, analytics, and simulation workloads.

Overview

LSF provides centralized scheduling, resource management, and job prioritization across distributed compute resources, enabling workloads from research institutions like Lawrence Berkeley National Laboratory, European Organization for Nuclear Research, and Los Alamos National Laboratory to share clusters. The platform interfaces with vendors and projects such as Intel Corporation, NVIDIA, Red Hat, Hewlett-Packard Enterprise, and Dell Technologies to support heterogeneous hardware. LSF integrates with workflow systems and middleware including Apache Airflow, Kubernetes, SLURM Workload Manager, HTCondor, and OpenStack to orchestrate pipelines and cloud bursting. Security and compliance integrations tie into LDAP, Microsoft Active Directory, and identity systems used by institutions like National Institutes of Health and European Bioinformatics Institute.

History and Development

LSF traces origins to research on distributed batch systems in the early 1990s at institutions connected with companies such as Platform Computing, which commercialized the software. Platform Computing later entered into partnerships and was acquired by IBM; IBM rebranded the product under its IBM Spectrum software family. The development timeline intersects with milestones in parallel and distributed computing represented by projects at Massachusetts Institute of Technology, Stanford University, and Lawrence Livermore National Laboratory. Key development phases paralleled advances in cluster computing, grid computing exemplified by Globus Toolkit, and cloud computing initiatives like Amazon Web Services and Google Cloud Platform.

Architecture and Components

LSF employs a multiprocess architecture with components including cluster management services, job schedulers, submission clients, and execution daemons. Core elements are the master host (LSF director), dispatcher, and execution hosts running agent daemons, comparable in role to components in Torque (software) and Grid Engine. The architecture supports resource discovery for CPUs, GPUs, memory, and licenses, interfacing with device vendors such as AMD, ARM Holdings, and IBM Power Systems. Storage and file-system integrations include Lustre (file system), Ceph, and NAS offerings from NetApp and EMC Corporation. Monitoring and logging work with observability stacks like Prometheus, Grafana, and Elastic NV products.

Features and Functionality

LSF provides advanced scheduling policies, fair-share algorithms, job arrays, array-based checkpoint/restart, preemption, and backfilling similar to features in systems used by Oak Ridge National Laboratory and Argonne National Laboratory. It supports heterogeneous resources, GPU scheduling for frameworks such as CUDA and TensorFlow, and affinity rules for NUMA-aware placement on platforms like Cray Inc. and Fujitsu. LSF includes native support for job submission via command-line clients, RESTful APIs for integration with tools like Jenkins and GitLab CI, and graphical dashboards akin to those from Hortonworks and Cloudera. Enterprise features include quota management, license-aware scheduling for commercial applications from vendors such as Schrodinger (company) and ANSYS, and audit capabilities for compliance with standards used by European Medicines Agency or Food and Drug Administration processes.

Deployment and Integration

LSF is deployed on-premises, in hybrid cloud, and as part of managed services, integrating with orchestration technologies like Kubernetes for containerized workloads and cloud platforms such as Microsoft Azure, Amazon Web Services, and Google Cloud Platform for elastic scaling. Integrations exist with workflow managers and science gateways employed by facilities like CERN and ELIXIR; connectors enable data staging with Globus, iRODS, and object storage systems from Amazon S3 and OpenStack Swift. Authentication and authorization integrate with Kerberos, OAuth 2.0, and enterprise directories used by University of California campuses and national laboratories. Deployment tools often leverage configuration management platforms such as Ansible, Puppet, and Chef.

Licensing and Editions

The product is offered under proprietary commercial licensing by IBM, with editions and entitlements aligned with enterprise support plans and feature tiers. Licensing models reflect node, core, or capacity-based metrics comparable to licensing options from vendors like Red Hat and Oracle Corporation. Academic and research institutions often procure enterprise agreements similar to arrangements with National Science Foundation-funded consortia or regional research infrastructures like XSEDE and PRACE.

Use Cases and Performance

LSF is used for large-scale simulations, genome sequencing pipelines, quantitative finance risk calculations, and computational fluid dynamics in sectors represented by organizations such as GlaxoSmithKline, Goldman Sachs, and ExxonMobil. Performance tuning leverages topology-aware scheduling, GPU affinity, and MPI-aware launches for libraries such as Open MPI and Intel MPI. Benchmarks reported by users mirror workloads run on supercomputers like Summit (supercomputer) and Fugaku, demonstrating scalability to thousands of nodes when paired with high-speed interconnects from Mellanox Technologies and Intel Omni-Path. Administrators combine LSF with telemetry platforms used by National Center for Supercomputing Applications and European Grid Infrastructure to measure throughput, latency, and utilization.

Category:Job scheduler software