Globus MDS

Globus MDS
Name	Globus MDS
Developer	Argonne National Laboratory; University of Chicago; Fermilab
Released	1990s
Programming language	C (programming language); Java (programming language)
Operating system	Unix; Linux; Microsoft Windows
Platform	x86; x86-64; ARM
License	BSD license; Apache License

Globus MDS is a metadata and resource discovery component originally developed within the Globus Toolkit ecosystem to support large-scale distributed computing. It provided directory services, information publication, and query interfaces that enabled resource discovery across heterogeneous infrastructures such as national laboratories, research universities, and international collaborations. MDS was integrated into production grids operated by organizations including National Science Foundation, Department of Energy (United States), and projects tied to Large Hadron Collider collaborations.

Overview

MDS acted as a distributed information service used by middleware stacks and client tools developed at Argonne National Laboratory, University of Chicago, and Fermilab to expose resource metadata for schedulers, data management systems, and monitoring dashboards. Early deployments connected compute clusters at Oak Ridge National Laboratory, Lawrence Berkeley National Laboratory, and Los Alamos National Laboratory with data stores at European Organization for Nuclear Research, supporting science communities such as High Energy Physics, Astrophysics, and Bioinformatics. The system interfaced with directory and schema standards influenced by Lightweight Directory Access Protocol and protocols used in projects like Globus Toolkit and Open Grid Services Architecture.

Architecture

MDS was architected as a hierarchical, federated set of information providers, aggregators, and directory services. Information providers were adapters that harvested state from local services such as batch schedulers (examples include Condor (software), PBS (software), Slurm Workload Manager) and storage systems like GPFS and HDFS (file system). Aggregation nodes collected provider outputs and offered indexed views consumed by higher-level directories and portals such as those used by XSEDE and PRACE. Core components interacted with service frameworks that resembled designs from Globus Toolkit and middleware stacks developed at NASA Ames Research Center and CERN IT Department.

Data Model and Protocols

MDS used schema-driven information models to represent resources, employing concepts and schema elements compatible with standards influenced by Lightweight Directory Access Protocol and early grid information models defined by Open Grid Forum. The data model captured attributes for compute elements, storage elements, and network endpoints, mapping to terms familiar to administrators of Torque (resource manager), LSF (software), and Netfilter. Protocol surfaces included LDAP-like directory queries and XML-encoded manifests; these enabled integration with monitoring systems like Nagios and visualization tools developed by European Grid Infrastructure. Interoperability patterns were guided by precedents set in projects such as GridFTP and Globus Online for data movement, and by service description practices from UDDI.

Deployment and Configuration

Deployment scenarios ranged from single-organization catalogs at Argonne National Laboratory to multi-institutional federations spanning European Organization for Nuclear Research and Brookhaven National Laboratory. Administrators configured information providers to poll local daemons, adaptors for schedulers including Torque (resource manager), and scripts for storage metadata. Configuration management practices commonly used tools influenced by Puppet (software), Ansible (software), and CFEngine. Scaling considerations mirrored those faced by large portals at XSEDE and PRACE, including replication, caching, and sharding strategies similar to systems deployed at National Energy Research Scientific Computing Center.

Security and Access Control

MDS deployments integrated security models compatible with grid authentication and authorization infrastructures such as those used by Globus Toolkit and Shibboleth. Access control was frequently layered atop certificate-based identification from Internet2 federations and service credentials issued by organizational certificate authorities like DOE NERSC and ESnet. Role-based and attribute-based controls referenced identity assertions employed in collaborations like OpenID and standards discussed at Internet Engineering Task Force. Integrations with firewalls and perimeter defense followed operational guidance from Department of Energy (United States) cybersecurity practices and incident response teams at facilities including Oak Ridge National Laboratory.

Use Cases and Applications

MDS supported resource brokering for metaschedulers used by projects at CERN, Fermilab, and Brookhaven National Laboratory; its catalogs enabled workload placement decisions for campaigns such as Large Hadron Collider data processing and ensemble simulations run at National Center for Atmospheric Research. Monitoring and portal applications integrated MDS outputs into dashboards employed by consortia like XSEDE and European Grid Infrastructure to visualize cluster health and job queues. Data management workflows for archives at Lawrence Berkeley National Laboratory and California Institute of Technology used MDS-derived metadata to orchestrate replication and staging across storage systems.

History and Development

Development began in the late 1990s as part of the broader Globus Toolkit effort led by research teams at Argonne National Laboratory and University of Chicago. Over successive releases MDS evolved alongside projects such as Globus Toolkit versions, responding to lessons from national initiatives funded by National Science Foundation and Department of Energy (United States). As cloud-native and RESTful paradigms emerged through influences like Amazon Web Services and Kubernetes, MDS capabilities were re-evaluated and some functionality migrated into newer information services and monitoring frameworks adopted by XSEDE, PRACE, and other research infrastructures. Many operational concepts pioneered in MDS informed later directory and discovery designs used by contemporary platforms at European Grid Infrastructure and National Center for Supercomputing Applications.

Category:Grid computing software