BDII

BDII
Name	BDII
Developer	European Organization for Nuclear Research (CERN)
Released	2005
Programming language	Python, LDAP
Operating system	Unix-like
Genre	Information services, Directory services
License	GPL

BDII

BDII is a robust information system originally developed to support large-scale distributed computing infrastructures for high-energy physics experiments. It serves as an aggregation and publishing service that collects resource and service descriptions from disparate sources and exposes them through standardized directory interfaces for consumption by schedulers, brokers, and monitoring tools. BDII played a central role in grid middleware ecosystems used by projects associated with major institutions such as European Organization for Nuclear Research, CERN, Fermi National Accelerator Laboratory, Deutsches Elektronen-Synchrotron, and collaborations like Worldwide LHC Computing Grid.

Introduction

BDII was designed to meet the needs of grid computing projects that required a federated, consistent view of computing resources, storage endpoints, and services across geographically distributed sites. Early adopters included experiments from Large Hadron Collider collaborations and national research laboratories coordinating via frameworks like gLite and ARC middleware. BDII combines data harvesting, attribute aggregation, and LDAP-compliant publishing to provide clients such as workload managers and information brokers from initiatives like HTCondor and Globus Toolkit with authoritative service information.

Architecture and Components

BDII's architecture comprises multiple cooperating components: a local information provider layer, a site-level information collector, an aggregator, and an LDAP publishing frontend. Information providers are implemented using probes and scripts that query service endpoints such as Torque (software), Apache HTTP Server, dCache, and GridFTP to produce attribute-value representations. The site-level collector runs periodic md5-checked pulls and merges data into a site BDII while an upper-level or top BDII aggregates site BDIIs, applying policies and attribute profiles drawn from standards like the Open Grid Forum specifications. The LDAP frontend exposes entries in a hierarchical namespace that maps to resource types recognizable by components developed in projects such as EGEE and ENES.

Key components include: - Information Providers: adapters for services such as SLURM, MySQL, OpenStack Nova, and Ceph that emit LDIF-like records. - Site BDII: a daemon that validates, normalizes, and caches local attributes, interacting with monitoring systems like Nagios and Ganglia. - Top BDII: an aggregator that reconciles site datasets, implements caching strategies, and responds to LDAP queries from clients including GridWay and custom brokers. - Publishing Layer: LDAP server implementation and schema definitions derived from standards embraced by groups like TERENA and RDA.

Deployment and Configuration

Typical deployments span hierarchical tiers: per-site BDIIs and one or more central aggregators. Administrators configure information providers by writing scripts or using existing probes packaged by distributions such as Scientific Linux or CentOS. Configuration files define attributes to publish, update cycles, and access controls; these files often reference schemas and object classes standardized by bodies such as the Open Grid Forum and the International Telecommunication Union. Integration with site management tools like Puppet (software), Ansible (software), and SaltStack facilitates reproducible deployments across compute clusters and storage farms used by projects like Belle II or LIGO Scientific Collaboration.

High-availability setups employ techniques from enterprise deployments: redundant top BDIIs behind load balancers like HAProxy or Keepalived, synchronized configuration via Git repositories, and monitoring integration through alerting channels tied to services such as JIRA or ServiceNow for incident tracking in large collaborations.

Security and Authentication

BDII itself is designed to expose metadata rather than carry sensitive payloads, but deployment in research infrastructures requires integration with authentication and authorization frameworks. Typical installations rely on X.509 certificate infrastructures promulgated by entities such as EUGridPMA and EDG for secure LDAP transport and to authenticate management operations. Access controls may be enforced using TLS, client certificates, and LDAP ACLs coordinated with identity providers like CILogon or federation systems including eduGAIN for multi-institution collaborations. Audit trails often integrate with logging ecosystems built around ELK Stack components and identity assertion records consumed by accounting systems such as those used by Open Science Grid.

Use Cases and Applications

BDII has been used to support job brokering, data replication planning, monitoring dashboards, and accounting in distributed research infrastructures. Workload management systems such as PanDA and Condor-G query BDIIs to discover compute endpoints and match job requirements to site capabilities. Data management services like Rucio and FTS consult BDII-provided attributes to plan transfers between storage endpoints like EOS and StoRM. Monitoring GUIs and submission portals developed by experiments including ATLAS and CMS display aggregated service status by consuming BDII directories. BDII has also assisted in resource federation efforts in multidisciplinary projects involving institutions like ESRF and EMBL.

Performance and Scalability

BDII deployments scale horizontally by adding site-level nodes and vertical aggregation tiers; performance considerations include LDAP query throughput, merge latency, and freshness of information. Caching strategies, delta-update mechanisms, and efficient LDIF diffing reduce bandwidth and load across wide-area links connecting sites such as CERN Data Centre and national computing centers like GridPP. Stress-tested installations demonstrate that tuned BDII topologies can serve thousands of concurrent clients used by production workflows from ATLAS and CMS without introducing significant scheduling bottlenecks. Ongoing operational best practices reference tools from projects like PerfSONAR for network performance validation and Prometheus for telemetry to ensure BDII responsiveness under peak loads.

Category:Distributed computing