Cray XT5

Cray XT5. It was a massively parallel supercomputer architecture developed by Cray Inc. and a key product in the company's line of scalar systems. Introduced in 2007, it represented a significant evolution of the earlier Cray XT4 platform, designed to scale efficiently to tens of thousands of processing nodes. The system was notable for achieving petaflop performance, most famously with the Jaguar (supercomputer) installation at Oak Ridge National Laboratory.

Overview

The Cray XT5 was engineered as a production supercomputer for capability-class computing at major United States Department of Energy laboratories and other high-performance computing centers. It built upon the proven interconnect and packaging technologies of its predecessors while introducing a major upgrade in processor technology and modularity. Key installations included systems at Oak Ridge National Laboratory, the National Center for Computational Sciences, and the Arctic Region Supercomputing Center. Its design philosophy emphasized balanced performance, high scalability, and reliability for demanding scientific simulation workloads.

Architecture

The architecture was based on a distributed memory MIMD model utilizing a high-bandwidth, low-latency interconnect known as SeaStar2. This network employed a three-dimensional torus topology to efficiently connect thousands of compute nodes. Each node contained one or two AMD Opteron processors, which communicated via a HyperTransport link to a dedicated SeaStar2 router chip. This design minimized latency and contention, a critical feature for applications requiring fine-grained parallelism. The memory hierarchy was non-uniform, with each processor having direct access to its local DDR2 memory.

Hardware Specifications

A standard compute blade housed four AMD Opteron sockets, typically using the "Budapest" or "Shanghai" quad-core or hex-core processors from the K10 microarchitecture. Peak performance per node ranged significantly based on processor generation and clock speed. System memory utilized DDR2 SDRAM, with later upgrades incorporating DDR3 technology. The SeaStar2 interconnect provided a peak bidirectional bandwidth of 9.6 GB/s per link. Large systems comprised cabinets containing multiple blades, with the largest configurations, like Jaguar (supercomputer), exceeding 200 cabinets and containing over 224,000 processing cores.

Software and Operating System

The system ran the Cray Linux Environment, a lightweight, compute-node kernel derived from UNIX System V and tailored for scalability. This environment was paired with the Portals (network protocol) messaging layer for efficient internode communication. Primary programming models included MPI for distributed memory parallelism and OpenMP for shared-memory parallelism within a node. Key supported compilers were from The Portland Group and GNU Compiler Collection, alongside scientific libraries like Cray LibSci and PETSc. System management was handled by the Advanced Cluster Management Software suite.

Applications and Usage

The Cray XT5 was predominantly used for large-scale scientific research across disciplines such as computational fluid dynamics, climate modeling, astrophysics, and materials science. At Oak Ridge National Laboratory, Jaguar (supercomputer) ran codes like NWChem for computational chemistry and S3D for combustion research. It also supported major initiatives like the Innovative and Novel Computational Impact on Theory and Experiment program. Its performance was instrumental in achieving milestones in fusion energy research with the GYRO code and in seismic analysis for the Southern California Earthquake Center.

Historical Context and Successors

The XT5 emerged from the lineage of the Cray T3E and the Cray X1, representing Cray's strategic focus on scalable massively parallel systems using commodity-derived microprocessors. It directly succeeded the Cray XT4 and was contemporaneous with systems like the IBM Blue Gene/P. Its success demonstrated the viability of the AMD Opteron processor in the high-performance computing market. The architecture was eventually superseded by the Cray XE6, which introduced the Gemini (interconnect) and support for AMD Magny-Cours processors, paving the way for the later Cray XC series.