Blue Waters

Blue Waters. It was a Cray supercomputer project designed to deliver sustained petascale performance for the open scientific community. Operated by the National Center for Supercomputing Applications at the University of Illinois Urbana-Champaign, it was one of the most powerful and capable supercomputers in the world during its operational lifetime from 2012 to 2019. The system was funded primarily by the National Science Foundation and the state of Illinois, with the goal of enabling transformative research across a wide array of scientific disciplines.

Overview

The project was conceived to provide a dedicated resource for computationally intensive, long-running simulations that were not feasible on other systems. Unlike machines focused on achieving peak LINPACK benchmarks for the TOP500 list, it was engineered for sustained performance on real-world scientific applications. Its architecture was a hybrid system, integrating traditional CPU-based nodes with a large number of GPU-accelerated nodes, which was an innovative approach at the time. The system was housed in a specially constructed facility at the University of Illinois Urbana-Champaign, featuring advanced cooling and power infrastructure to support its massive computational needs.

Hardware and architecture

The physical system was built by Cray Inc. and was based on the Cray XE6 and Cray XK7 architectures. It comprised over 22,000 AMD Opteron compute nodes and later incorporated more than 4,000 NVIDIA Kepler GPU accelerators within its Cray XK7 nodes. This heterogeneous design allowed applications to leverage general-purpose computing on graphics processing units for massively parallel tasks. The machine featured a high-performance Cray Gemini interconnect, which provided low-latency communication across its hundreds of cabinets. Its storage system was equally impressive, centered on a massive Lustre parallel file system capable of storing over 500 petabytes of data, managed by technology from DataDirect Networks.

Software and applications

A specialized software environment supported researchers, including the Cray Linux Environment and programming models like OpenACC and CUDA for GPU computing. The National Center for Supercomputing Applications provided extensive user support and training to help teams port and scale their codes. The system enabled groundbreaking work in fields such as astrophysics, simulating the assembly of the Milky Way; molecular dynamics, investigating the HIV capsid; and climate modeling, producing high-resolution simulations of hurricane formation. Other major projects included lattice quantum chromodynamics calculations for the USQCD collaboration and detailed models of earthquake ruptures along the San Andreas Fault.

History and funding

Initial planning began in 2007, with a major award from the National Science Foundation announced in 2007 as part of its Track 1 program for flagship supercomputing resources. The original vendor, IBM, withdrew from the project in 2011, leading to a recompetition won by Cray Inc.. The system entered full production in early 2013 at the National Center for Supercomputing Applications. Its total cost was over $300 million, funded by the National Science Foundation, the University of Illinois, and the state of Illinois. The computer was formally decommissioned in 2019 after exceeding its planned service life, having provided over 7 billion core-hours of computation to hundreds of research teams from institutions across the United States.

Impact and legacy

It is widely regarded as a landmark project that demonstrated the viability of sustained petascale computing for open science. The research it enabled resulted in thousands of scholarly publications in prestigious journals like *Science* and *Nature*. Its hybrid CPU/GPU architecture influenced the design of subsequent exascale-era systems, such as the Frontier and Aurora supercomputers. The project also trained a generation of computational scientists in large-scale, heterogeneous computing. Its data-intensive workloads helped advance the Globus Toolkit for research data management, leaving a lasting impact on the national cyberinfrastructure ecosystem.

Category:Supercomputers Category:National Center for Supercomputing Applications Category:Cray supercomputers Category:University of Illinois Urbana-Champaign Category:2012 establishments in Illinois