RDMA

RDMA
Name	RDMA
Caption	Remote direct memory access in a data center environment
Invented	1990s
Developers	Intel Corporation, IBM, Microsoft, Mellanox Technologies, Cisco Systems
Type	Computer networking technology

RDMA Remote direct memory access (RDMA) enables one computer to access the memory of another without CPU intervention, reducing latency and CPU load. It is used in high-performance computing, storage systems, and cloud services where low-latency, high-throughput communication is required. RDMA intersects with networking hardware, operating systems, and data-center architectures developed by major vendors.

Overview

RDMA provides direct transfer between the address spaces of two hosts by bypassing the host CPU path and kernel involvement, enabling efficient use of resources for applications such as Hadoop, Oracle Database, PostgreSQL, Redis, and Memcached. Hardware vendors like Intel Corporation, Mellanox Technologies, Cisco Systems, and Broadcom Inc. integrate RDMA offload engines into network adapters to support protocols adopted by cloud providers including Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Research institutions such as Lawrence Livermore National Laboratory and Oak Ridge National Laboratory use RDMA in large-scale simulations on systems procured from Cray Inc. and IBM.

Technology and Architecture

RDMA architecture comprises host channel adapters or network interface cards provided by companies like Mellanox Technologies and Intel Corporation, switch fabrics from Arista Networks and Cisco Systems, and software stacks in kernels maintained by projects like Linux kernel and implementations by Microsoft Corporation. The mechanism relies on concepts such as zero-copy I/O and memory registration popularized in projects associated with Message Passing Interface implementations used at centers like NERSC and on systems like Frontera (supercomputer). Hardware features include queue pairs, completion queues, and work requests present in adapters delivered to clients such as Facebook and Twitter to accelerate data-plane operations.

Implementations and Protocols

Common RDMA protocols and implementations include the InfiniBand architecture standardized by the InfiniBand Trade Association, RDMA over Converged Ethernet (RoCE) promoted by companies like Mellanox Technologies and Broadcom Inc., and Internet Wide Area RDMA Protocol (iWARP] developed through work involving Intel Corporation and standards groups in IETF. Software stacks are offered by vendors including Microsoft for integration in Windows Server and by open-source projects within the Linux Foundation. Middleware and libraries such as OpenFabrics Alliance releases, Open MPI, libfabric from the OpenFabrics Alliance, and vendor SDKs from Mellanox and Intel expose RDMA to applications like Apache Cassandra and Ceph.

Performance and Use Cases

RDMA delivers microsecond-scale latencies and high gigabit-to-terabit throughput leveraged by high-performance computing centers like Los Alamos National Laboratory and cloud-scale services at Google and Amazon.com. Use cases include distributed file systems such as Lustre and GPFS, storage fabrics like NVMe over Fabrics adopted by vendors like NetApp and Dell EMC, and database acceleration in deployments by Oracle Corporation and Microsoft SQL Server. Emerging deployments in machine learning clusters from NVIDIA and distributed training frameworks such as TensorFlow and PyTorch utilize RDMA to reduce inter-GPU communication time on systems integrated by companies like Supermicro and HPE.

Security and Reliability

Security concerns around RDMA involve access control, memory protection, and the potential for unauthorized remote memory access; industry responses include management frameworks from OpenStack and cloud security controls from Amazon Web Services and Microsoft Azure. Reliability features are provided by adapters and fabrics from Mellanox Technologies and Cisco Systems that implement congestion control and loss-recovery mechanisms; standards efforts by the InfiniBand Trade Association and networking research at institutions like MIT and Stanford University guide best practices. Vulnerability disclosures handled by vendor CERT teams such as US-CERT and corporate security teams at Intel Corporation have driven firmware and driver updates in response to discovered flaws.

History and Development

RDMA concepts date to research in the 1980s and commercialized during the 1990s with contributions from IBM and later standardization via the InfiniBand Trade Association in the early 2000s. The growth of data centers and cloud providers including Amazon.com and Google in the 2000s and 2010s accelerated adoption, while companies like Mellanox Technologies and Intel Corporation advanced NIC offloads and RoCE technologies. Academic work from UC Berkeley and Carnegie Mellon University informed protocol research that influenced implementations in Open MPI and the Linux kernel, shaping modern deployments across research laboratories and hyperscale operators such as Facebook and Microsoft.

Category:Computer networking