NVMe over Fabrics

NVMe over Fabrics
Name	NVMe over Fabrics
Developer	NVM Express, Inc.
Introduced	2016

NVMe over Fabrics NVMe over Fabrics connects remote non-volatile memory devices using the NVMe command set across networked transports, enabling high-performance, low-latency storage access for cloud, hyperscale, and enterprise deployments. It extends the NVMe protocol beyond local PCIe attachments to fabrics such as Ethernet, InfiniBand, and Fibre Channel, aligning with initiatives from standards bodies and vendors to scale storage for data centers, high-performance computing, and virtualization. Adoption spans hardware vendors, operating system projects, and cloud providers seeking to leverage solid-state storage innovations across distributed systems.

Overview

NVMe over Fabrics was defined by NVM Express, Inc. to carry NVMe commands over fabric transports, aiming to preserve the NVMe efficiency that benefitted platforms like Intel Corporation servers, Samsung Electronics SSDs, and Micron Technology flash arrays while enabling networked storage domains such as those operated by Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Major ecosystem participants include Broadcom Inc., Mellanox Technologies (now part of NVIDIA Corporation), Marvell Technology, Dell Technologies, Hewlett Packard Enterprise, Pure Storage, and open-source projects like Linux kernel community efforts. Industry events such as Flash Memory Summit, Open Compute Project, and SC Conference have driven interoperability testing and demonstrations. Related initiatives intersect with standards from organizations like Internet Engineering Task Force and collaborations with suppliers of management frameworks used in VMware, Inc. and Red Hat environments.

Architecture and Protocols

The architecture maps NVMe namespaces and submission/completion queue semantics across network fabrics, preserving features present in local attachments used by systems such as AMD EPYC platforms and Intel Xeon processors. Controller models tie into management constructs familiar to integrators like Cisco Systems and Arista Networks, and command handling aligns with controllers from firms like Western Digital Corporation and Seagate Technology. Protocol layering considerations reference transport encapsulation used by standards-producing organizations, with orchestration and discovery aligning with storage management stacks in Canonical distributions and orchestration platforms by Kubernetes and OpenStack. Administrative and security controls often integrate with identity and authentication services from Microsoft Active Directory and encryption solutions by Symantec/Broadcom, while telemetry integrates with observability tools from Prometheus and Grafana.

Transport Types and Implementations

Transport implementations include RDMA-based fabrics such as InfiniBand and RDMA over Converged Ethernet (RoCE), Fibre Channel implementations interoperating with ecosystems like Brocade Communications Systems (now part of Broadcom Inc.), and TCP-based transports that enable broader deployment across switching platforms from Arista Networks and Cisco Systems. Hardware acceleration and NIC offload features are provided by vendors like Mellanox Technologies/NVIDIA Corporation and Intel Corporation Ethernet products. Software stacks and initiators are maintained in projects like the Linux kernel and in proprietary drivers from VMware, Inc., while storage arrays implement target functionality in firmware from Pure Storage, Dell EMC, and NetApp. Standards committees coordinate protocol extensions alongside contributors from IBM and academic labs such as those at Massachusetts Institute of Technology and Stanford University.

Performance and Scalability

NVMe over Fabrics targets microsecond-class latencies and high IOPS densities comparable to direct-attached NVMe solutions used in systems from Supermicro and HPE. Benchmarks published by vendors like Intel Corporation, Samsung Electronics, and Micron Technology demonstrate scaling across multi-host clusters typical in Facebook and Netflix infrastructure. Scaling considerations address queue depth, multi-pathing with multipathing implementations in Linux kernel and VMware vSphere, and congestion management tied to switch-level QoS features in products from Cisco Systems and Juniper Networks. Performance optimization often relies on CPU affinity techniques documented in studies by University of California, Berkeley and tuning guides by Red Hat.

Use Cases and Deployment Models

Common deployments include disaggregated storage clouds deployed by Alibaba Group, NVMe-backed virtual machine storage in VMware, Inc. environments, container storage for Kubernetes clusters used by Cloud Native Computing Foundation, and high-performance compute workflows in scientific facilities such as Argonne National Laboratory and Lawrence Livermore National Laboratory. Enterprise use cases span database acceleration for systems like Oracle Corporation and SAP SE, real-time analytics for platforms like Splunk and Elastic NV, and storage tiering in backup solutions from Veeam Software. Deployment models range from converged infrastructure from Dell Technologies and HPE to cloud-delivered services by hyperscalers and managed offerings by Equinix and Digital Realty.

Interoperability, Standards, and Security

Interoperability testing programs coordinated at industry events involve vendors such as Intel Corporation, Samsung Electronics, Micron Technology, Dell Technologies, and VMware, Inc., often producing compliance matrices used by integrators like Wipro and Accenture. Standards governance resides with NVM Express, Inc. and interacts with networking standards bodies like the IETF for TCP/RDMA considerations and the T11 committee for Fibre Channel. Security considerations include transport encryption and authentication techniques leveraging technologies from OpenSSL, Intel Corporation hardware security features, and management integration with Microsoft Active Directory and Okta. Certification and conformance projects echo practices from PCI-SIG and firmware validation processes practiced by UL and national laboratories.

History and Development

Work on extending NVMe beyond PCIe accelerated after the NVMe base specification gained traction in SSD markets led by Samsung Electronics, Intel Corporation, and SanDisk (now part of Western Digital Corporation). The NVMe over Fabrics specification was ratified with contributions from vendors including Cisco Systems, Broadcom Inc., Mellanox Technologies, Dell Technologies, and IBM, and showcased at industry venues like Flash Memory Summit and SC Conference. Subsequent evolution included additions for TCP transport support influenced by networking research at institutions like Massachusetts Institute of Technology and product implementations by VMware, Inc. and cloud providers such as Amazon Web Services.

Category:Storage protocols