Dominant Resource Fairness

Dominant Resource Fairness
Name	Dominant Resource Fairness
Classification	Resource allocation mechanism
Introduced	2011
Authors	Cynthia Dwork, Frank McSherry, Michael Pitassi, Ittai Abraham
Field	Computer science, Algorithmic game theory

Dominant Resource Fairness

Dominant Resource Fairness is an allocation mechanism for multi-resource sharing in cluster computing and distributed systems. It was introduced to extend notions of fair division to settings with heterogeneous resources such as CPU, memory, and network, and has been discussed in contexts ranging from cloud computing to datacenter scheduling. The mechanism has been analyzed alongside concepts from Nash bargaining solution, Shapley value, Envy-freeness, and Pareto efficiency within literature on algorithmic mechanism design and distributed systems.

Introduction

Dominant Resource Fairness defines fairness by comparing agents' dominant shares across multiple resource types, situating the solution among classic results like the Nash equilibrium in non-cooperative game theory, the Kalai–Smorodinsky bargaining solution in bargaining theory, and the Proportional fairness notion used in network congestion control. The original formulation emerged from work at intersections of Microsoft Research, University of California, Berkeley, and groups studying MapReduce and Hadoop clusters, and it has been cited in discussions with researchers affiliated with Google, Amazon Web Services, Facebook, and IBM Research. The approach provides a robust alternative to scalar fairness measures used in systems designed by teams at Apache Software Foundation, OpenStack, and Kubernetes communities.

Model and Definitions

The formal model considers a finite set of indivisible or divisible resources indexed like resources in CPU architecture, DRAM, SSD, Ethernet, and GPU devices, and a set of agents often representing tenants from organizations such as Dropbox, Twitter, or Netflix. Each agent reports a demand vector similar to specification formats used in YARN and Mesos. Dominant share is defined as the maximum ratio of allocated units to capacity for any resource type, paralleling metrics used in Quality of Service specifications in ITU recommendations and IEEE standards. The solution concept is grounded in comparative measures that echo the axioms behind the Shapley value and the Core (game theory), while avoiding incentives exploited in mechanisms studied by Vickrey and Clarke.

Algorithm and Implementation

Allocation is computed by increasing allocations in lockstep to equalize agents' dominant shares until capacity constraints bind, an approach related to water-filling algorithms used in information theory and techniques from linear programming and convex optimization. Implementations embed the mechanism into resource managers like Apache Mesos, Kubernetes scheduler, and Hadoop YARN via admission control similar to schedulers designed by Google Borg and Omega teams. Practical deployments integrate with monitoring systems such as Prometheus, logging frameworks like ELK Stack, and orchestration tools inspired by Docker Swarm and HashiCorp Nomad. Computational steps reference subroutines from work by Dantzig on simplex methods and from Karmarkar on interior-point methods.

Properties and Fairness Guarantees

Dominant Resource Fairness guarantees Pareto optimality under divisible resources, envy-freeness for agents with identical entitlements, and strategyproofness for reporting of resource requirements under certain assumptions, echoing desiderata found in proofs by Arrow and impossibility results related to Gibbard–Satterthwaite theorem. The mechanism yields allocations comparable to those justified by axioms of Rawlsian justice in economic theory and notions of max-min fairness used by researchers at Cisco and Ericsson in network resource management. Analyses often use tools from probability theory, linear algebra, and worst-case bounds reminiscent of complexity results from Cook and Karp.

Variants and Extensions

Extensions handle dynamic arrivals and departures as studied in models related to queueing theory and stochastic processes, incorporate priority weights akin to admission controls used by AT&T and Verizon, and adapt to hierarchical resource pools following designs from Google Cloud and AWS Organizations. Variants include approximations for indivisible tasks linking to combinatorial allocation studied alongside Knapsack problem and bin packing, and incentive-aware variants integrating payments inspired by designs like the Vickrey–Clarke–Groves mechanism. Research has cross-fertilized with work on network virtualization, software-defined networking, and scheduling results from Edsger Dijkstra’s lineage.

Applications and Use Cases

The mechanism has been applied to multi-tenant clusters in companies such as Google, Facebook, Microsoft, and LinkedIn to allocate CPU, memory, and IO; to cloud orchestration in OpenStack deployments; to edge computing testbeds associated with Intel and Nokia; and to academic testbeds at institutions like MIT, Stanford University, Carnegie Mellon University, and University of Texas at Austin. Use cases include allocation for batch processing with MapReduce jobs, interactive services in microservice architectures practiced by Uber and Airbnb, and resource negotiation in federated platforms investigated at European Organization for Nuclear Research and National Institutes of Health collaborations.

Criticisms and Limitations

Critics highlight limitations in environments with highly heterogeneous task sizes or where dominant resource definitions conflict with business priorities seen at Amazon, Walmart Labs, and Bloomberg LP, and raise concerns about overheads when integrated with real-time schedulers like those of NVIDIA GPU clusters. Game-theoretic critiques point to strategic behaviors studied in experiments at Yahoo Research and limitations akin to those in impossibility theorems by Sen and Greenberg. Practical constraints include measurement errors comparable to issues addressed in IEEE 1588 time synchronization, and the need for approximations in large-scale systems as encountered by Alibaba and Tencent.

Category:Resource allocation