MurmurHash

MurmurHash
Name	MurmurHash
Author	Austin Appleby
First release	2008
Latest release	2014
License	Public domain / MIT-compatible
Programming languages	C, C++, Java, Python, Go

MurmurHash MurmurHash is a family of non-cryptographic hash functions used for fast hash-based lookup and probabilistic data structures. Designed for speed and good distribution on typical processor architectures, MurmurHash is widely adopted in systems software, databases, and networking libraries. Implementations exist across platforms and languages, and the algorithm has influenced subsequent hashing techniques and libraries.

Overview

MurmurHash was created to provide a high-performance alternative to hashing functions used in software such as LevelDB, Memcached, Redis, Hadoop, and Apache Cassandra. It competes with functions like those in CityHash, FarmHash, xxHash, SipHash, and FNV families, aiming to balance throughput and avalanche characteristics for structures like Hash table implementations in SQLite, PostgreSQL, and MySQL. The design considers constraints from processors by accounting for instruction pipelines in x86 architecture, ARM architecture, and modern Power architecture CPUs commonly found in Intel and AMD servers. Authors and maintainers engaged with communities around projects such as GitHub, Stack Overflow, SourceForge, and academic venues including USENIX workshops.

Algorithms and Variants

MurmurHash includes multiple variants: MurmurHash1, MurmurHash2, MurmurHash3 (including 32-bit, 128-bit x86, and 128-bit x64 variants). These variants relate conceptually to hashing approaches used in Knuth-era literature and to modern designs like Zobrist hashing in game engines and Consistent hashing schemes used by Akamai and Amazon Web Services. Comparisons often reference deterministic properties explored in papers at SIGMOD, VLDB, and ICDE. The algorithm uses mixing functions and rotations similar to techniques in Jenkins hash and shares goals with methods discussed in RFC 1321 and proofs in conferences like Crypto, though MurmurHash is explicitly non-cryptographic. Extensions and ports have been produced by contributors affiliated with organizations such as Google, Facebook, Microsoft, Twitter, LinkedIn, and NetApp.

Implementation Details

Implementations focus on unaligned memory access, little-endian processing, and word-sized arithmetic optimized for compilers like GCC, Clang, and MSVC. Key operations include bitwise XOR, left and right shifts, and multiplication by constants chosen for avalanche properties; similar low-level optimizations are found in libraries such as libc, Boost, LLVM, and glibc. Implementations exist in language ecosystems maintained by Oracle (company), Python Software Foundation, The Go Programming Language team, and projects hosted under Apache Software Foundation. Integration points include build systems like CMake, Autotools, Bazel, and package registries such as PyPI, npm, Maven Central, and RubyGems.

Performance and Quality

MurmurHash emphasizes throughput for streaming data and small keys, with performance benchmarks often run on servers from Dell, HPE, and custom clusters at Google and Microsoft Research. Quality assessments reference statistical tests such as Dieharder, SMHasher, and empirical collision analyses used by engineers at Facebook and researchers at University of California, Berkeley and Massachusetts Institute of Technology. Compared to cryptographic hashes like SHA-1, SHA-256, and MD5, MurmurHash trades resistance to adversarial collisions for speed, similar to the trade-offs discussed around SipHash in literature from Johns Hopkins University and EPFL.

Use Cases and Applications

MurmurHash is used in indexing and lookup engines such as Elasticsearch, Solr, Lucene, and Sphinx Search, as well as in streaming platforms like Apache Kafka and Apache Flink. It appears in big data stacks including Spark, HBase, and Cassandra for partitioning and sampling, and in analytics systems at companies like Uber, Airbnb, Snapchat, and Pinterest. Developers integrate MurmurHash into client libraries for gRPC, Thrift, Protocol Buffers, and Avro to provide deterministic sharding and bloom filter hashing in databases and caches used by Netflix and Dropbox.

Security Considerations

Because MurmurHash is non-cryptographic, it is unsuitable for cases requiring adversarial resistance such as password hashing or message authentication, unlike algorithms standardized by NIST such as SHA-3 or constructions like HMAC. Research into hash-flooding attacks affecting web servers and frameworks like Django, Rails, and Express (web framework) motivated adoption of keyed or cryptographic hashes in some libraries. Security teams at Google, Cloudflare, and Mozilla recommend using alternatives such as SipHash for hash table protection against denial-of-service vectors exploited in HTTP services and REST APIs.

History and Development

MurmurHash was authored by Austin Appleby and first published in 2008; it evolved with community contributions on platforms like GitHub and discussions on forums including Stack Overflow and Reddit. Over time, maintainers from organizations such as Google, Facebook, Twitter, and independent contributors ported and optimized variants for different architectures and languages. The algorithm’s development intersected with academic and industry work on hashing showcased at venues like USENIX Security Symposium, ACM SIGCOMM, and IEEE INFOCOM, and it influenced later high-performance hashing projects such as CityHash and FarmHash.

Category:Hash functions