Lempel–Ziv–Welch

Lempel–Ziv–Welch
Name	Lempel–Ziv–Welch
Inventors	Abraham Lempel, Jacob Ziv, Terry Welch
Year	1984
Type	Lossless data compression algorithm
Related	Lempel–Ziv–Storer–Szymanski, Lempel–Ziv–Markov chain algorithm, DEFLATE

Lempel–Ziv–Welch is a lossless data compression algorithm developed as an efficient dictionary-based compressor. It builds on earlier work by Abraham Lempel, Jacob Ziv, and advances by Terry Welch, and became widely used in formats and utilities associated with UNIX, GGI (graphics library), CompuServe, and GIF. The method influenced later standards and implementations by organizations such as ISO/IEC and projects including PKZIP, gzip, and PNG.

History and Development

The algorithm emerged from research threads initiated by Abraham Lempel and Jacob Ziv in the 1970s, including the LZ77 and LZ78 papers, and was published by Terry Welch in 1984 as an improvement to LZ78. Early adoption occurred in the context of computing projects at institutions like MIT, Bell Labs, and IBM, and it interacted with contemporaneous work at AT&T and Bell Telephone Laboratories. Implementations and adaptations were incorporated into commercial products by firms such as Sears Roebuck, CompuServe, and Unisys, while academic analysis was advanced by researchers at Stanford University, Carnegie Mellon University, and University of California, Berkeley.

Algorithm Overview

The algorithm is a dictionary coder that dynamically constructs a table of sequences drawn from the input stream, similar in lineage to LZ78 and related to variants like LZSS and LZMA. It initializes a dictionary with entries for the symbol alphabet (for example, the 256-byte alphabet common on IBM PC and DEC systems) and emits codes that reference dictionary entries rather than raw symbols. As the encoder reads symbols, it adds new sequences to the dictionary following a pattern influenced by the earlier Ziv–Lempel formulations, enabling compact representation of repeated substrings found in data from sources such as NASA telemetry, CERN datasets, and archives managed by National Archives and Records Administration.

Encoding and Decoding Processes

Encoding proceeds by scanning the input stream to find the longest sequence present in the current dictionary, outputting its code, and adding a new dictionary entry formed by that sequence plus the next input symbol. The decoder mirrors this process, reconstructing the same dictionary entries from received codes without transmitting explicit dictionary contents, a behavior examined in studies at University of Cambridge and Princeton University. Corner cases—such as the "KwKwK" scenario analyzed by researchers affiliated with University of Illinois Urbana–Champaign and ETH Zurich—require careful handling to preserve synchrony between encoder and decoder. Practical encoder/decoder pairs have been implemented in languages used at Bell Labs and Microsoft Research, including C and assembly for Intel architectures.

Implementation Details and Variants

Implementations typically use fixed-size or variable-width codewords and may reset the dictionary when full, strategies compared in implementations by teams at Bell Labs and Sun Microsystems. Variants include adaptive bit-width schemes, implementations combining hashing techniques from Google research with trie structures developed at AT&T Labs, and hybrid approaches that integrate with entropy encoders employed by IBM Research and Xerox PARC. Open-source projects hosted by Free Software Foundation and Apache Software Foundation have produced widely used implementations, and academic variants explore integration with Markov models and context-mixing techniques investigated at MILA and University of Toronto.

Performance and Complexity

The algorithm offers linear-time average performance under typical assumptions, a property analyzed in depth by scholars at Cornell University and University of Washington, with memory and time complexity depending on dictionary management strategies used in implementations at Intel Corporation and ARM Holdings. Compression ratios vary by data domain; text corpora archived by Project Gutenberg and image streams used in GIF tend to compress well, while already compressed multimedia from MPEG or JPEG show limited gains. Comparative benchmarks by Bell Labs, Oracle Corporation, and academic groups demonstrate that although newer methods like DEFLATE and LZMA can outperform it in many scenarios, the algorithm retains advantages in simplicity and decoder-side resource constraints, relevant to embedded systems designed by Texas Instruments and STMicroelectronics.

Applications and Usage

The algorithm has been employed in file compression utilities, network protocols, and archival formats, influencing standards adopted by bodies such as IETF and ITU-T. Notable uses include integration into legacy systems at CompuServe and archival tooling used by National Library of Medicine and Library of Congress digitization projects. It also appears in implementations within software packages from Microsoft Corporation, Apple Inc., and open-source ecosystems like GNU Project distributions. Educational courses at Massachusetts Institute of Technology and Imperial College London use it as a teaching example for dictionary-based compression.

Patents and Legal Issues

Patents related to dictionary compression and specific implementations were filed by entities including Unisys and prosecuted in jurisdictions handled by law firms associated with Baker McKenzie and Jones Day. Litigation over patent claims affected adoption in commercial products, with legal outcomes influencing policies at companies such as IBM and Sun Microsystems. The expiry of key patents and decisions in forums like the United States Patent and Trademark Office and courts in United Kingdom and Germany changed the commercial landscape, enabling wider open-source implementations propagated by organizations like Apache Software Foundation and Free Software Foundation.

Category:Data compression algorithms