CD-HIT

CD-HIT
Name	CD-HIT
Developer	BGI / University of Washington
Released	2001
Programming language	C++
Operating system	Linux, macOS, Windows (via ports)
License	Free for non-commercial use

CD-HIT CD-HIT is a widely used bioinformatics program for clustering and deduplicating large sets of protein and nucleotide sequences. It provides high-speed sequence clustering by identifying representative sequences, enabling downstream analyses in genomics, metagenomics, and proteomics. The tool has been integrated into many pipelines and databases and is cited across literature involving large-scale sequencing projects and comparative genomics.

Overview

CD-HIT performs fast clustering of biological sequences to reduce redundancy in datasets produced by projects such as the Human Genome Project, the 1000 Genomes Project, and large-scale initiatives at institutions like the Broad Institute and European Bioinformatics Institute. It is commonly used alongside tools and resources including BLAST, HMMER, UniProt, GenBank, and analysis platforms like Galaxy (platform), and is frequently cited in studies from groups at the Max Planck Society, Wellcome Trust Sanger Institute, and National Institutes of Health. CD-HIT's output aids work in comparative studies that cross-reference repositories such as the Protein Data Bank, the KEGG database, and the Ensembl project.

Algorithm and Implementation

The core algorithm relies on a greedy incremental clustering strategy and short-word (k-mer) filtering, concepts comparable to techniques in programs like BLAST, FASTA, and algorithms used in the Smith–Waterman algorithm family. Implementation in C++ emphasizes memory-efficient data structures and bitwise operations inspired by high-performance computing groups at institutions such as Lawrence Berkeley National Laboratory and Los Alamos National Laboratory. CD-HIT uses hashing and suffix-like k-mer indexing akin to methods employed in assemblers developed by teams at Celera Genomics and the J Craig Venter Institute, while optimizing for throughput in environments similar to compute clusters at Argonne National Laboratory and cloud services from Amazon Web Services. Parallelization approaches echo practices from projects at Intel and NVIDIA for multicore and GPU-accelerated algorithms, though CD-HIT primarily targets CPU architectures.

Applications and Use Cases

CD-HIT is applied in pipelines for microbial genomics from centers like the Centers for Disease Control and Prevention, environmental metagenomics led by collaborations with the Monterey Bay Aquarium Research Institute, and microbiome surveys aligned with work at the Human Microbiome Project. It supports protein family analyses relevant to studies published by researchers at universities such as Harvard University, Stanford University, Massachusetts Institute of Technology, University of California, Berkeley, and Johns Hopkins University. CD-HIT reduces input sizes for tools like MAFFT, Clustal Omega, PhyML, and RAxML used by consortia such as the International Nucleotide Sequence Database Collaboration and projects like the Earth Microbiome Project. In industrial settings, it is used by biotechnology firms such as Illumina and Thermo Fisher Scientific for reference curation and redundancy reduction.

Performance and Benchmarks

Benchmarking studies often compare CD-HIT with alternatives including UCLUST, VSEARCH, MMseqs2, and USEARCH across datasets from the NCBI Sequence Read Archive and curated collections like Swiss-Prot. Reports from computational biology groups at EMBL-EBI and the University of Tokyo demonstrate that CD-HIT excels in speed and low memory footprint for high-identity clustering thresholds, while some newer tools trade off memory for sensitivity. Performance metrics are commonly evaluated using standards from consortia such as the Genome Reference Consortium and datasets generated in collaborations with institutions like Cold Spring Harbor Laboratory and Sanger Institute sequencing centers.

Limitations and Criticisms

Critics note that CD-HIT's greedy algorithm can produce clusters that differ from results of exhaustive alignment approaches used in methods like the Needleman–Wunsch algorithm or iterative refinement techniques found in HMMER workflows. Some comparative studies from groups at University of California, San Diego and ETH Zurich highlight trade-offs between speed and sensitivity, especially for low-identity clustering or highly diverse metagenomic samples typical of research at the Max Planck Institute for Marine Microbiology. License constraints for commercial use have prompted organizations such as biotech startups and companies affiliated with Cambridge Innovation Capital to evaluate alternative open-source solutions like MMseqs2 and VSEARCH.

History and Development

CD-HIT was developed in the early 2000s by researchers connected to institutions including the Beijing Genomics Institute (BGI) and the University of Washington, building on algorithmic foundations established by earlier sequence comparison tools from labs such as National Center for Biotechnology Information and groups led by scientists affiliated with Stanford University and University of California, San Diego. Over time it has been maintained and extended in collaboration with international partners, cited in work from laboratories at Peking University, University of Cambridge, Imperial College London, and integrated into resources hosted by the European Molecular Biology Laboratory and national centers like Genome Canada.

Category:Bioinformatics software