Ensemble learning

Ensemble learning
Name	Ensemble learning
Type	Machine learning method
Field	Artificial intelligence

Ensemble learning Ensemble learning combines multiple predictive models to improve accuracy, robustness, and generalization. It aggregates diverse learners such as decision trees, neural networks, and support vector machines to reduce variance, bias, or model uncertainty. Widely used across industry and research, ensembles intersect with work by institutions and individuals in computing and statistics.

Overview

Ensemble methods build on contributions from researchers at Bell Labs, IBM, AT&T, Stanford University, Massachusetts Institute of Technology, Carnegie Mellon University, University of Toronto, University of California, Berkeley, University of Oxford, University of Cambridge, Harvard University, Princeton University, University of Washington, University of California, Los Angeles, Columbia University, Cornell University, University of Michigan, ETH Zurich, University of Tokyo, Tsinghua University, Peking University, Seoul National University, National University of Singapore, Microsoft Research, Google Research, Facebook AI Research, OpenAI, DeepMind, Yahoo! Research, Bellcore, Siemens, Baidu Research, NVIDIA Research, Adobe Research, Intel Labs, Amazon Web Services and Oracle Corporation. Foundational figures include researchers affiliated with Leo Breiman, Amit Singhal, Geoffrey Hinton, Yann LeCun, Yoshua Bengio, Vladimir Vapnik, Judea Pearl, Donald Knuth, John Tukey, Bradley Efron, Leo Breiman's contemporaries and collaborators from statistical and computational communities.

Methods

Popular ensemble techniques include bagging, boosting, stacking, and voting, developed across projects at Bell Labs, Stanford University, UC Berkeley, Carnegie Mellon University and companies such as IBM, Microsoft, Google, Amazon, Facebook, NVIDIA, Intel, Siemens, Baidu, Alibaba Group, Tencent, SAP SE, Siemens Healthineers, General Electric, Siemens AG, Samsung Electronics, LG Corporation, Sony Corporation, Hitachi, Fujitsu, Panasonic Corporation, NEC Corporation, Toshiba Corporation, Ericsson, Qualcomm, SpaceX, Tesla, Inc., Apple Inc., Oracle Corporation, SAP SE and Accenture. Bagging variants include random forests and bootstrap aggregating; boosting includes AdaBoost, Gradient Boosting Machines, XGBoost, LightGBM and CatBoost; stacking ensembles combine heterogeneous bases via meta-learners; voting ensembles use majority or weighted schemes. Hybrid approaches merge tree ensembles with deep learning backbones from groups at Google DeepMind, OpenAI, DeepLearning.AI, Microsoft Research Station Q, Facebook AI Research (FAIR) laboratories and academic labs at Imperial College London.

Theoretical foundations

Statistical learning theory and information-theoretic analyses underpin ensemble behavior, drawing on work by Vladimir Vapnik, Amitkumar V. V., Bradley Efron, John Tukey, Leo Breiman, Jerome Friedman, Robert Tibshirani, Trevor Hastie, Christopher Bishop, Richard Sutton, Andrew Ng, Michael Jordan, David MacKay, David Rumelhart, Paul Samuelson, Thomas Bayes, Pierre-Simon Laplace, Harold Hotelling, William Gosset, Ronald Fisher, Karl Pearson, Andrey Kolmogorov, Claude Shannon, Norbert Wiener, Alan Turing, Alonzo Church, John von Neumann, Srinivasa Ramanujan, Kurt Gödel, Emil Artin, Paul Erdős, John Nash, Norbert Wiener and Ralph Hartley. Generalization bounds, bias–variance decomposition, PAC learning, and margin theory explain how ensembles reduce error; boosting connects to additive modeling and forward stagewise regression; randomization and subsampling relate to bootstrap theory developed by Bradley Efron. Error-correlation decompositions and concentration inequalities from researchers at Princeton University and Harvard University inform ensemble design.

Applications

Ensembles power tasks in computer vision, natural language processing, bioinformatics, finance, healthcare, remote sensing and engineering, with deployments at Google, Facebook, Amazon, Microsoft, Apple, IBM Watson, DeepMind, OpenAI, NASA, European Space Agency, National Aeronautics and Space Administration, Centers for Disease Control and Prevention, World Health Organization, National Institutes of Health, Mayo Clinic, Cleveland Clinic, Johns Hopkins University, Roche, Novartis, Pfizer, GlaxoSmithKline, Siemens Healthineers, Philips and GE Healthcare. In competitions and benchmarks hosted by Kaggle, ImageNet, GLUE, SQuAD, TREC, KDD Cup, Netflix Prize, DARPA, NIST, UCI Machine Learning Repository and ICLR, ensembles often top leaderboards. Use cases include disease diagnosis, credit scoring, fraud detection, recommender systems, autonomous driving, satellite imagery analysis, genomics, and high-frequency trading as implemented by teams at Goldman Sachs, J.P. Morgan, Morgan Stanley, Two Sigma, Renaissance Technologies and Citadel LLC.

Evaluation and performance

Ensemble performance is assessed using cross-validation, held-out test sets, bootstrapping, ROC curves, precision–recall metrics, calibration plots and statistical tests, practiced in labs at Stanford University, Harvard University, MIT, ETH Zurich, University of Oxford, University of Cambridge, Princeton University and companies such as Google, Microsoft, Facebook, Amazon and IBM. Benchmarks from ImageNet, COCO, GLUE, SQuAD and MNIST reveal trade-offs between accuracy, latency, interpretability and compute cost. Model ensembling can improve robustness against distribution shift and adversarial attacks studied by research groups at OpenAI, DeepMind, Microsoft Research and Facebook AI Research.

Practical considerations

Practical deployment addresses inference speed, memory, energy consumption, model maintenance and compliance with regulatory bodies like European Commission, U.S. Food and Drug Administration, National Institute of Standards and Technology, UK Information Commissioner's Office, European Medicines Agency and standards from ISO organizations. Hardware acceleration uses GPUs and TPUs from NVIDIA, Google, Intel and AMD; model distillation and pruning techniques from teams at Google Brain, DeepMind, Stanford University and Berkeley AI Research reduce footprint. Production pipelines leverage platforms by Amazon Web Services, Microsoft Azure, Google Cloud Platform, IBM Cloud, Oracle Cloud and orchestration tools from Kubernetes, Docker, Apache Spark, Hadoop, Airflow and MLflow.

History and development

Early ensemble ideas trace to statistics and voting theory debated by figures associated with Royal Society, French Academy of Sciences, Prussian Academy of Sciences, Bell Labs, RAND Corporation, IBM Research, AT&T Bell Laboratories and universities including Cambridge University, Oxford University, Harvard University, University of Chicago, University of Pennsylvania, Yale University and Columbia University. Key milestones include bootstrap methods by Bradley Efron, classification and regression trees by researchers linked to Ludwig Maximilian University of Munich and University of California, Berkeley, bagging and random forests developed by Leo Breiman and collaborators, boosting algorithms formalized in work tied to Freund and Schapire and gradient boosting frameworks introduced by teams at Stanford University and University of Washington. Modern scalable implementations emerged from engineering groups at Google, Microsoft, Facebook, Amazon and startups across Silicon Valley and research hubs in Beijing, Shenzhen, Seoul, Bangalore, Tel Aviv, Zurich and Toronto.

Category:Machine learning