PAC learning

PAC learning
Name	PAC learning
Discipline	Computational learning theory
Introduced	1984
Notable persons	Leslie Valiant, Michael Kearns
Related	Probably Approximately Correct, Computational complexity theory

PAC learning.

PAC learning is a framework in theoretical computer science and machine learning that formalizes when a learning algorithm can, with high probability, produce a hypothesis approximating an unknown target concept. Originating in the 1980s, it connects ideas from Leslie Valiant, Valiant's model, computational complexity theory, statistical learning theory, and algorithmic learning theory to give rigorous criteria for feasibility of learning. The framework has influenced research at institutions like MIT, Harvard University, Carnegie Mellon University, and companies such as Google and Microsoft Research.

Introduction

PAC learning was introduced to analyze the learnability of concept classes such as Boolean function classes, conjunctions, disjunctions, decision lists, DNF, and CNF. It formalizes learning from examples drawn from distributions studied in probability theory and evaluated with respect to error measures from statistical decision theory. Foundational contributors include Leslie Valiant, Michael Kearns, Valiant's collaborators, and subsequent theorists at venues like STOC, FOCS, and COLT. Related theoretical tools draw on results from VC dimension, Occam's razor (philosophy), PAC-Bayes, and boosting research by scholars affiliated with University of California, Berkeley, University of Toronto, and Princeton University.

Formal Definition

A PAC learning model considers a learner interacting with a source of labeled examples according to a distribution over an instance space such as {0,1}^n used in studies at Bell Labs and IBM Research. The formal setup specifies parameters epsilon and delta, tied to probabilistic guarantees in lines of work by scholars associated with National Science Foundation grants and workshops at SIAM. The hypothesis class is drawn from families like linear threshold functions, decision trees, k-CNF, or monomials; correctness is judged by error under the distribution, echoing criteria used in evaluations at NeurIPS and ICML. Complexity measures used include sample size bounds connected to VC dimension and computational resource bounds linked to NP, co-NP, and other classes studied at Stanford University and University of Cambridge.

Sample Complexity and Bounds

Sample complexity results quantify how many examples suffice for PAC guarantees, using combinatorial parameters such as VC dimension and shattering numbers discussed in seminars at Courant Institute and Institute for Advanced Study. Upper and lower bounds often relate to information-theoretic limits studied alongside work on Shannon's coding theorem and algorithmic lower bounds from P vs NP problem literature. Tight bounds have been established for concept classes including halfspaces, axis-aligned rectangles, and finite automata by researchers at ETH Zurich and Ecole Polytechnique. Distribution-specific bounds connect to results on Gaussian distributions and uniform distribution assumptions invoked in papers from Princeton and Caltech.

Algorithms and Learnability Results

Constructive algorithms demonstrating PAC learnability include empirical risk minimization procedures linked to AdaBoost analyses, perceptron algorithms tied to Rosenblatt's early work, and linear programming techniques explored at Bell Labs and AT&T Research. Negative results show hardness via reductions from 3-SAT, Graph Coloring, and other NP-complete problems studied at Cornell University and University of Illinois Urbana-Champaign. Average-case analyses relate to themes from Erdős–Rényi model and randomized algorithm studies by groups at Columbia University and New York University. Key theorems relate learnability to representation size and computational feasibility, building on complexity theory results from Alan Turing-inspired programs and modern investigations at Microsoft Research.

Extensions and Variants

The PAC framework has spawned extensions such as agnostic learning developed in work by scholars at Harvard University and Yale University, distribution-free variants considered in collaborations with DARPA programs, and frameworks incorporating privacy constraints like differential privacy influenced by research at Stanford University and Harvard John A. Paulson School of Engineering and Applied Sciences. Other variants include online learning links to Littlestone's online mistake bound model, semi-supervised approaches studied at University of Washington, and computational models combining PAC with cryptographic assumptions explored at IBM Research and RSA Laboratories.

Applications and Limitations

PAC learning guides theoretical understanding of when classes such as support vector machines-representable concepts, neural networks, and boosted classifiers are learnable, informing applied work at institutions like Facebook, Amazon, and Apple Inc.. Limitations include computational intractability for certain rich classes shown by reductions to NP-hard problems, and distributional assumptions that may not hold in real-world settings examined in field studies at NASA and NOAA. Ongoing research connects PAC-style guarantees to contemporary topics like deep learning generalization, robustness studies after incidents reported by IEEE, and fairness evaluations influenced by panels at United Nations workshops.

Category:Computational learning theory