Binary entropy function

Binary entropy function
Original: Brona Vector: Alessio Damato Newer version by Rubber Duck · CC BY-SA 3.0 · source
Name	Binary entropy function
Notation	h(p)
Domain	[0,1]
Range	[0,1]
Field	Information theory, Probability theory
Introduced	1948
Notable	Claude E. Shannon

Binary entropy function The binary entropy function quantifies the uncertainty of a Bernoulli trial and appears in foundational work by Claude E. Shannon, in contexts involving Norbert Wiener-style signal analysis and Alonzo Church-era computability discussions; it connects to limits studied in Harvard University-based information theory and has influenced results at institutions like Bell Labs, Massachusetts Institute of Technology, and Princeton University. The function maps a success probability to an information measure used across research at Bell Telephone Laboratories, Stanford University, and IBM Research and is central to entropy bounds in experiments at Los Alamos National Laboratory and CERN collaborations.

Definition

The binary entropy function h(p) for p in [0,1] is defined by combining logarithmic expressions first formalized in the context of Claude E. Shannon's work on Shannon–Hartley theorem and related to combinatorial counts appearing in Srinivasa Ramanujan-adjacent partition studies; classical texts from Harvard University Press and lecture notes from Yale University present h(p) as the sum of two terms involving the binary logarithm, mirroring formulations in seminars at University of Cambridge and Oxford University. Alternative presentations appear in monographs from Princeton University Press and in course materials from California Institute of Technology that relate h(p) to maximum-likelihood estimators used in collaborations with Bell Labs and Microsoft Research.

Properties

The binary entropy function is symmetric about p = 1/2, attains its maximum at p = 1/2, and vanishes at p = 0 and p = 1—properties that parallel extremal principles discussed in the work of John von Neumann and in entropy inequalities developed at Institute for Advanced Study and Carnegie Mellon University. It is concave on [0,1], satisfies bounds used in Richard Hamming-style coding theory and in proofs of the Asymptotic Equipartition Property presented at IEEE conferences, and it underlies estimates in channel capacity results attributed to Harry Nyquist and Ralph Hartley. Inequalities involving h(p) are central to combinatorial arguments in papers from University of Illinois Urbana-Champaign and to concentration results used by researchers at Stanford University and University of California, Berkeley.

Calculus and Analytic Behavior

Differentiation of the binary entropy yields expressions involving logarithms that appear in analytic treatments by scholars at University of Cambridge and Imperial College London, with first derivative forming a ratio tied to Kullback–Leibler divergences discussed in seminars at Columbia University and New York University. The second derivative is negative on (0,1), confirming concavity in lectures at Massachusetts Institute of Technology and in texts from Springer-Verlag; Taylor expansions around p = 1/2 and p = 0 are used in asymptotic analyses in dissertations from University of Oxford and École Normale Supérieure, and analytic continuation into complex neighborhoods is studied in papers from University of Chicago and University of Michigan. The function's behavior under limits and scaling appears in work associated with Bell Laboratories and in numerical treatments by researchers at Los Alamos National Laboratory.

Applications

Binary entropy is applied to compute limits in coding theorems like those by Claude E. Shannon and to evaluate error exponents in channel analyses presented at IEEE International Symposium on Information Theory and at workshops hosted by IETF and ACM. It quantifies compressibility in algorithms developed by teams at Google and Microsoft Research and guides threshold phenomena in probabilistic combinatorics studied at Princeton University and University of Cambridge. In statistical mechanics contexts influenced by ideas from Ludwig Boltzmann and Josiah Willard Gibbs, h(p) models two-state systems analyzed at Los Alamos National Laboratory and in experiments at CERN. Applications also include bounds in hypothesis testing explored at Harvard Medical School and in bioinformatics methods advanced at Broad Institute.

Generalizations and Related Functions

Generalizations include the Shannon entropy for multinomial distributions discussed in courses at Massachusetts Institute of Technology and Rényi entropy formulations studied at Royal Society meetings and in publications from Springer-Verlag; these relate to Tsallis entropy topics presented at conferences at Max Planck Institute and Kavli Institute. The binary entropy's role in large deviations links it to Sanov's theorem as developed in work originating from Russian Academy of Sciences and to Chernoff bounds used widely by researchers at IBM Research and Microsoft Research. Connections to mutual information and cross-entropy appear in texts from Cambridge University Press and in doctoral theses from University of California, Berkeley and Stanford University.

Category:Information theory