This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| Quadratic weighted kappa | |
|---|---|
| Name | Quadratic weighted kappa |
| Other names | Quadratic kappa, QWK |
| Field | Statistics |
| Developed by | Jacob Cohen (relation), Robert G. White (applications) |
| Introduced | 1960s |
| Used for | Inter-rater reliability, agreement assessment |
Quadratic weighted kappa Quadratic weighted kappa is a statistical measure for assessing agreement between two raters that assign categorical ordinal ratings; it adjusts raw agreement for chance by weighting disagreements with a quadratic function. Originally developed in the context of inter-rater reliability studies, it has been applied across clinical research, psychometrics, and machine learning evaluation pipelines. Prominent uses appear in settings linked to National Institutes of Health, World Health Organization, Food and Drug Administration, European Medicines Agency and large-scale challenges hosted by Kaggle, Netflix and academic venues such as NeurIPS and ICML.
Quadratic weighted kappa quantifies agreement by comparing an observed rating matrix between two raters to an expected matrix under chance, applying quadratic weights to penalize disagreements more heavily as they diverge. It generalizes unweighted agreement coefficients used in clinical trials overseen by World Health Organization branches and statistical assessments in projects from Harvard University, Stanford University, Massachusetts Institute of Technology, University of Cambridge, University of Oxford and other institutions. Quadratic weighting is often contrasted with linear weighting employed in protocols at Centers for Disease Control and Prevention and standardized-testing organizations such as Educational Testing Service.
Given N items rated by two raters into K ordered categories, form an observed histogram matrix O where O_{i,j} counts items rated i by rater A and j by rater B. Construct an expected matrix E under statistical independence using the product of marginal totals scaled to N. Define a weight matrix w_{i,j} = ((i - j)/(K - 1))^2 (quadratic form). Quadratic weighted kappa κ_q is then κ_q = 1 - (sum_{i,j} w_{i,j} O_{i,j}) / (sum_{i,j} w_{i,j} E_{i,j}). This formulation is analogous to coefficients used historically alongside measures from Jacob Cohen, extensions from Krzysztof Gajewski-style psychometric work, and mathematical treatments in texts associated with American Statistical Association and Royal Statistical Society.
Values of quadratic weighted kappa range from -1 (complete disagreement worse than chance) through 0 (agreement no better than chance) to 1 (perfect agreement). Because quadratic weights penalize larger category differences more than smaller ones, κ_q is sensitive to ordinal structure and preserves monotonicity properties sought in clinical scales used by American Medical Association, American Psychiatric Association, Royal College of Physicians and research consortia at Johns Hopkins University. It is symmetric with respect to rater interchange and reduces to Cohen's kappa under specific weight choices used at institutions like University of California, Berkeley and University of Chicago.
Cohen's kappa, introduced by Jacob Cohen, is a special case of weighted kappa with uniform weights (unweighted) or binary weights; quadratic weighted kappa uses a polynomial weight matrix. Linear weighted kappa, historically used in assessments by Educational Testing Service and methodological papers from Columbia University, uses w_{i,j} proportional to |i-j|/(K-1). Quadratic weighting gives greater penalty to distant disagreements, a choice reflected in scoring rubrics from International Consortium for Health Outcomes Measurement and adjudications in competitions held by Kaggle and IEEE.
Compute O and E from contingency tables; then apply the quadratic weight matrix to obtain κ_q. Implementations exist in statistical packages and machine-learning toolkits maintained by organizations such as Python Software Foundation projects, RStudio packages, and libraries from Scikit-learn-adjacent communities, with code examples featured in repositories affiliated with GitHub, educational materials from Coursera and tutorials from Stanford University. Numerical stability concerns for sparse counts prompt regularization strategies used in workflows at Google Research and Microsoft Research.
Quadratic weighted kappa is widely used for evaluating ordinal predictions in medical imaging contests organized by ISBI, diagnostic scoring validated by Food and Drug Administration, pathology grading at institutions like Mayo Clinic and Cleveland Clinic, and rubric scoring in automated grading systems developed at Carnegie Mellon University and Massachusetts Institute of Technology. High-profile competitions on Kaggle and benchmarks at NeurIPS have used κ_q to rank models for tasks such as cancer staging, diabetic retinopathy severity, and essay scoring, with publications appearing in journals from Nature Publishing Group and Springer Nature.
Critics from methodological groups at American Statistical Association, Royal Statistical Society and research teams at University College London note sensitivity to category prevalence, marginal distributions, and small sample sizes; these issues mirror concerns raised about other chance-corrected indices used in evaluations by European Medicines Agency and reporting standards from CONSORT-aligned trials. Alternative approaches, including ordinal regression metrics promoted by ICML and resampling-based uncertainty estimates advocated by National Institute of Standards and Technology, address some limitations but may trade interpretability tied to κ_q.
Category:Statistical measures