OntoNotes

OntoNotes OntoNotes is a large, multilayer annotated corpus developed to support research in natural language processing and computational linguistics. It provides aligned annotations for syntax, proposition structure, coreference, and named entities across multiple genres and languages, enabling comparative evaluation of models that perform parsing, semantic role labeling, and entity resolution. The project involved collaboration among research groups at Princeton University, University of Pennsylvania, Brandeis University, and BBN Technologies and contributed to shared tasks and evaluation campaigns used by the DARPA community and industry partners.

Overview

OntoNotes was conceived as a response to limitations in earlier resources such as the Penn Treebank, ACE corpora, and the FrameNet project. It sought to provide broad-coverage, multilayer annotations to facilitate robust system development for tasks pursued by organizations like DARPA, National Science Foundation, and commercial labs including Google and IBM Watson. The corpus spans broadcast news, magazine text, conversational telephone speech, weblogs, and web data drawn from sources like New York Times, CNN, and the Wall Street Journal styles, supporting cross-genre generalization. Project leadership and contributors included researchers affiliated with Carnegie Mellon University, University of California, Berkeley, Stanford University, and industrial groups such as Microsoft Research and Nuance Communications.

Annotation and Ontology

Annotation in OntoNotes integrates syntactic trees inspired by the Penn Treebank formalism with a semantic ontology influenced by resources like WordNet and PropBank. Named entity annotation adopts label schemes comparable to those used in the ACE evaluations, distinguishing entities such as United Nations organizations, NATO bodies, and geopolitical entities like United Kingdom and People's Republic of China. Coreference links connect mentions across documents in ways that informed later initiatives at Google Research and within the ACL community. Predicate-argument structures align with semantic role schemes used in PropBank and evaluated at conferences such as NAACL, EMNLP, and COLING.

Corpus Composition

The English portion comprises over one million words annotated across genres: broadcast news similar to BBC transcripts, newswire reminiscent of Reuters and Associated Press, conversational speech echoing corpora like Switchboard, web data parallel to collections used by TREC, and newsgroup-style postings akin to Usenet. Chinese and Arabic subcorpora correspond to language-specific sources including content similar to Xinhua and Al Jazeera. Source materials and editorial practices drew on standards from LDC and were used by teams at MITRE and SRI International for interoperability. Inventories of entity types include persons comparable to Barack Obama and Angela Merkel, organizations similar to World Bank and European Commission, and locations like Paris and Tokyo to support multilingual named-entity normalization.

Applications and Impact

OntoNotes has been widely used to train and evaluate models for syntactic parsing, semantic role labeling, coreference resolution, and entity linking employed in systems by Facebook AI Research, Amazon Web Services, Apple, and research groups such as DeepMind and OpenAI. Shared tasks at SemEval, CoNLL, and TREC used OntoNotes-derived splits to benchmark progress in neural architectures like LSTMs, Transformers, and BERT-style pretraining developed by teams at Google Brain and Microsoft Research Asia. The dataset influenced industrial products including question answering in Amazon Alexa, information extraction for Bloomberg, and summarization research at Reuters and The New York Times Company. Academic impact includes citations in papers from University of Illinois Urbana-Champaign, Princeton University, and Columbia University exploring cross-lingual transfer and multilingual embeddings.

Criticisms and Limitations

Critiques of the corpus highlight coverage and annotation choices: genre balance has been questioned by scholars at University of Washington and University of Edinburgh; the ontology’s mapping to lexical resources like WordNet and frame inventories from FrameNet has prompted debate at workshops convened by ACL and IJCAI. Licensing and access restrictions limited participation by some industry groups and smaller academic labs compared to fully open datasets used by Hugging Face and Common Crawl. Further limitations include annotation consistency concerns raised by annotators with backgrounds at Columbia University and Brown University, and the representativeness of contemporary web and social media genres critiqued in venues like WWW and SIGIR. Despite these issues, the resource remains a pivotal benchmark referenced in evaluations by ACL Anthology, EMNLP proceedings, and major NLP toolkits such as those from Stanford NLP Group and spaCy.

Category:Corpora