CoNLL Shared Task

CoNLL Shared Task
Name	CoNLL Shared Task
Discipline	Computational linguistics
Frequency	Annual/Biennial
First	1990s
Organiser	Association for Computational Linguistics
Country	International

CoNLL Shared Task The CoNLL Shared Task is a recurring competitive evaluation in computational linguistics that has driven advances in natural language processing, information extraction, and machine learning through community-organized challenges. It brings together research groups from institutions such as Massachusetts Institute of Technology, Stanford University, University of Cambridge, University of Edinburgh, and industry labs like Google Research, Microsoft Research, and Facebook AI Research to benchmark systems on standardized datasets and tasks.

Overview

The Shared Task fosters collaboration among participants affiliated with organizations including Association for Computational Linguistics, European Research Council, National Science Foundation, DeepMind, and OpenAI by presenting tasks influenced by real-world applications from partners such as Wikimedia Foundation, Allen Institute for AI, Amazon (company), IBM Research, and Apple Inc.. Typical events feature contributions from teams at Carnegie Mellon University, University of Oxford, Tsinghua University, Peking University, ETH Zurich, Universität des Saarlandes, Johns Hopkins University, University of Washington, University of Tokyo, Seoul National University, University of Melbourne, University of Toronto, McGill University, University of British Columbia, National University of Singapore, Imperial College London, École Polytechnique Fédérale de Lausanne, INRIA, Max Planck Institute for Informatics, SRI International, Baidu Research, Alibaba Group, Tencent AI Lab, Naver Corporation, SAP SE, and Siemens AG.

History and Editions

The Shared Task series evolved from early competitions tied to conferences such as COLING, ACL (Association for Computational Linguistics), EMNLP, NAACL, and EACL, tracing roots to initiatives at European Summer School in Logic, Language and Information and projects funded by bodies like Horizon 2020 and Marie Skłodowska-Curie Actions. Prominent editions focused on problems originating in datasets and campaigns associated with OntoNotes, ACE (Automatic Content Extraction), SemEval, TAC (Text Analysis Conference), CLEF, IWSLT, TREC, BioNLP, Shared Task on Parsing, and specific multilingual benchmarks for languages studied at SALT (Syntax and Semantics), LREC (Language Resources and Evaluation Conference), and national labs such as NIST. Over time notable methodological shifts mirrored breakthroughs at University of California, Berkeley, Columbia University, New York University, University of Pennsylvania, Princeton University, Yale University, Brown University, Duke University, University of Illinois Urbana-Champaign, and Rutgers University.

Tasks and Datasets

Past editions have encompassed named entity recognition, dependency parsing, semantic role labeling, coreference resolution, information extraction, and joint tasks using corpora influenced by OntoNotes, Penn Treebank, Universal Dependencies, Wikipedia, Common Crawl, Gigaword, Europarl Corpus, and domain corpora like PubMed, arXiv, CORD-19, Reuters-21578, and Brown Corpus. Competitions have attracted systems leveraging resources developed at Linguistic Data Consortium, ELRA, Meta AI, Hugging Face, AllenNLP, and model families originating from research at Google Brain, DeepMind, OpenAI, Facebook AI Research, Microsoft Research Asia, and labs tied to Stanford NLP Group, Berkeley NLP, CMU Language Technologies Institute, and Johns Hopkins CLSP.

Evaluation Metrics and Methodology

Standard evaluation protocols have adopted precision, recall, F1, labeled attachment score, and exact match metrics established or popularized in workshops at ACL (Association for Computational Linguistics), EMNLP, NAACL, EACL, and competitions like TREC and SemEval. Organizers often employ blind test sets curated by teams affiliated with LDC (Linguistic Data Consortium), ELRA, NIST, European Language Grid, and research groups from University of Cambridge, University of Edinburgh, Stanford University, Princeton University, MITRE Corporation, and SRI International. Methodological standards reflect reproducibility efforts championed by initiatives at NeurIPS, ICML, ICLR, and policy discussions involving European Commission and US National Institutes of Health.

Notable Results and Impact

Outcomes from Shared Task editions have catalyzed advances informing commercial products and academic research at Google, Microsoft, Amazon Web Services, Apple Inc., IBM, Meta Platforms, Inc., and startups incubated through Y Combinator and Techstars. Breakthrough systems that first excelled in these competitions were later linked to model families and toolkits such as BERT, ELMo, GPT (language model), Transformer (machine learning model), spaCy, Stanford CoreNLP, Moses (software), MALLET, and UIMA. The Shared Task has influenced curricula at universities including Massachusetts Institute of Technology, Stanford University, Carnegie Mellon University, University of Oxford, and has driven citations in high-profile conferences like NeurIPS, ICML, ACL (Association for Computational Linguistics), EMNLP, and COLING.

Organizers and Sponsorship

Organizing committees typically include members from Association for Computational Linguistics, research centers at Microsoft Research, Google Research, Facebook AI Research, DeepMind, academic labs at Stanford University, University of Cambridge, University of Edinburgh, Carnegie Mellon University, and funding or sponsorship from agencies and companies such as National Science Foundation, European Research Council, Google, Microsoft, Amazon, IBM, Facebook, Allen Institute for AI, Google DeepMind, NVIDIA, Intel Corporation, and publishing partners like ACL Anthology and Springer Nature.

Category:Computational linguistics competitions