Corpora of Contemporary American English

Corpora of Contemporary American English
Name	Corpora of Contemporary American English
Type	Corpus
Location	United States
Created	Late 20th–21st century
Languages	English (American)
Size	Millions–billion+ words
Creator	Academic and commercial projects

Corpora of Contemporary American English Corpora of Contemporary American English are large collections of modern American English texts compiled for linguistic, computational, and cultural study. They emerge from projects associated with institutions such as Harvard University, Stanford University, Massachusetts Institute of Technology, Columbia University and organizations including Google, Microsoft and National Science Foundation. These corpora underpin research by groups at University of California, Berkeley, University of Pennsylvania, New York University, University of Michigan and Yale University.

Overview and History

Early modern American English corpora trace lineage to initiatives at Brown University, Brown Corpus, and later efforts by researchers at Corpus of Contemporary American English, Linguistic Data Consortium and projects supported by Andrew W. Mellon Foundation and National Endowment for the Humanities. Influential milestones involved collaborations with Oxford University Press, Cambridge University Press and technical contributions from teams at Bell Labs, IBM and AT&T. Growth accelerated with digitization campaigns at Library of Congress, National Archives and Records Administration and media digitization by The New York Times Company, The Washington Post and broadcasters like National Public Radio.

Composition and Design

Corpus composition often balances genres drawn from publishers such as Penguin Random House, HarperCollins, newspapers like Los Angeles Times, Chicago Tribune, and transcripts from broadcasters including CBS News, NBC News, ABC News and CNN. Design frameworks reference standards from American National Corpus projects and guidance from committees at Modern Language Association, Association for Computational Linguistics and International Organization for Standardization. Sampling strategies incorporate text from legal sources such as United States Code and court opinions from Supreme Court of the United States, alongside material from Library of Congress digital collections and datasets produced by Wikimedia Foundation.

Corpus Annotation and Metadata

Annotation schemes adopt tagsets influenced by work at Pennsylvania State University and standards like those developed by Princeton University and the Penn Treebank. Metadata fields document provenance using identifiers tied to repositories such as Digital Public Library of America, HathiTrust, and archives maintained by Smithsonian Institution and National Institutes of Health. Techniques for part-of-speech tagging, named-entity recognition and syntactic parsing build on research from labs at Carnegie Mellon University, Johns Hopkins University and University of Illinois Urbana-Champaign.

Access, Licensing, and Tools

Access models range from open initiatives similar to projects hosted by Project Gutenberg and Creative Commons to licensed distributions managed by Linguistic Data Consortium and commercial providers like LexisNexis and ProQuest. Tooling ecosystems draw on software from Google Research, Facebook AI Research, and open-source projects maintained by communities around Apache Software Foundation, GitHub and The Apache Software Foundation. Platforms for corpus querying and concordancing reference systems developed at Mark Davies (corpus linguist)’s lab, software influenced by AntConc creators, and interfaces inspired by repositories at Oxford Text Archive.

Applications and Research Uses

Researchers apply these corpora in studies published by journals such as Language, Computational Linguistics and Journal of English Linguistics and in projects at National Institutes of Health, Defense Advanced Research Projects Agency and European Research Council funded collaborations. Applications include training language models used by companies like OpenAI and DeepMind, sociolinguistic analysis involving scholars from University of Texas at Austin and University of California, Los Angeles, and legal language studies informing decisions at Supreme Court of the United States and policymaking bodies in United States Congress.

Limitations and Criticisms

Critiques cite representativeness issues raised by scholars at Princeton University, Harvard University, and Rutgers University and concerns about copyright raised by organizations such as Authors Guild and Association of American Publishers. Ethical and privacy debates reference rulings and guidance from United States Court of Appeals, regulatory frameworks at Federal Trade Commission, and statements by Electronic Frontier Foundation and American Civil Liberties Union. Methodological limits link to reproducibility discussions in venues like Nature and Science.

Notable Derived Resources and Subcorpora

Prominent derived resources and subcorpora include genre-specific collections modeled after corpora produced by Mark Davies (corpus linguist)’s lab, specialized subsets influenced by datasets from COCA-style initiatives, annotated treebanks echoing standards from the Penn Treebank, and speech-focused subcorpora comparable to collections from Linguistic Data Consortium and speech archives at British Library. Other notable spin-offs relate to digital humanities projects at University of Virginia, historical corpora curated by Yale University and multilingual parallel corpora built in collaborations with European Commission programs.

Category:Corpora