This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| SWDE | |
|---|---|
| Name | SWDE |
| Type | Dataset |
| Domain | Web archiving, Information retrieval |
| Creator | Unknown |
| Released | 2000s |
| Format | HTML, text, metadata |
| Size | ~1 million pages |
SWDE
SWDE is a web page dataset used for research in information retrieval, web mining, natural language processing, and digital preservation. It contains crawl-based collections drawn from multiple domains and is used alongside corpora such as the TREC collections, the ClueWeb datasets, and the Common Crawl. Researchers from institutions like Microsoft Research, Carnegie Mellon University, Stanford University, and University of California, Berkeley have used SWDE for experiments in search engine evaluation, web page classification, and named entity recognition.
SWDE provides a labeled corpus of web pages sampled across dozens of real-world websites including portals, e-commerce sites, news outlets, and academic pages. It is commonly cited in studies comparing extraction techniques against baselines from TREC Web Track, KDD Cup, and the INEX workshops. The collection includes HTML markup, textual content, and structural metadata that facilitate experiments in information extraction, document clustering, topic modeling, and feature engineering for machine learning systems developed at labs such as Google Research and Yahoo! Research.
SWDE emerged in the context of early 2000s efforts to create standardized web corpora after initiatives like the KDD Cup 1999 and the TREC web experiments. It was developed to fill gaps left by corpora such as the Wall Street Journal Corpus and the Reuters-21578 set by focusing on diverse site templates and real-world noise patterns seen in crawled pages from sources like Amazon (company), eBay, CNN, BBC News, The New York Times, and various academic institutions. Subsequent work at research centers including MIT and ETH Zurich adapted SWDE-style sampling to study template detection, wrapper induction, and robustness of parsers used by projects at IBM Research and Microsoft.
The SWDE collection is organized by site and page type, with each site represented by a set of pages covering multiple templates (product pages, article pages, category pages). Pages include raw HTML, extracted text, DOM trees, and annotation files that mark fields such as title, author, price, and date—annotations used in comparative evaluations with datasets like MUC-style corpora and OntoNotes. The dataset’s schema supports linking to external resources indexed in systems like Apache Lucene and Elasticsearch and aligns with metadata conventions used by Dublin Core and web archiving efforts at institutions such as the Internet Archive.
Researchers apply SWDE for tasks including wrapper induction, template detection, and field extraction as benchmarks for algorithms proposed in venues like SIGIR, KDD, The Web Conference, and ACL. It has been used to evaluate machine learning models such as conditional random fields developed at University of Massachusetts Amherst and deep neural architectures inspired by work from Google DeepMind and Facebook AI Research. Other use cases include experiments in duplicate detection relevant to projects at Pinterest and Twitter, layout understanding comparable to systems from Adobe Research, and dataset augmentation for training named-entity taggers used in production by LinkedIn and Bing.
SWDE serves as a benchmark in empirical comparisons published in journals and conferences like ACM Transactions on Information Systems, Information Retrieval Journal, SIGMOD, and ICDE. Metrics reported in evaluations typically include precision, recall, F1-score, and field-level accuracy, and comparisons are often made against baselines derived from TREC tasks and newer corpora such as ClueWeb09 and ClueWeb12. The dataset has appeared in shared tasks and challenge tracks that attracted participants from labs including University of Illinois Urbana–Champaign, Princeton University, and University of Washington.
Critiques of SWDE note limited coverage relative to web-scale datasets like Common Crawl and potential biases introduced by the selected websites (commercial portals, major news outlets, and specific domain topologies). Scholars referencing work from Stanford NLP Group and Berkeley NLP highlight concerns about annotation consistency, representativeness for long-tail sites, and temporal drift when compared with live web snapshots held by the Internet Archive. Additional criticism points to the dataset’s licensing and access constraints contrasted with fully public resources such as Wikipedia dumps and openly licensed corpora used in competitions like ImageNet (for multimodal research).
Datasets related to SWDE include the TREC Web Track collections, ClueWeb09, ClueWeb12, Common Crawl, and specialized corpora like WebTables and the OpenWebText corpus. Successor efforts and resources that extend SWDE’s aims are found in projects by Common Crawl, the Internet Archive, and curated benchmarks produced by groups at Allen Institute for AI and Hugging Face. Comparative studies often reference corpora such as Kaggle competition datasets, the CORD-19 collection for domain-specific work, and multilingual resources maintained by organizations like ELRA and LDC.
Category:Datasets