PyWb

PyWb
Name	PyWb
Developer	Webrecorder / Rhizome / Various contributors
Released	2015
Programming language	Python
Operating system	Cross-platform
License	Open source

PyWb

PyWb is a Python-based web archiving toolkit used for replaying, indexing, and serving archived web content. It is associated with archival projects and institutions concerned with long-term preservation and access to web captures, and it interoperates with standards and tools from libraries, museums, and digital preservation communities. PyWb powers replay interfaces and tooling that integrate with large-scale archives, research initiatives, and cultural heritage platforms.

Introduction

PyWb was created to provide a flexible replay and indexing layer for archived web content, addressing challenges faced by organizations such as the Library of Congress, Internet Archive, British Library, National Library of Australia, and university libraries engaged in web archiving. The project fits into ecosystems involving Web Archiving initiatives, interoperability efforts like WARC (file format), standards bodies including the International Internet Preservation Consortium and collaborations with projects such as Memento (web archiving), OpenWayback, Heritrix, and HTTrack. Institutions and projects in the cultural heritage sector, including the Smithsonian Institution, Museum of Modern Art, Harvard Library, Stanford University Libraries, Yale University, Columbia University, and the New York Public Library use or reference components that interoperate with PyWb.

Architecture and Components

The architecture separates capture, indexing, storage, and replay, enabling integration with diverse stacks used by organizations like Amazon Web Services, Google Cloud Platform, European Nucleotide Archive-style large data repositories, and institutional storage clusters. Core components include a replay engine for HTTP transactions, an indexer compatible with WARC (file format) records, a query API used by portals at institutions such as National Archives (United Kingdom), and middleware that can be deployed alongside reverse proxies like Nginx, Apache HTTP Server, or served within application servers used at MIT and Princeton University. PyWb’s modular layout allows linking to indexing backends used in projects like Elasticsearch, Solr, and database systems from vendors including PostgreSQL and MongoDB.

Features and Functionality

PyWb implements replay features such as request rewriting, link rewriting, and timegate-style access aligning with Memento (web archiving). It supports full-text search integration via Elasticsearch and faceted discovery patterns employed by digital library interfaces at DPLA and national libraries. Functionality includes handling of complex JavaScript-driven pages similar to approaches used by Puppeteer, Selenium (software), and headless browsers championed by engineering teams at Google, Mozilla, and Microsoft. PyWb also provides APIs and CLI tools compatible with workflows from preservation platforms like Archivematica, content management systems used by Drupal and WordPress, and ingestion pipelines adopted by research libraries.

Deployment and Use Cases

Deployments range from single-server setups for academic projects at University of Oxford and University of Cambridge to cloud-hosted services used by organizations such as Internet Archive and national institutions like the Library and Archives Canada. Use cases include scholarly research supported by projects at Harvard and University of California, Berkeley, legal evidence preservation as seen in litigation archives, journalism archiving initiatives at outlets like The New York Times and The Guardian, and cultural heritage exhibitions curated by institutions including the Victoria and Albert Museum and Tate Modern. PyWb is used in workflows for event-based archiving for major events such as elections, sporting events tied to FIFA World Cup, and crisis archiving during natural disasters documented by agencies like United Nations offices.

Development and Community

Development began within communities around Webrecorder and Rhizome, with contributions from academic labs, national libraries, and open source developers associated with organizations such as GitHub, Apache Software Foundation projects, and university digital preservation groups at Cornell University and University of Toronto. The community interacts through code hosting platforms used by GitHub and issue trackers familiar to contributors from Mozilla Foundation and research centers. Funding and collaboration have involved grants from bodies like the Andrew W. Mellon Foundation, partnerships with standards bodies including the Open Web Application Security Project in adjacent work, and joint efforts with projects such as LOCKSS and Perseus Digital Library-style initiatives.

Security and Privacy Considerations

Security and privacy considerations involve handling sensitive captures, access controls used by institutions like National Archives (United States), and compliance with legal frameworks such as General Data Protection Regulation and national access laws. Deployers must consider cross-origin concerns similar to those addressed by Content Security Policy standards, replay fidelity versus sanitization policies practiced by libraries, and mitigations for serving archived JavaScript that could interact with live infrastructures as studied in security research from groups at Carnegie Mellon University and SRI International.

Licensing and Availability

PyWb is distributed as open-source software, enabling reuse by cultural heritage institutions, research labs, and commercial vendors. Its code is available on common code collaboration platforms used by projects like GNU Project-adjacent repositories and attracts contributors from institutions including Stanford University Libraries, MIT Libraries, and digital preservation initiatives funded by entities like the National Endowment for the Humanities.

Category:Web archiving software