doctest

doctest
Name	doctest
Paradigm	Unit testing, Documentation testing
Implemented in	Python
Author	(Guido van Rossum)
First appeared	2004
License	Python Software Foundation License

doctest is a Python module that extracts interactive Python sessions from documentation strings and verifies that the examples produce the expected output. It combines testing and documentation to help developers keep examples correct within projects such as Python (programming language), Django (web framework), NumPy, Pandas (software), and SciPy. The module is included in the Python Standard Library and is commonly used alongside tools like pytest, unittest, Sphinx (documentation generator), and Travis CI for continuous integration.

Overview

doctest allows authors to embed example sessions in docstrings or text files that resemble interactive interpreter input, then compare actual output to the documented output. Prominent projects such as IPython, Jupyter Notebook, Matplotlib, TensorFlow, and Scikit-learn have used or referenced doctest-style examples when illustrating usage. The approach supports literate programming advocates like Donald Knuth and tooling ecosystems exemplified by Read the Docs, GitHub, and Bitbucket that emphasize synchronized documentation and verification. doctest aims to lower barriers to writing tests by leveraging examples produced during tutorial creation, influenced by testing frameworks and documentation initiatives led by figures such as Guido van Rossum and organizations like the Python Software Foundation.

Usage

Developers place interactive-looking examples in function or module docstrings, or in external text files, then run the doctest runner to validate them. Common workflows integrate doctest with unittest test suites, or run doctest as part of pytest collection using plugins maintained by contributors from projects like NumPy and Pandas. Continuous integration platforms such as Travis CI, GitHub Actions, and CircleCI often execute doctest checks to ensure documentation examples remain accurate across contributions from developers at companies like Google, Microsoft, Facebook, and research labs like DeepMind or OpenAI. Documentation systems such as Sphinx (documentation generator) can extract and test code blocks using extensions inspired by doctest conventions.

Syntax and Assertions

doctest recognizes lines starting with the interpreter prompt (>>> and ...) followed by expected output lines. It supports value comparisons, exception checks, and ellipsis (...) for wildcard matching to handle non-deterministic output similar to assertions found in xUnit-style frameworks like JUnit and TestNG. Advanced features include option flags for whitespace normalization, ellipsis matching, and allowing for doctest failures to be reported in a manner similar to unittest tracebacks. Integration examples reference practices from projects such as NumPy, SciPy, and Matplotlib where complex numerical output or object representations require tolerant matching or custom repr handling.

Integration and Tooling

doctest integrates with Python tooling chains: it can be invoked as a script, embedded in test runners, or called programmatically from build systems used by organizations like Canonical (company), Red Hat, and IBM. IDEs and editors including PyCharm, Visual Studio Code, Sublime Text, and Vim can be configured to run or check doctest examples. Documentation builders like Sphinx (documentation generator) and publishing platforms such as Read the Docs provide extensions and hooks to run doctests during documentation builds. Large-scale scientific projects from institutions like CERN, NASA, and universities such as MIT and Stanford University often combine doctest examples with regression testing pipelines that use orchestration tools from Ansible, Jenkins, or Kubernetes-backed infrastructures.

Limitations and Criticisms

Critics note that doctest can be brittle for outputs that include memory addresses or timestamps, similar to concerns raised about example-based testing in communities around Apache Software Foundation projects. Large projects such as Django (web framework) and NumPy often prefer more feature-rich testing frameworks like pytest or unittest for complex test parametrization, mocking, and fixtures. doctest’s reliance on exact textual matching can lead to maintenance overhead, and its use in high-security or high-reliability contexts at institutions like NATO, European Space Agency, or FDA-regulated environments is limited because of stricter verification and traceability requirements. Tooling workarounds and community conventions from groups around SciPy and Pandas mitigate some concerns, but debates persist among maintainers from projects hosted on GitHub and committees like those of the Python Enhancement Proposal process.

Examples

A typical example in a docstring mirrors an interactive session: >>> sum([1, 2, 3]) 6 Advanced doctest usage in projects like NumPy or SciPy uses ellipsis and option flags to match approximate numerical output: >>> import math >>> math.sqrt(2) # doctest: +ELLIPSIS 1.4142... Exception checking is also supported and used in web frameworks such as Django (web framework): >>> int('a') Traceback (most recent call last): ValueError: invalid literal for int() with base 10: 'a'

History and Development

doctest originated as a module created for Python to encourage executable documentation, evolving through contributions from core developers associated with the Python Software Foundation and community contributors on platforms like SourceForge and later GitHub. Influences include literate programming pioneers such as Donald Knuth and testing philosophies from Kent Beck and the xUnit family. Over time, doctest’s design and usage patterns have been shaped by contributions from projects such as Django (web framework), NumPy, SciPy, and the broader open-source community including corporate contributors from Google, Microsoft, and Red Hat.

Category:Software testing