Meet Doctests: The Shy Giants of Python Testing

Python is renowned for its simplicity, versatility, and extensive ecosystem of powerful tools. But even experienced Pythonistas can overlook some of the most useful gems in the standard library. One such hidden treasure is the doctest module, a deceptively simple tool for writing automated tests.

At first glance, doctests might seem like little more than a quirky way to embed interactive examples in your documentation. But don‘t be fooled by their unassuming appearance. With a little creativity, doctests can become an indispensable part of your testing arsenal, catching bugs and verifying the behavior of your code in a wonderfully lightweight and readable way.

In this post, we‘ll dive deep into the world of doctests, exploring their history, dissecting best practices, and showcasing real-world examples. By the end, you‘ll see why doctests deserve a place alongside more traditional testing frameworks in any well-rounded Python project. Let‘s get started!

What are doctests?

At its core, a doctest is a way to write a test inside a function‘s docstring. It looks something like this:

def greet(name):
    """
    Generate a personalized greeting.

    >>> greet("Guido")
    ‘Hello, Guido!‘

    >>> greet("monty python")
    ‘Hello, Monty Python!‘
    """
    return f"Hello, {name.title()}!"

The >>> lines represent an interactive Python session, showing the expected input and output. When you run this code with python -m doctest myfile.py, the doctest module will execute those snippets and compare the actual output to what‘s written in the docstring. If they match, the test passes. If not, you‘ll get a detailed failure message.

This approach has several notable benefits:

1. Tests and docs are one and the same. Doctests serve as both documentation and test cases, ensuring your examples stay in sync with your implementation.

2. Test code lives alongside production code. No need to maintain a separate set of test files – everything is neatly organized in one place.

3. Tests are readable and writable by non-programmers. The interactive format is intuitive even for people unfamiliar with the intricacies of testing frameworks.

4. Running tests is trivial. No complex test harnesses or dependencies required – just execute the file with the doctest module.

Of course, there are trade-offs to using doctests. They lack many features of more full-featured testing tools, like setup/teardown logic, parameterized tests, and rich assertion utilities. And by nature, they‘re tightly coupled to the specific implementation of each function.

But in many cases, the simplicity and convenience of doctests more than makes up for their limitations. Let‘s take a closer look at how they work and how to use them effectively.

A brief history of doctests

The idea of executable documentation has been around for decades, with origins in languages like Lisp and Smalltalk. But Python‘s doctest module, introduced in 2001 by the inimitable Tim Peters, was one of the first widely adopted implementations for a mainstream language.

The initial motivation was straightforward: Tim was frustrated with the state of Python‘s documentation, which was often unclear, inconsistent, or outright incorrect. He envisioned a system where examples could be automatically verified, ensuring they were always accurate and up to date. Thus, doctests were born.

Over the years, doctests have become an integral part of the Python ecosystem. They‘re used extensively in the standard library, as well as in popular third-party packages like NumPy, requests, and Flask. While they haven‘t replaced traditional testing methodologies, they‘ve carved out a valuable niche as a lightweight way to verify and document code.

Recent versions of Python have introduced several notable improvements to doctests, such as better support for handling exceptions, floating-point comparisons, and multi-line strings. And with the rise of tools like pytest and tox, it‘s easier than ever to integrate doctests into a modern testing workflow.

Writing effective doctests

So what makes a good doctest? The key is to treat them like any other kind of test: focus on clarity, concision, and coverage.

Here are some tips for writing doctests that are both readable and effective:

• Keep examples simple and focused. Each doctest should demonstrate a single behavior or use case, with minimal setup and teardown.

• Cover edge cases and error conditions. Doctests are a great place to showcase how your code handles unusual inputs or failure modes.

• Be explicit about types and values. Include the expected types of arguments and return values to make the contract of your function clear.

• Use ellipses (…) for irrelevant or variable output. This includes things like stack traces, object reprs, or generated timestamps that may change across runs.

• Organize and group related tests. Use blank lines and comments to break up your doctests into logical sections, making them easier to scan and understand.

Here‘s an example that demonstrates these principles:

def find_anagrams(word, dictionary):
    """
    Find all anagrams of a word in a dictionary.

    Args:
        word (str): The word to find anagrams for.
        dictionary (list of str): The list of valid words.

    Returns:
        list of str: All anagrams of the given word in the dictionary.

    >>> find_anagrams("listen", ["enlists", "google", "inlets", "banana"])
    [‘inlets‘]

    >>> find_anagrams("pencil", ["licnep", "circular", "pupil", "nilcpe", "leppnec"])
    [‘licnep‘, ‘nilcpe‘]

    >>> find_anagrams("", ["foo", "bar"])
    []

    >>> find_anagrams("dog", [])
    []
    """
    return [
        d for d in dictionary
        if sorted(word.lower()) == sorted(d.lower())
    ]

See how each example is compact, yet descriptive? The docstring includes all the essential information a user would need to understand the purpose and behavior of the function, while also providing a solid set of test cases covering various scenarios.

Real-world examples and use cases

Doctests truly shine when used to verify the behavior of pure, stateless functions. But that doesn‘t mean they‘re limited to toy examples or trivial code. With a bit of creativity, you can leverage doctests to test a surprisingly wide range of real-world tasks.

Consider this function that converts a Markdown-style link to an HTML anchor tag:

def markdown_to_html(markdown):
    """
    Convert a Markdown-style link to an HTML anchor tag.

    >>> markdown_to_html("[Google](https://www.google.com)")
    ‘<a href="https://www.google.com">Google</a>‘

    >>> markdown_to_html("[A link with spaces](https://www.example.com/a link with spaces)")
    ‘<a href="https://www.example.com/a%20link%20with%20spaces">A link with spaces</a>‘

    >>> markdown_to_html("[A link with a [nested] link](https://www.example.com)")
    ‘<a href="https://www.example.com">A link with a [nested] link</a>‘

    >>> markdown_to_html("[A link with no URL]()")
    Traceback (most recent call last):
        ...
    ValueError: Invalid Markdown link format: [A link with no URL]()
    """
    match = re.match(r‘\[(.*)\]\((.*)\)‘, markdown)
    if not match:
        raise ValueError(f"Invalid Markdown link format: {markdown}")

    text, url = match.groups()
    url = urllib.parse.quote(url, safe=‘/:‘)

    return f‘<a href="{url}">{text}</a>‘

The doctests here serve as a concise, yet comprehensive, specification for how the function should behave. They demonstrate the expected output for various input patterns, as well as error handling for malformed input. A developer could read this docstring and quickly grasp the full scope of the function, without having to dive into the implementation details.

You can even use doctests to verify behavior of functions that depend on external interfaces, like databases or web APIs. While mocking is often a better choice for complex scenarios, judicious use of monkeypatching can enable self-contained doctests that still provide meaningful verification:

import requests

def get_github_user(username):
    """
    Retrieve the Github profile data for a given username.

    >>> import requests
    >>> original_get = requests.get

    >>> class MockResponse:
    ...     def __init__(self, json_data, status_code):
    ...         self.json_data = json_data
    ...         self.status_code = status_code
    ...     def json(self):
    ...         return self.json_data

    >>> def mock_get(*args, **kwargs):
    ...     return MockResponse({"login": "octocat", "name": "The Octocat"}, 200)

    >>> requests.get = mock_get
    >>> get_github_user("octocat")
    {‘login‘: ‘octocat‘, ‘name‘: ‘The Octocat‘}

    >>> requests.get = original_get  # Restore original requests.get
    """
    response = requests.get(f"https://api.github.com/users/{username}")
    if response.status_code == 200:
        return response.json()
    else:
        raise ValueError(f"Failed to retrieve data for user {username}")

Here, we‘re temporarily replacing the requests.get function with a mock that returns a hard-coded response. This allows us to write a self-contained test without actually hitting the GitHub API. Note that we also carefully restore the original requests.get at the end, to avoid interfering with any other tests.

Of course, this specific example is a bit contrived. In a real project, you‘d probably structure this code differently to allow for easier mocking (perhaps injecting the requests.get function as a dependency). But it illustrates the core idea: with a little ingenuity, you can use doctests to verify more than just simple, pure functions.

Integrating doctests into your workflow

So you‘re sold on the benefits of doctests and you‘ve started sprinkling them throughout your codebase. Now what?

First, you‘ll need a way to run your doctests. The simplest approach is to add a block like this to the end of each file:

if __name__ == "__main__":
    import doctest
    doctest.testmod()

This will execute all the doctests in the current module when you run it with python myfile.py. You can also run doctests for an entire directory with python -m doctest mydir/.

However, for any non-trivial project, you‘ll probably want to use a more sophisticated test runner. Most modern Python testing frameworks, including unittest, nose, and pytest, have built-in support for discovering and running doctests alongside your regular test suite. For example, to run doctests with pytest, you‘d simply pass the --doctest-modules flag:

$ pytest --doctest-modules

This will scan all your source files for doctests and run them just like any other test. You can even put your doctests in separate files, allowing you to test your code without cluttering up your source files.

Integrating doctests with continuous integration is also straightforward. Most CI platforms, like Travis, CircleCI, and Github Actions, have pre-configured Python environments that include pytest or nose. So adding doctests to your CI pipeline is often just a matter of updating your test command to include the appropriate flags.

The beauty of this approach is that doctests become just another part of your testing strategy. They live alongside your unit tests, integration tests, and other verification methods, providing an additional layer of checks and documentation. And because they‘re discovered and run automatically, they remain front-and-center as your codebase evolves, rather than being relegated to an afterthought.

The future of doctests

Despite their long history and deep integration with the Python ecosystem, doctests remain a somewhat underappreciated tool. Many developers still view them as little more than a quirky feature for basic examples, rather than a powerful verification and documentation technique.

That said, there‘s a growing recognition of the value of executable documentation and literate programming techniques. Tools like Jupyter Notebook and R Markdown have shown the power of blending prose, code, and results into a cohesive whole. And projects like Rust‘s rustdoc and Elixir‘s ExDoc have taken the idea of doctests to new levels, with support for advanced features like custom formatters and property-based testing.

In the Python world, there are ongoing efforts to modernize and expand the doctest module. For example, Python 3.8 added support for doctest.NORMALIZE_WHITESPACE, allowing doctests to ignore insignificant differences in whitespace. And there are proposals to add new directives for controlling doctest behavior, like IGNORE_RESULT_IN_EXCEPTION and NUMBER, which would make it easier to write doctests for functions that return unpredictable or irrelevant values.

At the same time, there‘s a healthy debate about the role of doctests in a modern testing strategy. Some argue that they blur the line between tests and documentation, making both harder to maintain. Others point out that they can be fragile and hard to debug, especially for more complex scenarios.

Ultimately, like any tool, doctests are not a silver bullet. They‘re not a replacement for traditional unit tests, integration tests, or other verification methods. But when used judiciously and in concert with other techniques, they can be a remarkably effective way to improve the quality and clarity of your code.

Conclusion

Doctests are one of Python‘s hidden gems: a deceptively simple, yet powerful tool for testing and documenting your code. By embedding executable examples in your docstrings, you can create a living specification that evolves alongside your implementation.

While doctests are not without their limitations and trade-offs, they occupy a valuable niche in the Python testing ecosystem. For verifying the behavior of straightforward, pure functions, they offer an unbeatable combination of simplicity, expressiveness, and ease of use. And when integrated into a larger testing strategy, they can help ensure that your code remains correct and well-documented as it grows and changes.

So if you‘re not already using doctests in your Python projects, give them a try. Start small, perhaps by adding a few examples to your most commonly used utility functions. Over time, you may find that they become an indispensable part of your development workflow, catching bugs and clarifying expectations in a way that traditional tests can‘t match.

With a little creativity and care, doctests can help you write more reliable, maintainable, and understandable code – one example at a time. Happy testing!