Testing Your Package README Code Snippets

Package maintainers use a README to show readers how to install a package, configure it, and complete a task. If the package changes but its README example does not, readers follow instructions that fail or give the wrong result. The package’s regular tests can still pass.

A documentation test runs an example in the documentation and checks its result. A thorough README check runs the instructions readers see. It also loads documented settings through the package’s public interface and verifies the result the README promises. Installation checks should use the built package file that users install, rather than relying only on the copy in the repository.

Run the examples readers see

Tie tests to the README’s actual code whenever possible. If you copy an example into a separate test by hand, the test and the README can drift apart, leaving a passing test beside a broken instruction.

The right tool depends on the language and the README format. Python’s doctest runs interactive examples in text files and compares their output with the expected result. That does not mean it automatically extracts every fenced code block from Markdown. Similarly, pytest’s documented --doctest-glob option selects matching text files; it does not establish that pytest reads arbitrary fenced blocks from README.md. Check which snippets your chosen runner actually collects.

Rust’s rustdoc provides a way to include a README in documentation tests. Those tests compile and run examples, and Rustdoc supports setup lines that tests can compile while hiding them from rendered documentation. Use the mechanism that fits your language, and confirm that it exercises the published text rather than a manually maintained copy.

Install the built package

A test that imports code from the repository does not prove that the package users download includes the same files or dependencies. Build the release package, install it in a separate environment like the one a package user would use, and run the README’s documented import or command there.

For Python, the packaging flow distinguishes a source distribution from a wheel. If a release publishes both, test both: they can contain different files or follow different installation paths. Packages with compiled extensions also need the relevant platform-specific wheels tested.

For npm packages, npm pack creates a tarball from the package folder. Install that tarball in a separate consumer project, then run the README’s public import or command. Running the check outside the repository helps reveal accidental imports from local source files.

The check should verify more than successful installation. Confirm that the consumer can find the installed package and complete one small operation through the documented public interface. A successful local artifact test does not, by itself, verify registry publication, release permissions, or every supported platform.

Check configuration by its effects

A configuration example can parse successfully while setting an option the package ignores. Test the documented configuration through the same public loading mechanism that readers are told to use, then assert the behavior the README claims.

For example, if the README says a setting enables a feature, load that configuration and verify the feature’s observable result. If the example combines a configuration file with an environment variable or command-line flag, test that combination and the documented precedence. Testing each value alone will not show whether the settings work together as the README describes.

Keep the tested configuration tied to the README. A test that duplicates the settings in a separate fixture can become stale, so either extract the example from the README or make the fixture the single source used to render the documentation. Use deterministic local inputs for examples that otherwise depend on credentials, external services, or machine-specific paths.

Assert the promised usage result

A snippet that exits without an error has only shown that it ran. It has not shown that it returned the right value, created the expected file, or changed state as the README claims. Add an assertion or output check for the result that matters to the user.

Prefer checks on stable behavior, such as a returned field or a created file, over snapshots of lengthy output. Python doctest compares actual output with expected output, and its documentation describes how whitespace and ellipsis options affect matching. Relaxed matching helps with incidental variation, but broad patterns can accept incorrect output. Keep any relaxed match narrow.

For Rust examples, a test that compiles and runs without panicking still needs an assertion to verify any claimed value or outcome. Rustdoc’s no_run option checks compilation without executing the example, so it cannot establish runtime behavior.

Label examples that cannot run

Some README examples require a live service, credentials, or an action that should not run in ordinary CI. Replace external dependencies with deterministic local substitutes when that still tests the documented behavior. Keep a separate integration check for a real external boundary when the package depends on one.

When an example cannot run safely or reliably, mark that limitation clearly. Rustdoc’s ignore attribute excludes an example from testing, while no_run checks compilation but skips execution. An ignored snippet proves nothing about whether it compiles or runs; a no_run snippet proves only that it compiles. Make those limits visible in the surrounding documentation, and avoid treating excluded examples as verified usage.

Run README checks in CI

Run documentation tests alongside the package’s regular tests on pull requests. This catches README instructions that no longer match a changed public API or configuration option before release. Add a separate artifact check where the pipeline builds the release package. It then installs the package in a consumer environment and runs a short documented user journey. GitHub’s Actions documentation describes workflow jobs that run repository checks and report their outcomes.

Keep the checks focused on the main ways readers use the package: install it, configure it, and perform its central task. A passing test then provides specific evidence about those examples. It does not prove that every README code block works, that every platform behaves identically, or that publishing will succeed.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com