Mutation testing

A test suite that passes proves the code runs. It does not prove the tests would notice if the code were wrong. Mutation testing checks that: change one thing in the source, run the tests, and see whether anything fails. A change that nothing notices is a survivor, and it marks a rule the tests do not actually hold.

cargo mutants does the changing. This repository adds one thing on top of it: a sweep you can switch off and pick up again.

Why the sweep needs a harness

A full sweep of this workspace is 4 733 mutants and several days of machine time. cargo mutants has no resume: interrupt it and the whole run is lost.

scripts/mutation_sweep.py splits the work into chunks, records each finished chunk, and starts the next session where the last one stopped. The most a power cut costs is one chunk.

Using it

scripts/mutation_sweep.py status
scripts/mutation_sweep.py run --minutes 120 --cores 4,5 --jobs 2
scripts/mutation_sweep.py survivors

run stops cleanly when the budget is spent, or on Ctrl-C after the chunk it is on. Run it again and it continues. With no --minutes it runs to the end.

option	what it is for
--minutes	stop after this long. 0 runs to the end.
--cores	pin the work, e.g. 4,5, so the machine stays usable
--jobs	mutants tested at once
--chunk	mutants per chunk, on a NEW sweep only. Default 50.

Progress goes to the terminal; the full cargo-mutants output goes to sweep.log in the state directory, which run names.

State lives in ~/.config/muundo/mutation-sweep/. That is a durable volume: /tmp and /var/tmp are on the container's overlay and are lost when it is rebuilt.

What invalidates a sweep

Any edit to a tracked file. A mutation result belongs to one exact tree: the mutant is a change to a specific line, and the verdict is the reaction of a specific test suite. Move either and the recorded verdict is about code that no longer exists.

So the harness fingerprints the content of every file git tracks, and resumes only a sweep whose fingerprint still matches. Edit anything and status says so, names the sweeps it is not resuming, and offers to start a new one. The old results are kept under their own fingerprint, not mixed in.

The practical consequence: a multi-day sweep and active development do not mix. Run the sweep on a tree you are leaving alone.

The chunk size

A chunk costs one full build before its first mutant, and that cost is paid again for every chunk. Small chunks lose less to an interruption and spend more on rebuilds; large chunks the other way. 50 is the default because it puts the rebuild at roughly a tenth of the chunk.

The test timeout, and how it wasted a day

A mutant that makes the tests hang has to be cut off, so every run has a test timeout. Getting it wrong is expensive in a way that does not look like a failure: a run cut short is recorded as timeout, which is not a verdict — the machine spent the time and learned nothing.

Two ways to get it wrong, both seen here on 2026-08-26:

• A fixed timeout that is too tight. A sweep ran with --timeout 300 against a suite that takes 96 s alone and far longer with four mutants sharing the cores. 451 of 760 results came back as timeouts.
• Letting cargo-mutants derive it per chunk. It sizes the timeout from a baseline it runs itself, and that baseline covers only the packages the chunk touches. A chunk of cli mutants measures 6 s and sets a 41 s timeout; the next chunk's mutants are in core, whose suite takes 96 s, and every one of them is cut short.

The harness measures the whole workspace suite ONCE per sweep and passes the same generous timeout to every chunk. status prints the number it measured. --timeout-multiplier (default 6) is the headroom for mutants sharing cores; --timeout overrides it outright.

Reading a survivor

A survivor is not automatically a defect. Three kinds turn up:

• A real gap. The tests never exercise the rule. Write the test.
• An equivalent mutant. The change computes the same thing — | and ^ on disjoint flag bits, release(0) and fetch_sub(0). Nothing can catch it. Record why, in a comment beside the code.
• A field that carries nothing today. The line is real but the value it handles is always empty. Leave it and say so.

Where survivors are left in place deliberately, the reason is written next to the code as a MUTATION NOTE.

Just the change you made

A full sweep is for a release. During normal work:

cargo mutants --in-diff <(git diff origin/main)

That mutates only the lines you touched, and takes minutes.