What the first runs showed.
Two small modules, tested by an AI agent working text-only, checked by a harness, then attacked by mutations written by other models. Those runs were on our own issues in our own repository, and a person approved every step that left the machine. The test files were reviewed by two other models, not yet by a maintainer. A later section runs other projects' existing tests; nothing was sent to those projects.
Five things, one of them ours to own.
- Three of the five tasks we filed were wrong. We checked for a test file next to each module and missed tests elsewhere. A reviewer found it. The label came off those three, and the harness now searches every test for imports.
- The agent's first draft failed 5 of 19 checks. It cannot run code, so its expected values were its own arithmetic. The harness ran the real tests and caught it.
- Those failures pointed at a real off-by-one. For ink ending on row 79 of 100, one function returns 0.81. For ink flush with the bottom it returns 1.01, past the image bottom the documentation calls 1. At the sizes the page draws it, that is a fraction of a pixel.
- Another test exposed a real bug. The code that inserts a generated section into a page collapses
$$to$. The agent had pinned that as expected. Reviewers asked for a test that fails today and turns red when the bug is fixed. - One review model produced nothing usable. It returned about 106 KB of deliberation and ended mid-sentence. A second reviewer that checked its own output on the first try did the job.
The first thing it did was stop itself.
An agent wrote a test file that used https://example.com/logo.png, a fixture the code under test needs. The harness read it as a URL outside its allowlist and tripped the kill switch.
$ run-one issues.json 26 --reply-file reply.txt --tool-uses 0 outcome stopped reason output of #26 references a host outside the allowlist cause https://example.com/logo.png written nothing state STOP is set until a person clears it
The latch was right to hold and the rule was too blunt. The fix taught the output check about reserved documentation hosts. It did not bypass the stop. The agent's own network access stayed strict.
Tuned to one set, 86% became 50% on the next, then 72% on a third.
Each set was written from the source alone by a model that was not allowed to read any test. The share the tests catch is a floor on their quality, not a gate.
logo-baseline
example-chat
Grey bars are scores on sets that had already shaped the tests. Only the dark and orange bars show whether the tests generalise. The five mutations that still get past the example-chat tests on the unseen set all look real. Against the bar we set, at least 80% caught and every escaping mutation caught, logo-baseline clears it and example-chat does not yet.
Their own tests caught 78% of the changes we planted.
We took 9 modules from 7 open-source projects, one of them ours, and ran each project's own tests in the container verifier, with no network and the source mounted read-only. Two different models, neither of which had seen the tests, each wrote 10 small changes per module meant to break it: flip a condition, swap two arguments, drop a guard. A test suite that notices has done its job.
- 136 of 175 changes were caught (78%). Opus caught 69 of 90 and Hermes 67 of 85. Five of the 180 written were discarded because the code they named was not unique.
- The two models agreed where they overlapped. They chose the same spot 27 times and the outcome matched all 27. Their scores per module differed by up to 20 points, and by 10 or less on 7 of the 9.
- Some kinds of change get through more than others. Negations were caught 96% of the time (46 of 48), boundaries 71% (25 of 35) and operators 59% (13 of 22).
- Two full runs gave identical results. Not one of the 180 outcomes changed, and every source file was restored byte for byte afterwards.
- Our own compliance code had gaps. On the module that checks outreach emails, each model got 8 of 10, 16 of 20 together. The four that survived show three real holes in our tests: nothing checks the unsubscribe wording with a capital letter, nothing pins what the opt-out check answers for a blank recipient address, and the order of the sender and postal address in the footer is not asserted. We are writing those tests.
What this does not show. These are existing, human-written tests. None of our contributions was tested, and we proposed nothing to any of these projects and contacted none of them, so they are not named here. Ten changes per model per module is a small sample, and the changes were written by models, not drawn exhaustively. A few survivors are probably harmless changes that behave the same, and we did not remove them, so the true rate is somewhat higher. Line coverage ranged from 79% to 100% on the eight outside modules and did not predict the score.
Your own key, and no rewards.
- API keys only. A person's own key for a provider is the one credential that fits every provider's published terms we read. Consumer subscription logins are out for any tool we ship.
- No rewards or credits. GitHub's rules bar automated activity that is incentivised by tokens or credits, so nothing is paid for contributing.
- Not legal advice. This is our reading of published pages fetched on 2026-10-05. We could not verify whether any provider addresses unattended contributions to third-party projects.