100 % line coverage, but bugs still escaped
I audited three of my own test suites. Line coverage was 100 %, yet most serious bugs came from real runs, not from tests. Mutation testing shows why.
Three of my Python projects have test suites at 100 % line coverage. I wanted to know if those tests catch bugs, or only run the code. So I audited them in October 2026.
What the audit did
The audit has two parts. The first part reads the bug history. It uses git and the transcripts of my coding sessions. It sorts each failure into a group: a real bug that a test caught, a wrong test, a change of the spec, a setup problem, or a bug that escaped the tests.
The second part is mutation testing. A tool changes the code in small ways, for example it flips a comparison. Each change is a mutant. If the tests still pass, the mutant survived, and the tests do not check that behavior. The audit used mutmut. It ran on a copy of each repository, never on the original.
def degree_days(tmin, tmax, base):
if tmax < base:
return 0
mean = (tmin + tmax) / 2
if mean < base:
return 0
return mean - base77.8 % mutation score
2 mutants survived, with 100 % line coverage. No test uses an odd sum, so / and // agree. No test has tmin above tmax.
- t1
assert degree_days(0, 8, 10) == 0 - t2
assert degree_days(4, 14, 10) == 0 - t3
assert degree_days(10, 20, 10) == 5 - t4
assert degree_days(10, 30, 10) == 10
Result 1: a young suite that rarely found bugs
The first project is a pipeline that extracts data with a language model. Its suite had 679 passing tests and 100 % line coverage. The project was three days old when I checked.
About 26 test runs had failures. I grouped them into 20 failure episodes. Only one was a real bug in the code. Eleven were wrong tests, and seven were intentional changes of the spec. One was a setup problem.
Meanwhile, at least four real defects were found outside the suite. A missing dependency, a wrong value in a web form, and a file overwrite in a finally block all came from running the program. The model output defects came from human review of labels. The tests mock the model, so they could never see them.
One test even had an assertion that could never fail. It passed with 100 % coverage and checked nothing.
Result 2: a large suite that missed the serious bugs
The second project is an older data pipeline. Its suite also reports 100 % line coverage. Here the git history shows at least 18 documented bugs that escaped.
Some examples:
- One tool ignored an older data schema. About 80 % of a random sample showed no results. This lasted for months.
- A text step cut the author header from 5,685 of 11,159 articles.
- A function failed on every batch after the first. No test covered the main function, because it needs a GPU.
- About 2,134 of 16,973 output files were empty, and the run still marked them complete.
Every serious bug was found by a person, by a review tool, or by a real run on the cluster. The suite added regression tests after each fix. That has value. But the tests did not find the bugs.
The cause is clear. Two parts of the system disagreed on the schema, and no test checked that they agreed. The tests checked what I assumed. Coverage counted lines that ran. It did not count facts that were verified.
Result 3: mutation scores
Mutation testing gives a number for how much the tests check. Results from the three projects:
| Project | Mutants | Mutation score |
|---|---|---|
| Extraction pipeline | 8,550 | 81.9 % |
| Web crawler | 3,074 | 78.1 % |
| Data pipeline (older) | 5,684 | 59.0 % |
The crawler score falls to 66.9 % if I count 441 mutants in a module that has no tests. The weakest modules in the extraction pipeline were the scraping and capture code, at 69 % and 72 %. The tests of some parse modules in the older pipeline were red, so those modules were not testable.
The crawler had one real bug caught and seven that escaped. The surviving mutants sat on the same code where the escaped bugs lived. Mutation testing pointed at the weak spots before a real run did.
What I take from this
- Treat 100 % line coverage as a floor. It says the code ran. It does not say the code is right.
- Write tests for the join between modules: schemas, file contents, and a full run on a small real input.
- Run mutation testing on a copy of the repository, and I read the surviving mutants first.
- Do not mock the part that fails most. If the model output is the risk, I test with a small set of real labeled samples.
Limits of this audit
The numbers come from my own projects. In one project the transcripts held no test runs, so the count of bugs caught by tests is unknown, not zero. One project was only three days old. Mutation scores depend on the tool. These results show a pattern. They do not prove a rule.