The agent proudly announces: “47 tests added, all passing, 94% coverage.” You open a random file. expect(result).toBeDefined(). Another test mocks the function under test itself. Then you ask the same agent, in the same conversation, to review its code. It finds two cosmetic tweaks. Coverage is real. Protection is almost zero.

Verification has two failure modes with agents: tests that verify nothing, and reviews that confirm instead of challenge. Fix both.

Test first, then test the tests

Extreme Programming, popularized by Kent Beck in the late 1990s, and lean practices installed a triad: write the test before the code, move in small batches, integrate continuously. It maps cleanly onto agents: short task, a test that defines success, a CI run that decides.

The trap: agents optimize for “tests pass,” and the shortest path is tests that are easy to pass. Symptoms:

  • Weak assertions: toBeDefined, toBeTruthy, type instead of value.
  • Tests that reimplement the logic they check.
  • Excessive mocks, up to mocking what you claim to test.
  • Tests rewritten to match the code.

Coverage catches none of it: it measures what ran, not what was verified. The real question is “would the tests fail if the code were wrong?” That’s mutation testing: flip a condition, change a constant, delete a line, and check a test fails. Surviving mutants are unverified behavior — and empty assertions can’t fool it.

  • TypeScript: vitest for runs, fast-check for property-based tests on invariants, Stryker for mutation testing (incremental on the diff).
  • Rust: cargo nextest, proptest, cargo-mutants --in-diff.

Make the mutation score on the diff a CI gate, not an optional report.

The author doesn’t judge (Fagan)

In 1976, Michael Fagan published “Design and Code Inspections to Reduce Errors in Program Development” (IBM Systems Journal): distinct roles, individual preparation, checklists of typical defects, follow-up. Two ideas survive fifty years: the reviewer is not the author, and the review follows a checklist.

An agent reviewing in the same context has its generation reasoning right in front of it; the wrong assumptions are already “validated.” So make separation real:

  • A separate critic agent, in a fresh context.
  • It receives only the diff, the spec or ticket, and the checklist — never the generation reasoning.
  • It reports, it doesn’t fix. An implementation agent, or you, decides the fixes.

A checklist turns vague comments into defects found:

  • Are external inputs validated at the boundary?
  • Is every error branch tested?
  • Did any observable behavior of a public API change?
  • Was a new dependency added?
  • Do the tests verify behavior, or just that the code runs?
  • Does any function exceed the complexity thresholds?

Every bug that escapes review adds a line. The critic is a filter, not the final judge: your human review focuses on design and direction.

Takeaways

  • Green tests are not proof: weak assertions are a known agent failure mode.
  • Mutation testing is the gate that catches trivial tests — Stryker, cargo-mutants --in-diff; property tests with fast-check / proptest.
  • Fagan (1976): reviewer distinct from the author, driven by a checklist.
  • Critic agent in a fresh context: diff, spec, checklist only. It reports; it doesn’t fix.
  • Every escaped bug adds a line to the checklist.

Share this post

← All posts