In part 3, every failure came from something CI couldn’t see. “No visual change” is just as hard to check.

Adopting the shared quality gate on this site turned on a duplication check. It found 22 duplicated blocks across the components (1.19 %). An agent refactored them into shared components and shared CSS classes, down to 5 blocks (0.23 %). The diff touched components used on most pages.

The pull request could have said “no visual change, I checked a few pages.” Agents write that kind of sentence often, with nothing behind it. I asked for proof.

What “nothing changed” means, measured

A screenshot comparison is the obvious tool. It tells you that something moved, and it can flag antialiasing noise along the way. The agent compared the elements themselves, which tells you what changed.

A Playwright script loaded each page from the build before the refactor and from the build after it. For every element on the page, it recorded:

  • 46 computed style properties (getComputedStyle): colors, fonts, spacing, borders, display, position and so on.
  • The box geometry (getBoundingClientRect): position and size.

Then it diffed the two lists, element by element.

The matrix:

  • 16 pages: the home page, resume, open source and projects pages, the blog index and two posts, three course pages, two module outlines, and in draft mode the drafts listing and three lessons, so every refactored component was rendered at least once.
  • 2 widths: 1280 px and 390 px.
  • 2 themes: light and dark.

Result: 0 differences.

Test the measurement first

A comparison that reports 0 differences proves nothing if it can’t report anything at all. So before comparing before and after, the script ran on the same build twice.

That run had two purposes:

  • It had to report 0 differences. Otherwise the measurement itself is noisy, and a real change would hide in the noise.
  • It showed what’s not deterministic. The home page has a rotating line driven by a timer, so two loads of the same page differ. The home page is compared with JavaScript turned off.

Only after that run came back clean did the before/after comparison mean anything.

A test that tested nothing

I learned the same lesson the same evening, from a mistake of mine.

To prove that the pre-commit hook blocks secrets, I committed a fake token and expected the commit to fail. It went through. The hook looked broken.

My fake token had a 38-character body, and the real format has 36. secretlint ignored a string that isn’t a token, as it should. My test was wrong. With a realistic fake, the hook blocked the commit.

A test you’ve never seen fail doesn’t tell you anything. That holds for unit tests (it’s the whole idea behind mutation testing, covered in Verify), and it holds for gates and for measurement scripts.

The same habit, everywhere

Once you start asking for proof, it changes how every pull request is written:

  • The bun → pnpm migration was diffed, not assumed equivalent: 682 resolved packages compared, 3 transitive patch differences, each one listed in the pull request.
  • README reorganizations were checked line by line: nothing lost, and every deliberate rewrite listed.
  • Every pull request carries its proof: the commands that ran and their real output, plus what wasn’t tested.

With agents, that last item counts most. “Tests pass” in a pull request body is a claim. The output of vp run verify pasted below it (332 tests, 0 type errors, 1180 internal links checked) is evidence. And “not tested: the first npm release through the new workflow” is honest information I can act on.

Agents get this wrong in a predictable way, as part 5 shows. The rule I give them now: only report a check you saw pass.

Takeaways

  • “No change” is a measurable claim. Compare computed styles and geometry for every element. Pseudo-elements, text and image content need their own check, or a screenshot pass on top.
  • Cover the matrix: every page that renders a changed component, at each breakpoint and theme.
  • Run the measurement against itself first. It has to report 0 on identical input before its 0 means anything.
  • Find what isn’t deterministic (timers, animations, dates) and neutralize it.
  • Make tests fail once. A fake secret has to be realistic enough to be caught.
  • Ask agents for evidence, not claims: commands, real output, and what wasn’t tested.

In the last part: 40 pull requests in one evening, and the decisions I kept for myself.

Share this post

← All posts