Rincón, D., with Claude · phronesis · 2026 · checking again is not checking
The examples here are not hypothetical and not curated from a career — they are one audit day on this site, 18–19 July 2026. The checker whose failures are catalogued was Claude, working as the site’s engineer; the checks that failed were its own. What made the day unusual was not the number of defects but their camouflage: every serious one sat behind a check that had already passed. The passing check was the disguise.
A verification script printed twelve lines reading BAD and then concluded PASS. The failure flag was being set inside a subshell and died with it; the summary read a variable nothing could ever have written. The check was not lying — it was incapable of telling the truth. Its cousin is the test suite recorded from a system’s current behaviour: it passes on the day it is written and forever after, because it asserts that the system does whatever it does.
The tell: a check that has never failed is unproven. Break the thing on purpose once and watch the check go red. If it cannot be made to fail, it is not an instrument; it is a decoration.
A search for href="/courses" found nothing, because the framework emits href="/courses/". A dollar sign meant literally was read as a regex anchor and matched nothing. A price never appeared as a contiguous string because the renderer inserts comment nodes between text runs. An audit expecting min="0" met JSX writing min={0}. In each case the check was well-formed — for a substrate that was not the one deployed. Checker and checked spoke two dialects, disagreement came out as silence, and silence read as health.
The tell: before trusting a checker’s negatives, hand it one known positive. If it cannot find the thing you know is there, its “nothing found” means nothing.
The site map rendered blank for most of a day, and every audit passed while it did. The links were audited in source, where all of them were present and correct. The console was checked for errors and had none, because a parse error kills a script before any handler exists to report it. The HTML fallback held every link, hidden by CSS on desktop. Each check looked where checking was easy. The failure lived where the reader lives — the rendered page — and the only instrument pointed there was a person, who said: nothing really visible.
The tell: check at the layer where the consequence happens. A source audit is a fact about source. Only a rendered check is a fact about the page.
The writing instrument on this site reported a “coherence” score — the algebraic connectivity of a sentence graph. It was stable, reproducible, mathematically well-defined, and mostly measuring length: the same document truncated to eight sentences scored 0.39; whole, 0.11. Identical prose, a fourfold swing. A measure can be perfectly reliable about the wrong thing. What exposed it was an invariance test: apply a manipulation the measure should ignore — truncating the same prose — and watch whether it matters. The repair was not a cleverer denominator (four were tried; all failed) but a fixed scale of measurement.
The tell: reliability is not validity. Find a transformation the measure should be indifferent to, apply it, and see.
To test that metric, four sample texts were first written by hand — a tight argument, a rumination, a disconnected list, a technical passage. They confirmed the confound the author expected and concealed the one he hadn’t imagined, because they were written varying the suspected thing and holding steady the unsuspected one. Seventy-four documents written for other purposes — the site’s own papers and course modules — reversed the finding in a single run.
The tell: a test set authored by someone who knows the hypothesis is the weakest evidence there is. Use material that existed before the question did.
Three experimental designs in a row failed to find a hysteresis loop that was really there. The first held each measurement so long that the memory it hunted had already relaxed — quasi-static by construction, and hysteresis vanishes in the quasi-static limit by definition. The second compared the effect at its best against the control at its worst. The third took the ratio of two numbers that were both noise. The loop appeared on the fourth design — paired, rate-matched, absolute — and had been there all along.
The tell: ask what would distinguish the effect from the artefact that mimics it, and design for the distinction. A design that only asks is there an effect? will find one, or erase one, by construction.
The contact form posted to a worker that had not existed for weeks. Every submission failed; every failure was swallowed by a catch block; the form looked fine and told no one. The suggestion box had the identical disease. A game’s scores went to endpoints never deployed, each request ending in catch(()=>{}). Nothing looked broken, because looking broken had been explicitly handled away. Silence is ambiguous between no failure and no signal, and each of these systems had been built to prefer the flattering reading.
The tell: absence of an error is not presence of a success. Probe the path end to end and require the positive signal — the 201, the stored record, the reply — before believing it.
A page argued that a steel bench and a pine bench sit about 9 °C apart at the skin. The figure was real, computed from a table of conductivities and densities that lived in the same file. Then the table was edited — better numbers, sound reasons, and every test still green, because the code was never wrong. Only the sentence was. A document quoting a figure it did not measure is an untested assertion, and it fails silently: nothing throws, nothing 404s, the page simply says a thing that is no longer true and keeps saying it.
The fix was to write the sentences down as tests. Not one assertion per function — one per claim, named after the sentence it defends: steel and pine sit ~9 °C apart, a steel bench at 65 °C burns, cork moves 6 °C across a 75 °C swing while steel moves 69. Within seconds of being written the suite failed, and the failing assertion was the one that had been overclaimed: it said every plant material beat every petroleum one, and polyurethane foam had the best number on the board. The page had to be weakened from “biomaterials feel warmer” to “one of them matches the best synthetic there is.” That is a truer sentence, and it exists because an assertion refused the flattering one.
Two more fell out of the same suite. A rule change made minerals admissible, and an assertion reading clean surfaces span less than concrete began comparing concrete against itself — the words had held still while the category beneath them changed meaning. And a superlative flipped: foam has the lowest effusivity was true, was asserted, and stopped being true the moment a new material was added. It carried a note saying what to rewrite when it flipped, so the flip arrived with its own instructions.
The sharpest version of the move is to assert what the writing must not say. When two figures were close enough to call level rather than a win, the suite gained a check that fails if anyone upgrades the prose to a victory while the gap stays small — and would announce it if the evidence ever justified the stronger claim. A document that cannot drift toward exaggeration is a rarer thing than a document that is currently accurate.
The tell: find every number in the prose that came from somewhere else, and ask what would happen if the somewhere else changed. If the answer is “nothing would happen”, the sentence is unguarded. Assertions written from the claim catch this; assertions written from the behaviour never do, because the behaviour is fine.
A script replaced a string in a source file, printed updated, and had changed nothing. The pattern assumed six spaces of indentation where the file used four, so it matched zero times and reported success anyway. This happened four times in one day, and one of those silent misses left a build gate in place that passed because it was checking nothing — a rule matching no input is indistinguishable from a rule finding no fault. Another survived a full deploy.
This one sits upstream of the others. They are ways a check goes green while the thing is broken; this is a way the thing was never touched while the tooling said it was. Everything downstream then behaves correctly and misleads: the test passes because it is still testing the old code, the page looks unchanged because it is unchanged, and the conclusion drawn is about the wrong artifact entirely.
The tell is that the edit and the report of the edit come from the same process, which has no way to tell a successful replacement from a vacuous one. A tool that errors on a missed pattern removes the category. Failing that: assert the match count, and read the file back rather than the log.
A mobile fix, a leaderboard, a rewritten index page and a corrected count each appeared not to have deployed. Each had deployed. The apex domain was serving edge-cached HTML with a week-long lifetime, so the verification kept measuring an old copy of the right page — and query-string cache-busting was unreliable enough that it sometimes returned the new copy and sometimes the old, in the same minute.
The reverse is worse and quieter. A change can look live because a cached copy happens to match what you expected, and the same session found two versions of that: a Worker whose deployed code no longer matched the source in the repository, where only one of three routes it defines actually answers, and a directory of built output standing in for source that had moved on. In each case the artifact under inspection was real, current-looking, and not the one that had been edited.
§3 is about looking where the light is. This is about looking in exactly the right place, at yesterday’s version of it. The move that works is to verify against the immutable thing the deploy just returned — the specific deployment URL, the commit, the build output — and treat the friendly alias as a question about propagation rather than about correctness.
Against the ten, the moves that worked were few:
A check is an instrument, and a passing check is a claim of coherence. The claim is worth exactly the displacement the check could have registered — a needle that cannot move measures nothing, whatever it points at. Checking again interrogates the same sector of the plane. Checking differently rotates the sector, and the failures, this day, were always in the sector nobody had rotated to.
One day, one site, one checker — and the checker writing the catalogue is the one that made the errors, which biases the catalogue toward errors that were eventually caught. The ones that were not caught are, by construction, absent. That is not a flourish; it is the strongest reason to expect this list to be incomplete.
These get worked out in the open, at whatever length the problem takes. I do the same thing on a problem of yours — one thing diagnosed and written up plainly, no build. what that costs
Source: Hacker News — This article was automatically imported from the source. Read full article at original source →
