§01
The question
If nobody reads every change, something else has to judge it, and that judge needs judging too. I am looking at what makes a verdict trustworthy: whether reviewers are actually independent of each other and of the author, whether a passing check means what we think it means, and how you notice behaviour drifting while every gate stays green.
§02
Why it matters now
Fluency is not correctness. A grader that shares the author’s blind spot approves a plausible wrong answer with confidence, and a reviewer built from the same material as the author is not an independent opinion. As more review moves to models, the failure to worry about shifts from the missed bug to confident agreement about the wrong thing.
§03
The landscape
Using models as judges is now ordinary practice, and its biases are on the record. Zheng et al. (2023), introducing MT-Bench, found LLM judges prone to position, verbosity and self-enhancement bias. Panickssery et al. (2024) showed LLM evaluators can recognise their own outputs and tend to favour them. Kim et al. (2025) found that models’ errors are substantially correlated, including across providers.
Software reliability ran this experiment long before models. Knight and Leveson (1986) had independently written versions of one program and found they failed together far more often than independence predicts, because some parts of a problem are simply harder than others. Redundancy still helped; it just helped less than the arithmetic promised.
§04
What I’m testing
My working position: agreement between judges is not evidence until you can tell it apart from correlated error, and that takes ground truth. So the questions I am working are measurable ones. How many effective votes does a panel of model reviewers really give you? Does varying the question (adversarial, checklist, conformance to the spec) buy more independence than varying the vendor? Should a check that is not a model, a type checker or a property test, outrank one more model reviewer because it fails for different reasons? And when reviewers agree and are wrong, is that telling me about the spec rather than the change?
Related