Research note · 22 Jul 2026 · 3 min

Reading the adhd evals honestly

The harness won five of six problems. The sixth is the one worth writing about, and the ratios are the number most likely to mislead you.

Udit Akhouri · evals · methodology

We judged adhd head-to-head against a single-shot baseline across six open-ended engineering problems, scored 0–10 on five dimensions. It won five. The headline everyone quotes is trap detection at 5.2×. That is the number I trust least, and this note explains why.

What the ratio actually measures

A ratio between two scores on a bounded scale is not a speedup. When the baseline sits near the floor, the ratio inflates without the numerator moving at all. Trap detection scored 1.83 for the baseline and 9.50 for the harness:

r = \frac{s _{adhd}}{s _{base}} = \frac{9.50}{1.83} \approx 5.2

Nudge the baseline down by half a point and the ratio jumps past 7 while the harness has not improved by a single token. The delta — $+ 7.67$ — is the honest figure, because it moves one-for-one with the thing we actually changed.

The problem we lost

The sixth problem was one the baseline already knew cold: a well-trodden caching question with a canonical answer sitting squarely in the training distribution. Divergence bought us nothing there, and cost us real tokens finding that out.

That is not an embarrassing result, it is the shape of the method. Divergence pays when the answer is not already known. On a problem with a known answer, running fifteen frames in parallel is an expensive way to arrive where a single pass arrives immediately.

Condition	Baseline	Harness	Worth running?
Answer in-distribution	strong	equal, slower	no
Answer contested	weak	strong	yes
No known answer	anchors early	strong	yes

The useful question is not "does the harness win" but "does this problem have a known answer" — and that is usually apparent before you spend the tokens.

Where the scores came from

Scoring is a critic pass over every candidate, not a human rubric applied after the fact:

const scored = await Promise.all(
  candidates.map(async (candidate) => ({
    candidate,
    // The critic never sees sibling candidates — otherwise the highest-scoring
    // branch quietly becomes the frame for judging the rest.
    verdict: await critic.score(candidate, { siblings: null }),
  })),
);
 
const survivors = scored
  .filter((s) => s.verdict.score >= threshold && !s.verdict.trap)
  .map((s) => s.candidate);

The siblings: null is the part that matters. An earlier build let the critic see all candidates at once, and it reliably converged on whichever candidate it read first — reproducing, inside the critic, exactly the anchoring the harness exists to avoid.

Anchoring is not a failure of effort. It is what happens when every step is conditioned on the step before it.

Fig. A — a candidate graph before the critic pass. The filled node is the first plausible direction; in a single-shot run everything downstream inherits it.

What I would not claim

Six problems is a small sample and the scorer is a model. This is a proof-of-concept preprint, not a benchmark. The result I would defend is narrow and specific: on open-ended problems, isolated parallel candidates surface traps that a single forward pass does not. Everything past that is a hypothesis with a number attached to it.

[ ← All writing ][ Shared context is shared bias → ]