The exam it passed, the gate it failed

You keep the human work. We build the rest.

Every number below was read on 28 August 2026, Denver time, from files our own systems wrote for themselves. Nothing was started, stopped, or changed to produce it.

In post one we promised that if the student missed, we would publish the miss with the same numbers attached. It missed, and the miss is why we now have a fix that builds itself.

We trained two small models to take over the reading job from the 27-billion-parameter model doing it now. Both sat the 400-question exam we held back. The 4-billion-parameter student won, with a perfect record on answer format.

Then we put the winner inside the real pipeline, on real work, beside the model it would replace. It finished 1 of 25 notes. The incumbent finished 25 of 25.

Two students, one exam, 400 questions neither had seen.

The bar to beat was 0.53, which is from post one: how much two competent large models agreed with each other on the same paragraphs. Both students cleared it. The 4B cleared it by more and never wrote broken JSON, so the 4B was crowned.

Post one also promised a serving speed on a free graphics card, because the savings case rests on it. This is the best measurement we have, and it is still not the clean one.

The winning 4B served the same exam at two speeds on the same morning: 18 to 26 tokens per second across the first 135 items, and 63.5 tokens per second, 12.44 seconds an item, across the other 265. Same model, same adapter, same questions, an hour apart. The only thing that changed is how busy the machine was. A separate 100-item run of the same model landed at 65.0, so the faster figure is not a fluke. The 1.7B measured 86.1 tokens per second and 8.86 seconds an item across its own 400 while the machine was loaded, and an earlier checkpoint of it reached 99.9.

We do not promote on an exam score. We promote on a canary: the candidate runs the whole pipeline on real work, and the incumbent runs the identical work beside it. Same 25 notes from our own backlog, same order, same graph store.

The student was faster per note, 93.9 seconds against 139.5. It was fast because it gave up.

The exam could not see the reason. The memory engine does not make one kind of request. It makes six, each asking for a different answer shape: pull out the entities, pull out the relationships, is this entity one we already have, is this fact a duplicate or a contradiction, and two more. Our student was trained on exactly one of them, because we harvested its training pairs from one hand-written prompt.

On the shape it was trained for, it was perfect inside the real pipeline: 25 of 25 clean.

On the shape it had never seen, it scored 0 out of 534. That shape is the duplicate-and-contradiction check, the call that keeps the graph from becoming a pile of repeated facts. The student failed it the same way every time. Handed the answer shape it was supposed to fill in, it printed the shape back:

{"properties": {"duplicate_facts": {"description": "List of idx values of
duplicate facts (only from EXISTING FACTS range). Empty list if none.", ...

{"properties": {"duplicate_facts": {"description": "List of idx values of
duplicate facts (only from EXISTING FACTS range). Empty list if none.", ...

The first 500 characters are identical across 394 of those attempts. The other 140 are the same thing, pretty-printed.

The pipeline never checks that an answer carries the keys of the question it asked. It parses the JSON, reads the key it wanted, finds nothing there, and moves on. A well-formed JSON object with the wrong keys is not an error anywhere in the stack. It silently becomes "found nothing."

So "100% valid JSON" was never the real bar. It was a number that felt like proof and measured almost nothing. The exam graded the student on its own trained prompt, in a format check the pipeline does not perform, and returned a clean pass on a model that could not run the job. We now grade the shape of every call instead, which is what the canary did.

We tested the obvious excuse. The student's server cannot enforce an answer format, so we had put the required shape in the prompt. Maybe it copied the shape only because we showed it the shape.

So we ran the same real question three ways, five times each. Shape in the prompt: 0 of 5, copied. Shape removed: 0 of 5. Through the incumbent: 5 of 5 correct. The test rig did not cause this.

But look at what the student said with the shape removed:

duplicate_facts=[]
contradicted_facts=[1]

duplicate_facts=[]
contradicted_facts=[1]

That is the right answer: it spotted that the new fact contradicted the stored one. It knew the answer and could not write it down. A formatting gap, not a thinking gap, and that is the kind we can train out.

The training data for the missing five shapes already exists. The incumbent answers those questions correctly all day, and nobody was writing them down.

So we put a recorder on the production request path. Every question the engine asks and every answer it gets back is now copied to disk, tagged with which of the six shapes it belongs to.

• All six shapes were flowing within twelve minutes of install.
• 56 pairs in the first thirty minutes, about 112 an hour at that rate, zero dropped and zero write errors. 25 of the 56 were the exact shape the student failed 534 times.
• The install cost 0.44 seconds of downtime, fired at a gap between notes. The note in flight across the restart still finished, with 5 entities and 2 links.

The corpus builds itself while the work runs. The next student trains on the mixed set and faces the same canary. The builder drops any pair whose answer is missing the required keys, which is the gate the canary proved matters.

The canary was pointed at the student. It also found things about the system we are replacing, which nobody had measured before.

• It goes blank. 5 of 25 notes with real content in them came back with zero entities. The fastest refusal took 1.93 seconds. The student extracted from all 25.
• It is not repeatable. Set to be deterministic, the same note from the same starting state gave 0 entities on one run and 9 on the next.

None of that changes the verdict. A slow engine that finishes 25 of 25 beats a fast one that finishes 1. But we have been filing part of our own backlog as empty, and we would not know it if we had not built the gate to judge the replacement.

A second small model is in training for a different job, capturing facts from finished work sessions. Its 7,026 training pairs were written by a large cloud model that won a five-way audition, scored on how much of its output survived our live validator. Details next post.

It is the machinery, not our data: the harvester that turns a running loop's logs into training pairs, the redaction that runs before anything leaves the machine, the dataset builder that de-duplicates and splits them, the LoRA recipe, and the exam harness — whose scorer now takes a list of keys an answer must carry, which is the check this whole post is about. The canary methodology ships with it. Shipping the exam alone would hand you the tool that told us yes when the answer was no.

The pairs stay private, as post one said, and that costs you nothing. Your teacher writes your pairs, from your loops, on your machine.

The re-canary verdict for the next student, against the same 25 notes and the same six shapes. And the measured model time per day after deployment, set beside the 4.06 and 8.11 hours projected in post one.