The first student in production

You keep the human work. We build the rest.

Every number below was read on 30 August 2026, Denver time, from files our own systems wrote for themselves. Two of them we produced on purpose, by running work we would otherwise have waited for, and both say so where they appear.

Post two ended with a small model in front of the gate that stands between a candidate and real work, and the gate saying no. Today a small model is through that gate and doing the job.

Between those two sentences we caught our own serving layer lying to us twice, and threw away a win that turned out not to be real.

Figure 1

Five real captures, against the ceiling that was raised

• Time the student took
• The old ceiling

We are training two small models for two different jobs. One reads a finished work session and writes down the facts worth keeping. The other reads a note and builds a map of the things it mentions and how they connect.

The first one is live. On one of our agent seats, on real sessions, through the real code: five captures, 26 memories stored, 36 of the 39 facts it proposed kept. Median 76.4 seconds against the engine it replaced at 118 to 138. That is not a head-to-head, and the head-to-head is in this post too, because it failed.

The second one is still failing, but it fails less each round: 0 notes finished out of 25, then 8, then 18.

A small model like ours is a large open model plus a small trained attachment. The attachment is the training. It is the entire product. You start a server, you tell it which model to load and which attachment to put on top, and it answers questions.

The version of that server we were running looked the attachment up under the wrong name, found nothing, and quietly loaded the plain untrained model instead. No error. No warning. It answered every question, as the model that had never been taught anything.

We proved it three ways: read the lookup code and showed it can never hit; fed the server questions straight out of its own training set and watched it get them wrong; then loaded the attachment by hand and watched the same questions come back right.

Both of our student servers had been doing this for two days.

The exams never caught it because exams load the attachment directly, the way we did by hand. Only the gate talks to a server. So the exam graded the student, the gate graded the plain model, and nobody compared the two.

Figure 2

One setting, 51 identical questions

• The default setting
• Best answer

The second lie took another day to find.

The memory engine's settings object has a sampling temperature, and it defaults to 1. At that setting the model does not pick its best next word. It draws one at random out of its whole vocabulary. Neither test run ever set it.

Take 51 identical questions, one server, one attachment, the same load, and change only that setting:

42 questions went from failing to passing. None went the other way.

And 17.6% is not a coincidence. Run the real pipeline with the setting left alone, as we did the day before, and it comes back at 17.3%. The reproduction lands on the field number.

Nothing in either test harness ever set that value, and it is the library's own default, so it applies anywhere the setting is left alone. Pinning it is one argument to one function. The report that found it called that the single highest-value line of code in the programme, and it was right.

A benchmark tells you what it measured. It does not tell you what it measured it on.

Post two's lesson was that a well-formed answer with the wrong keys silently becomes "found nothing." This is the same lesson one level down: an attachment that fails to load looks exactly like an attachment that loaded and did not help, and a setting nobody pinned looks exactly like a model that cannot write JSON. Both faults are invisible to a benchmark and obvious to a gate, which is the whole argument for gates.

Figure 3

The same 25 notes, three rounds apart

• Measured failure against the bar

The map-building student went round three times. Two of those rounds killed a named failure outright. The third exposed one that is still open. The bar it has to clear is 23 notes of 25.

Here is the honest shape of it.

• v2, the student in post two, re-graded on a serving path we can trust. It finished 0 of 25 and got 0 of 31 calls in the required shape. It never reached the second call of any note. Post two concluded it had been trained on one of the six shapes the pipeline asks for and was perfect on that one, 25 of 25. The honest re-grade is harsher: it had never seen any of the six, and the 25 of 25 was the plain model too. Post two was too kind to it.
• v3 fixed that by training on the real questions, and it fixed the call post two named as the blocker. Post two recorded that call at 0 of 534. Measured properly, v3 answers 100% of 52 of them inside the real pipeline. Five of its six calls came in at 96 to 100%. One did not: 10.3% of 68, and because every note makes that call, that one shape failed 17 notes on its own. 61 of the 62 failed calls in the whole run were that single shape.
• v4 rebuilt the corpus around it. That shape went from 61 of 62 failures to zero failed calls — not fewer, none — and notes finished went from 8 to 18.

One number in that arc needs a correction, because we would have quoted it wrong. The first pipeline run of v3 read 3 of 25, and that is the same student as the 8 of 25 above. Nothing was retrained between them. The only difference is that the first run left the sampling setting alone. A tempting three-step story — 3, then 8, then 18 — would have been two-thirds a measurement artefact.

v4 still cannot be promoted. Twenty-four of its 25 remaining failed calls are one thing: the model starts a list and does not stop, running to the token ceiling. Six of the seven prompts that trigger it are the call every note makes first, so each one costs a whole note and about 24 minutes.

So round four traded a defect that cost 17 notes for one that costs 7. Real progress, not a promotion, and not a third kill.

Figure 4

What actually predicts how long a capture takes

• The assumption we tested
• What the measurement found

The capture student had a problem: fast on small sessions, slow on large ones. The obvious fix was to send it only the small ones and leave the rest with the old engine. That was approved, built, and tested.

Before switching it on, we measured the idea it rests on, across 27 real sessions spread over five size bands. Size does predict how long a capture takes. It just predicts it badly.

The student writes at about 26 tokens a second. A session worth eight memories needs a long answer and takes 45 to 90 seconds however short the input was. A session worth one memory answers in 9 to 13. So the fast captures are the ones with almost nothing worth saving, and every capture that actually produces memories is a slow one. A size gate sorts on the weaker of the two signals.

The individual sessions say the same thing. A 2,693-character session timed out. An 8,759-character one finished in 73 seconds with all six of its facts kept.

Priced out honestly, a tight gate would have admitted 6 of the 27 sessions, served 4 of them inside the time limit, and made the other 2 slower than doing nothing — because a session that blows the limit burns the new engine's clock before the old one even starts.

The measurement that killed the idea pointed at a better one. If output length is what costs time, the fix is not a size gate, it is a longer clock. That is what shipped.

Before anything was switched on, both engines ran the same real sessions side by side, scored by the same screen that admits or rejects every fact in production. The bar was set before the run. The student missed it.

It kept 22 of the 36 facts it proposed, 0.611; the old engine kept 27 of 44, 0.614. It was also far outside the speed bar the run had set. The gate's verdict was FAIL, and the swap it was asked about was not made.

What changed the decision was not a better score. It was the measurement that killed the size gate, which also priced the old engine honestly: over the 27 probe sessions the student kept 0.781 of what it proposed, against that 0.614. Those two numbers are not from the same sessions, and we are not going to pretend otherwise — one is the student over 27 probe sessions, the other is the old engine over the six the gate ran. It is the comparison the decision was taken on. A matched one would be better and does not exist yet.

The reading behind the call is that this was never a safe engine against a risky one. It is two imperfect engines, and the newer one loses less of what it finds.

So what shipped is not what the gate refused. The gate refused a straight swap at the old 60-second limit. What shipped raises that limit to 120 seconds, for the student alone, on one seat, with a rollback threshold that fires without asking.

One seat. Every capture. The old engine still behind it, catching anything that fails.

These five were fired on purpose, so there would be evidence before the seat was left running on its own. The sessions and the code are real; the timing was ours.

26 memories stored. 36 of 39 proposed facts kept, which is 0.923. No capture came back empty. Every one of them recorded which engine answered, so this is read, not inferred from a change in timing.

Time: median 76.4 seconds against the old engine's recent 118 to 138. Read that as a before-and-after on the same seat, not as a race: the two figures come from different sessions, and only the gate run above compared the engines on the same work.

Every single one took longer than 60 seconds, which was the old ceiling. At that ceiling all five would have been cut off and handed back with nothing gained. That is the clearest evidence that raising it was the right call. Two of these five are also among the largest sessions the machine produces and two sit around the middle, so the spread is real work rather than a favourable sample.

We found a limit on the way in. The first attempt fired all five at once, and every one failed. One small server has no queue, so concurrent requests share its throughput instead of waiting their turn, and a burst degrades everything inside it. Traffic has been seen arriving two at a time; five at once was our test, not the workload. Nothing was lost — a capture that fails this way writes no receipt, so the session is retried rather than dropped, and all five succeeded when run one at a time. That is the table above.

Figure 5

Trained small, examined small, served large

• What the student was trained on
• What the exam graded
• What production actually sends

The exam that cleared this student graded it on work the machine does not send.

The cause has a line number. The training data was built by slicing every real session into 4,000-character windows. The live code allows 26,000. The slicing was chosen to get many examples out of each session, and its own comment calls the windows "production-shaped" — which is true of their structure and false of their size. The student has never once seen a whole session.

It survived until now because the exam was graded on the held-out slice of the same sliced data. Training and every check shared the same wrong size, so nothing inside that loop could reveal it. Held out by split is not held out by distribution. Only real traffic could show it, which is what the gate is for, and it showed it on the first row.

The risk of switching on is not bad memories. Every candidate fact still passes the same screen it always did. The risk is a capture that quietly stores nothing where the old engine would have stored several.

So the code now writes a marker on exactly that event, and a separate one when the student proposed facts the screen then rejected, because those are two different faults. A healthy capture carries no marker at all, so counting them is a search rather than a judgement.

Reading now: 5 captures, 0 empty, rate 0.0% against a bar of 10%. Five is not fifty. The verdict is "keep watching."

The window was widened from 4,000 characters to 24,000 and the harvest restarted the same morning. It is running as this publishes.

That middle row was measured on the first 389 examples, part-way through the run.

The old corpus had it backwards, with 76.5% of production above its ceiling. The new one inverts that: 80.2% of it sits above the old ceiling, and its percentiles land on production's. Useful facts per example are up from 4.8 to 6.2. At the reading taken for this post it holds 780 examples at an 89.1% survival rate through the same screen production uses, with no fallback to a second model.

One warning is written into the file itself, so the next person cannot re-make the mistake by reading it: the old training window was 4,096 tokens, and of 60 real requests sampled, two thirds are longer than that on their own. Re-cutting the data without also widening the window would reproduce the defect exactly.

Every exam states its own input distribution beside live traffic's. With the off-distribution percentage printed. No exam may be quoted as evidence about production unless it passed that comparison first. This is the third time we have paid for the same trap: a student trained on the wrong shape, then a check run at a different setting than production, and now a corpus of the right shape and the wrong size.

A serving path has to prove it is serving what you think it is. The capture service does this before it will answer anything: five questions out of its own training set, scored by the same screen production uses. It passed on start at 0.886 against a bar of 0.75.

That bar was measured, not guessed. The obvious test — does it repeat its homework word for word — was tried first and would have wrongly refused the real student, scoring it 0.026, because extraction produces different-but-correct sentences rather than the teacher's exact ones. What separates them is how much of their output the production screen accepts: on twelve questions, the student averaged 0.902 and the plain model 0.573.

And the check is proven to refuse, not only to pass. Run against the plain model with the training switched off, which is what a silently broken load actually serves, it failed at 0.650 with the reason written out. A check only ever tried on the case that passes has not been checked. The map-building student's path does not have this yet; the gate there is still a step a person runs.

A switch ships with its own undo and a threshold that fires without us. A number, a window, a delegated authority to roll back, and a one-file change to do it, all written down before the switch rather than after.

The map-building student's fifth round is training as this publishes, and its verdict lands after you read this. We are publishing without it on purpose, because waiting for a result before describing the method is how the method stops being honest. Whichever way it goes, it is post four, with the same numbers attached.

Also in post four: whether the capture student's rate holds over 50 captures rather than 5, what the rebuilt corpus does to a speed comparison that is currently not a race, and the matched head-to-head this post could not give you.

What this post adds to it is smaller than it sounds, and it is the part we would want from someone else: check the size of what you trained on against the size of what you serve, and print both. Prove the server loaded the thing you trained before you believe any number it gives you. Then measure the idea your shortcut rests on before you ship the shortcut.

The pairs stay private, as posts one and two said, and that still costs you nothing. Your teacher writes your pairs, from your loops, on your machine.