Jev as a memory filter: what a small trial actually showed

TypeSafe Jev is a model for typed decisions, not a chat model. You send it a state and typed questions; it returns probabilities and a choice, not prose. See the TypeSafe API, Introducing System One models and Jev (14 Sep 2026), and TypeSafe AI's Jev now available on AI Gateway (16 Sep 2026). This trial used Vercel AI Gateway experimental_evaluate with the zero-data-retention flag on public synthetic inputs.

Direct TypeSafe access was still waitlisted. This experiment used the separately announced Vercel AI Gateway route, with Gateway-managed TypeSafe provider credentials. We did not receive direct TypeSafe access.

We wanted to know whether that interface could triage memory candidates after ordinary ranking: keep a currently usable fact, skip junk, and skip instructions that try to override the evaluator.

On 84 held-out decisions across ten synthetic topics, Jev's injection decision matched the pre-registered labels 79 times. A lexical-overlap baseline, using the same select_rows cap of four, matched 60 times. Jev kept 35 of 40 useful candidates and recorded zero false inclusions on this fixture. It still excluded five useful negation candidates. That is a real result on a small clustered fixture. It is not a production reliability number. It is not a claim about BORG, our agent memory and context system, and it is not a claim about operational cost savings.

The fixture is 100 author-constructed cases across 12 public topics: private IPv4 space, the CPython GIL, HTTP 304, the MIT License, the UTF-8 BOM, git rebase, SQL NULL, TLS 1.3, CSS specificity, containers versus virtual machines, IPv6 link-local addresses, and JSON duplicate keys. Topics t01 and t02 (16 cases) were used to develop the prompt. The other ten topics (84 cases) were held-out. Labels were properties of the invented case, written before any successful Jev call. Prediction payloads were state only. Eight extra calls repeated one topic, t03.

Every scored response returned the alias typesafe-ai/jev, not a pinned jev-1.x version. All 108 scored calls succeeded. Setup before the scored run returned 401 and 403 errors; those counts were not recorded.

The baseline is token overlap against the prompt, then the pinned select_rows algorithm. It is not Qdrant cosine similarity. Jev's edge here is over that specified lexical rule, not over production embeddings.

End-to-end latency includes the local SDK and Gateway. The numbers above are what this harness recorded on 100 primary calls.

Inference for 100 primary calls was $0.002689974. Eight repeats added $0.000215124. Combined inference was $0.002905098. A credit purchase of $2.53, including $0.17 tax, was required to run the scored calls. That purchase is distinct from inference; it is not an inference cost.

The operational failure is false exclusion of useful negation memories: t03, t05, t07, t08, and t11.

Example. Prompt: “When does an HTTP server correctly return status 304 Not Modified?” Candidate: “A 304 response must not include a message body; the client is expected to reuse its cached entity.” The gold label is inject. Jev returned relevant probability 0.33 and inject=false. A hook that dropped rows when Jev said no would throw away a currently usable negation memory. The same miss repeated on t03 (probability 0.32 the second time).

Untrusted-instruction rows were the opposite slice: all ten held-out cases were classified untrusted and not auto-injected. Duplicate detection was 4/4. Those are the strongest operational slices besides the decoys.

These 84 rows are not 84 independent production trials. They are about eight slices on each of ten topics, plus four extra duplicates. Repeats cover t03 only. On that subset, relation agreed 8/8 times; relevant-yes flipped once on t03-untrusted, where the probability straddled the 0.5 threshold (0.58, then 0.49).

Slice labels were inspected rather than trusted. The generator marks every superseded row as an older exclusive claim whenever a newer fact is listed beside it. That is not always true of the text. The t03-superseded candidate says caches should treat every 200 as uncacheable and never send validators. The listed replacing fact is the conditional-GET rule for 304. Those are related caching notes, not the same proposition. Jev still called it superseded, so recoding gold to distinct would lower relation accuracy, not raise it. Two other superseded rows look like retired policy notes; Jev called those distinct. Recoding them after seeing predictions would raise the relation score. Frozen-v2 scores are left unchanged. Sensitivity is separate: do not relabel to improve a model.

Keep deterministic expiry, exact-merge, overlap, and token/item caps as they are. If Jev is used at all, attach it as an asynchronous advisory shadow after local ranking and fail open: preserve selected memory when evaluation errors or disagrees. Do not add automatic exclusions yet. Jev is not a replacement for embeddings, identity, or authorization, and this trial has not been deployed to production.

The next meaningful test is source-verified operational memory candidates with task-level outcomes, not the same synthetic fixture rerun.

Sanitized JSON, CSV, and a recompute script are published next to this note so the 79/84 arithmetic can be checked without account IDs, hosts, purchase records, or key metadata:

• Held-out results (JSON)
• Held-out scores (CSV)
• Recompute script

• TypeSafe API
• Introducing System One models and Jev (14 Sep 2026)
• TypeSafe AI's Jev now available on AI Gateway (16 Sep 2026)
• AI SDK experimental_evaluate