Where an Eval can say “I don't know”

Abstract

Most of what an agent does can be checked exactly: whether the tool exists, whether its arguments are legal, whether the trajectory is right, whether the goal state was reached.

In the atomic agent we run in production, each of those checks is plain code and a mutation test in CI proves it is wired in. Judges earn their place in prose claims and tone and a person in intent. One more outcome has to be designed in: no-signal, for when nothing was measured. Our most misleading result was pass@k = 0 for a run where no episode completed.

A typed classifier like Jev fits closed questions, but it cannot supply the claim it checks, the reason, or this question is not mine.

All our atomic agents run on one runtime, Blueprint. An agent is a configuration of it (its tools, phases and roles), never a copy, so the checks it runs on every request come with every agent built on it.

Read this as:

a map of where each kind of check sits in the systems we run, and of the one output every eval needs.

Our last post was a bar joke. Someone asks where AGI is. An LLM judge replies that the question is absolutely right. A classifier answers YES, with p = 0.998. (The numbers belong to the joke.)

Neither judge has a way to say I don’t know. The first one agrees: assistants trained on human preferences measurably lean toward answers that match the asker’s view (Sharma et al., 2023). The second answers YES because a yes-or-no question is the only kind it has.

Evals have the same gap. An eval that can only pass or fail has to report I could not measure this as one of the two, usually as a number. Classification has had the fix since 1970: a reject option (Chow, 1970). That answer has to come from the instrument, not from the judge’s goodwill. This note maps where that option belongs in the agent workflows we build and run, and which of the checks around it are exact, which need a judge, and which need a person.

The evals map

Most of what our agents do can be checked exactly. The tool an agent called exists or it doesn’t; its arguments validate against the schema or they don’t. Whether the segment it created holds the right users — one database query. None of that needs a model to judge it.

A judge earns its place in two rows of the map: claims made in prose and tone. A person earns one: intent.

The rule behind Table 1 is written into the tool contract of the cohort agent, which turns one sentence into a customer segment: “Tools, related actions (policy), defined space are required upfront. We don’t rely on a prompt.” A sentence in a prompt that exists to compensate for the schema (“month is 1..12”) is a defect in the schema [S1]. The strongest form of the rule is for irreversible actions. They get a gate in code, never a prompt rule [S2].

What is checked	How	When it runs	Determinism	Where it runs for us
The action space did not drift	Tool schema exported from the domain’s own registry, pinned by hash; the agent refuses to start on a mismatch	every start, in production	exact	Blueprint runtime
The tool exists; its arguments are legal	JSON Schema validation before dispatch; the service validates again	every call, in production	exact	Blueprint runtime; the service
The trajectory: right tools, right order	create accepts only a hash that preview returned in the same episode	every call, in production	exact	Blueprint runtime
A claim in the reply holds	A judge on a different model reviews the result before approval	every request, in production	sampled	Blueprint runtime (review phase); Agentic-CI review lenses*
Product intent	A person defines and approves the intent at a hard gate; no run starts without it	before every run	a person, behind a gate	Agentic-CI; the approval gate in Blueprint
The run ended cleanly	Exactly one terminal event, with a reason	every request, in production	exact	Blueprint runtime
The request was routed correctly	Hash of the chosen criteria ∈ the declared set of acceptable answers	every eval run, offline	exact, once the acceptable answers are declared	64 cohort scenarios
The goal state was reached	The segment exists; its member count and user ids match ground truth	every eval run, offline	exact	a seeded test tenant
A service still honours its API	Integration test collections per service (Bruno)	against dev	exact	loyalty services; Agentic-CI’s own API
A change is tested before it ships	QA plan, recheck against the change, QA run on dev, each behind a gate; test cases kept in per-feature regression suites	every pipeline run	exact where a test runs; sampled where a lens reviews	Agentic-CI
Tone, locale	A lint for banned words, then an independent model’s read	every piece, before publish	exact for the lint; sampled for the read	this publishing pipeline
The checks themselves run	A mutant deletes each check’s wiring; a named test must turn red	every build, in CI	exact	Blueprint, this publishing pipeline, Cortex, our knowledge base
The measurement happened at all	ok / FAIL / no-signal; skipped rows and unmeasured axes reported by name	every run of the harness	derived	this pipeline’s health report; Cortex; the Blueprint eval runner (per row)

Sources: S1–S4, S8, S11, S13, S15–S17.

* Judge mutants: our next iteration of the concept. A judge mutant hands a lens a document with a planted defect and counts whether the lens flags it. Enough of them give a lens a measured catch rate in place of a typed-in one. The same method can measure the critic step of our GOD-prompt rotation (novelty, critic, tester), which we use to triage ideas and extract patterns. We will publish the numbers once the harness exists [S8].

Every check marked exact in Table 1 is plain code, with no model call. A judge costs one model call per sample [S2], and its answer is sampled, not exact.

Where exact checks win: the cohort agent

Our cohort agent turns one sentence from an operator, such as women from Madrid with Telegram, into a customer segment in a loyalty program. It runs on Blueprint, our agent runtime, in production.

Every layer has a deterministic check: plain code that runs on every request. Mutation testing is separate and runs in CI: it deletes the line that wires each check in and requires a named test to turn red [S3]. If deleting it changes nothing, the check was decoration.

in production · every requestcontract pinned: refuses to start on a mismatch · arguments validated before dispatch · judge review on a different model · approval by a person · provenance gate on create · one terminal event, with a reason

offline · every eval runrouting by criteria hash over 64 scenarios · exact verifier on a seeded tenant: count and user ids

in CI · every builda wiring mutant for each check above: delete the line, a named test must turn red

The schema is where most of the value sits. When we wrote the tool contract, the parser accepted age_in_range {} with no bounds, read it as any age, and returned a plausible count [S1]. Nothing downstream could tell that from a correct answer. The fix was a schema that requires the bound, not a better judge.

On the seeded test tenant the agent scores 15 of 15, exact on count and user ids [S6]. The earlier version scored 15 of 15 with a GPT model, on counts alone [S7]. That is a ceiling, not a result. Our spec says so: 15 of 15 “proves the plumbing, not the policy” [S3]. A verifier that everything passes cannot rank two policies [S5].

Where a judge earns its place: Agentic-CI

Agentic-CI is the internal Agentic CI we use in production. It takes a change from idea to deployed code through gated phases, with LLM review lenses at each gate.

Lenses catch what no assertion can. Three times, a lens caught a document contradicting itself, with no memory of what the document was meant to say. They also fail where a two-line deterministic check does not. One lens invented a file:line citation; “every citation must resolve” catches that with no model in the loop. The most valuable check the evaluations proposed has no model in it either: a conclusion about N services must show a search across N services. And five times, a correct observation was raised as a defect when the design was intentional: the lens saw what the code does, not why [S8]. Agentic-CI keeps every step’s artifacts, from the task brief to each review [S18], so the why can be written down; our evals proposal asks for exactly that [S8]. A person still defines and approves the intent, at a gate [S16].

A judge you have not calibrated is an unmeasured instrument [S2]. Strong judges validated against people reach more than 80% agreement with them, the level people reach with each other (Zheng et al., 2023). The validation is what counts, and the next section is how it gets earned.

Checking the checks

A check you have never seen fail is taken on trust. Mutation testing replaces the trust with a test: break what the check guards, on purpose, in a copy, and see whether it notices (DeMillo et al., 1978). If it stays green, it was not checking.

An eval is a check, so the same test applies. For an exact check it is simple: swap one user id in a segment, and the verifier has to fail. For a judge, hand it a document with one planted defect and count how often it flags it. Enough planted defects give the judge a measured catch rate instead of an assumed one. That is the judge mutant in the note under Table 1, our next step [S8].

What a battery finds is where checks are missing. The first one against this publishing pipeline broke 15 properties it claimed to guarantee. The checks caught 11, and all four escapes were in one script: the one that marks a page published. The test suite was green throughout [S10]. The score hides that. The location is the finding.

Two limits. A battery that catches every mutant covers the properties someone thought to break, and nothing else [S10]. And the first run of a harness mostly tests the harness: two of our first five memory mutants were broken as written, and both produced plausible results [S2].

Where a typed classifier fits

A typed classifier is a judge with a fixed answer type. TypeSafe describes its model, Jev, as “a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out.” It “gives up string generation” and is built for “smart if-statements… classify, route, score, extract, or branch” (TypeSafe, 2026). The speed and accuracy claims on that page are TypeSafe’s.

That shape fits several rows of the map: is this reply in Ukrainian?, is this prompt an injection attempt?, which of these three branches? Each is a closed question with its evidence in the input, which is where our own decision table puts a guard [S2].

It does not fit three other things an eval needs.

A claim nobody supplied. A classifier checks the claim in its input; it cannot propose one. Our agents’ memory cards come with no pre-registered answers, no list of which card should answer which question [S12], so there is nothing to hand it. We test memory by mutation instead. In one probe a single fact lived in one card. With the card intact, the agent chose the same next step three times out of three. With the card deleted, corrupted or blanked (four mutants), it chose a different step every time and said the fact was missing [S2]. The test needed no answer key for the card, only the agent’s own decision with and without it.

A reason. One of those mutants was itself broken: it left a visible placeholder in the card. We caught it only because the agent’s written justification quoted the placeholder [S2]. A model that gives up string generation can return a reason code its schema lists, but not a justification in its own words.

A question outside its type. “Where?” has no yes-or-no answer. In fairness to the design, TypeSafe says every answer carries a calibrated probability. That is a graded I don’t know inside the type: set a threshold and abstain below it, which is what selective classification formalises (Geifman & El-Yaniv, 2017). What a type cannot say is this question is not mine, unless its schema was written with that answer in it.

The third state

Our most misleading result was a zero for a run that never ran. Of the cohort agent’s 64 scenarios, 10 were skipped with a stated reason, and all 54 others failed to reach the database. No episode completed. The result file records pass@k = 0 [S6]. The same harness records skipped rows with their reasons and lists the axes it could not measure. It is honest per row and wrong in the headline.

run

→

executed, and property observed?if not → no-signal

→

ok

FAIL

no-signal

our corpus run: 54 attempted · 0 executed · recorded aspass@k = 0

The fix is a third outcome, no-signal, derived from the run rather than asked of the judge: did the run execute, and did the check observe its property? Our pipeline’s health report shows ok, FAIL or no-signal per component and ranks no-signal above ok, because unmeasured is where every failure in that repository came from [S13]. The mutation script for our knowledge base refuses to print a number on a red baseline, and Cortex reports NO-SIGNAL when it cannot run a check [S11].

Asking is not enough. The score sheets for our own human reviewer offered ?: “genuinely can’t tell — legitimate, and it is data.” Across 33 scores it was never used [S14]. The first sheet also produced a neat position pattern: 0, +1 and −1 by slot in three of five groups. Position bias is real in LLM judges (Wang et al., 2023), but this one came from our sheet: fixed order, paraphrased answers. A shuffled sheet with the models’ own words and the ground truth removed it [S14]. Models can estimate whether they know an answer (Kadavath et al., 2022). In our four cases, the gap was in the instrument.

What’s next

Our agents remember through small memory-cards, each carrying its own coordinates: lane, chapter, tags, status and date. Next we test reading them the way associative memory works. Draw a hard window over those coordinates, a stencil, then let a Hopfield-style update pull out the cards that belong together. We wrote the pass bar before running anything, and it has three possible outcomes: PASS, FAIL and NO-SIGNAL.