Half the calls, twice the speed, and citations that cannot be invented

Attest checks a claim against the documents you send it and returns a verdict with the exact sentences behind it. The sentences are the product. A verdict nobody can check is an opinion, and the whole reason to buy a fact checker rather than ask a model is that you can go and read what it read.

Producing that record cost three language model passes on top of the verdict, four calls per claim in total. Worse, those passes fed corrections back into the verdict, and they were not good at it. On our labelled set they were costing 2.2 points of accuracy and roughly a third of the overclaim rate, which is the number we tell customers to care about.

So why keep them? Because deleting them deletes the citations, and the citations are the product. We needed the record to come from somewhere else before the passes could go.

TypeSafe describes Jev as its flagship model and the first of what they call System One models: “No text generation, no parsing”, but typed values and probability distributions that your code can branch on. You hand it a body of state once and ask questions of it. A Choice picks one option from a list you define, a Noul returns a truth value between zero and one, and a Score places the state on a rubric. One request carries as many questions as you like against one copy of the state, and the answers come back under the keys you chose.

That last property is the one that ends up mattering, and not for the reason we expected. We reached for it as a way to make a cheap decision cheaply. It turned out to be a way to ask for a great many small structured things at once and get them all back in the time one of them would have taken.

The record was the job we had in mind, but it would have been careless not to check the more ambitious one on the way past. So we tried it as the grader: 5 arrangements, on its own, as a one-shot holistic grader, over our rubric, as a first pass that handed on what it was unsure about, and inside a hybrid gated on its own confidence. On its own it reaches 82.8% where the language model reaches 93.8% ±1.0. Once we looked properly at what our grading step actually asks for, that is what we should have expected, and it says more about the shape of our task than about the model.

A verdict here often turns on a discrepancy between documents. Draft minutes against final minutes, a figure on one stated basis against the same figure on another, an amendment that governs the clause it amends. Settling one of those is a chain rather than a judgement: hold one passage, find the other, work out which of them governs and why. That is reasoning, and reasoning is what a reasoning model is for. It is also, measured, exactly where our own check reaches its ceiling: the overclaims it does not catch are the ones where no single passage contradicts the claim and the contradiction only exists once you put two of them together.

And the arithmetic was never going to be any model’s job. Claims in this material are routinely derived rather than stated: an overrun of 19.6%, a bridge that has to sum, revenue per employee above a threshold. We settle those in code, with a hand-written parser, on a rule we had long before Jev arrived: the model decides what to compute and the code computes it. TypeSafe say much the same about where their model is weak. A grader that should not be doing arithmetic is not a shortcoming we discovered; it is a constraint we had already built around.

So the grading contest was the wrong question. What we were really asking was whether a model that answers in one step could do a job that needs several, on material where we deliberately keep the numbers away from models altogether. The useful question was the other one: what is it better at than what we already had?

A citation is not prose. It is a choice among the sentences in front of you. So we stopped asking for the record in words and started asking for it as Choice questions whose options are the span ids actually in the packet: which sentence states the assertion, which carries the value, which establishes the entity and period, which tells against the claim.

A model that can only return a key you defined cannot cite a sentence that does not exist. Referential integrity is not validated afterwards, it holds by construction, and across 170 claims we measured it at 1.000 with 0 fabricated references. The passes we were replacing could and did invent span ids, which is why there is a validator in the codebase whose whole job was catching them.

The probabilities that come back are worth as much as the answers. They are well calibrated, and on the third of claims it is surest about it agrees with the language model almost exactly: 97.4% against 97.6%. That is not a number we can route on, because being sure and being right about hard things are different properties, but it is the reason a check built on it holds its shape rather than firing at random.

With the record coming from one typed call, the three explanation passes had nothing left to do, and we removed them. That is the whole mechanism, and it is worth being exact about who did what, because the flattering version of this story is not the true one.

A verdict Attest was not sure about used to reach you looking exactly like one nothing doubted. There was no way for it to say so, because nothing was asking. Now something is, and a report the check is uneasy about arrives marked as held rather than settled, with the sentence to go and read.

Once the questions were already being asked, one more cost nothing. The verdict is restated as a plain sentence about the evidence, because a label like PARTIALLY_SUPPORTED is the name of a bucket and not something you can check, and Jev is asked whether that sentence is too generous to the claim, and separately whether anything in the evidence tells against it that the sentence fails to account for.

It is not allowed to change anything, and that is a result rather than a preference. We tried the obvious thing first: when the check fires, step the verdict down a level, on the theory that a verdict credited with too much should be credited with less. Over 1,098 audited verdicts it corrected nothing at all. The overclaim rate was 3.0% before and 3.0% after, and accuracy fell by most of a point, because every time the check was wrong it had damaged a verdict that was fine.

Why it failed is the reason Attest exists. Twenty-one of those 33 overclaims were not inflated confidence but missed refutations: the evidence contradicted the claim and the report walked straight past it. Stepping a verdict down says “less of this claim is established”. It never says “the evidence is against you”. The repair was moving along an axis the mistake was not on, because it was derived from the same reading of the documents that produced the mistake, and that reading had not seen the refutation in the first place.

A model cannot correct what it did not notice. Ask it to soften its own answer and you get a softer version of the same blind spot, which is exactly why a fact checker built out of one model marking its own work is not worth much. So the check flags and a person reads, because the only thing that reliably matches a missed refutation is somebody going back to the document.

The same 190 claims rebuilt as document sets, with 6 unrelated filler documents each so the search has to find the right passages among others. The agent’s verdicts are 84.7% accurate with 6.8% overclaims, against 93.8% ±1.0 and 3.1% ±0.4 when the same claims arrive with their evidence attached. Finding the evidence is most of the difficulty, and a reader cannot see that difficulty from the outside.

The check holds 11.6% of those results and takes the overclaims reaching a reader from 6.8% to 2.1%. We fitted its sensitivity on that data rather than carrying over the setting from the easier path, because a threshold belongs to the evaluator and the composition it was measured on and to nothing else.

We are really looking forward to getting this one out to you, and there are a couple of things we want to finish behind the scenes first. The Attest you can call today is still the one its own page describes, spending 4 model calls on a claim and writing its record in prose.

The first is citations on the other path. When an agent searches a document set you uploaded, what it cites is still checked after the fact rather than being impossible to get wrong, and we would rather hand you one guarantee that holds everywhere than two that each hold half the time. The second is making a held report obvious wherever somebody reads one, because a report we held back that still reaches you looking like all the others has not really been held back at all.

Neither is far away. When they are done the figures on the Attest page move with the service rather than ahead of it, the verdicts printed there are captured again from the real thing, and we will come back here and say what changed.

Look for the work around your model’s decision rather than the decision itself. Ours was a record we were generating in prose because prose was what we had, and it was the slowest and least reliable part of a system whose selling point is that record. We did not go looking for somewhere to put a new model. We had a part of Attest we already wanted rid of, and this is the first thing we have seen that could do that job properly.

Ask for structure the model cannot get wrong. A citation as a choice over ids that exist cannot be a citation to something that does not, and that is a stronger guarantee than any amount of checking after the fact. And ask for everything at once: the difference between our five-question call and our eleven-question call was eleven milliseconds in the wrong direction, which is to say nothing at all.