The runs behind the figures
Our product pages carry numbers. The runs behind them are published here, with the day each was run and what came back.
Quorum Technologies · newest run 22 September 2026
01
The latest
Quorum × Jev: half the calls, twice the speed, citations that cannot be invented
Three language model passes replaced by one typed call, and what that was worth in speed and in citations that cannot be invented. Includes why our grading step is the wrong job for a decision-only model, and which job is the right one.
4 → 2
model calls per claim
3202 → 1294ms
per claim, measured end to end
0 in 170
citations invented, where the passes could and did
A second reader that catches Attest overreaching, on 190 claims
Every verdict checked by a model that writes no text and only answers typed questions, asked whether the conclusion is more generous than the evidence. Five runs against one of the deployed product, with each run published.
1.4%
of overclaims reach a reader, from 4.7%
6.0%
of reports held for a person
5 v 1
runs, because the deployed product can only be measured once
Attest on 190 hand-labelled claims
A labelled set of claims: multi-hop reasoning, arithmetic, distractors, scope traps, conflicting sources. The run also found a defect, published on the same page.
91.6%
verdict accuracy
4.7%
overclaim rate
190
hand-labelled examples
Kerf Online against Firecrawl, on 639 RAG questions
The same pages read by both, questions asked of the chunks a retriever picked from each extraction, answered and graded by models that never knew which was which.
83.7%
answered correctly with optimiseForRAG
78%
for Firecrawl on the same questions
639
questions, graded blind
02
Everything we publish
| Date | Subject | What we published | What it is | Where |
|---|---|---|---|---|
| 22 Sept 2026 | Attest | Quorum × Jev: half the calls, twice the speed, citations that cannot be invented | Study | Full run |
| 18 Sept 2026 | Attest | A second reader that catches Attest overreaching, on 190 claims | Study | Full run |
| 18 Sept 2026 | Attest | Attest on 190 hand-labelled claims | Study | On /attest |
| 14 Sept 2026 | Kerf | Kerf Online against Firecrawl, on 639 RAG questions | Study | Full run |
| 13 Sept 2026 | Kerf | Kerf Online against TinyFish, on 76 addresses | Study | Full run |
| 12 Sept 2026 | Kerf | A pricing page and a deck, as Kerf read them | Saved call | On /kerf |
Study: a fixed set, a method written down before the run, every per-item result published. Where the set was also in use while the product was being developed, the study discloses this. Saved call: one real answer, replayed on the page it illustrates.
Dates are the day the work ran, not the day the page shipped. A repeated run replaces its figures and its date together.
The savings calculators are not in this list. They are assumptions about how long a job takes by hand, and they say so where they sit.
03
How we publish
01
Freeze
The set is written to a file with a timestamp before anything is called. The runner reads only that file.
02
Run
Each provider is called once on each item. The answer is saved verbatim, next to its source. No retries.
03
Score
The saved answers are measured against that source, or against a label written beforehand.
Where the measure allows it, no model marks our work: extraction is scored against each page's own HTML, claim verification against labels written before the run. Where a study needs a model to answer or grade, it is blind to which side it is reading, audited, and the measured agreement is published.
Runs that go badly are published as well. Attest's own measurement turned up a robustness defect, and it sits beside the accuracy figure on the Attest page.
They are also how we decide what to fix. What a run turns up in our own service becomes work, and the next run is where you see whether the work landed.
Where only one column of a comparison has been run again, both dates are printed. In the extraction comparison, Kerf's column was re-run after changes to how it reads pages; TinyFish's answers are the ones saved on the day, and every page is still scored against the HTML saved then.
The extraction run recorded three failures which turned out to be our own edge reporting a site's refusal as our error. They stay in the run as they happened, with the re-check under them rather than in place of them: a run is a record of a day, not a claim about today.
04
Ask us to measure something
If a figure you need is missing, or you would rather see the comparison run against your own pages than against ours, say so. We will run it and publish what comes back.
Attest's labelled set is one author's work on one distribution, and hosted inference is not bit-reproducible, so small differences between runs are noise. Both caveats are stated on the Attest page. If your documents look nothing like that set, the useful number is one measured on yours.
Figures are read from the run that produced them rather than typed onto a page. Naming another product is not a claim about anything except what it returned on the addresses we tested, on the day we tested them.