Kerf Online against Firecrawl, on 639 RAG questions

Almost nobody hands a whole page to a model. A retrieval pipeline cuts the Markdown into chunks of a few hundred to a couple of thousand characters, stores them, and at question time hands the model the few that look most like the question. What the model can answer depends on whether the fact it needs landed in one chunk, with enough around it to say what it is.

That is a property of the extraction as much as of the chunker. A price that arrives on its own line, three headings away from the name of the product, is retrievable only by luck. A table row that arrives without its column names answers nothing. A page that came back as a shell, because its figures are written in by a script the extractor never ran, answers nothing at all, however tidy the Markdown looks.

Why not measure this with the structure scores we already publish? Because they measure the wrong end of the pipe for this question. Headings kept is a good proxy for a person reading a document and a poor one for a retriever, which does not read headings, and the same page can score well on structure and badly on answers. So this study puts a retriever between the extraction and the reader and counts answers.

Pricing pages, product pages, API references, standards, encyclopaedia articles, repositories and essays: pages with facts in them that someone would ask a model about. Up to six questions were written for each page by a model that could see two extractions of it, in random order and without knowing which product made which, and was told to draw half its facts from each. Every question names a short gold answer.

github.com/openssl/openssl. A repository page: a README inside a great deal of application chrome, with the counts and the latest release written in by script after the page loads.

Models answer and grade. A person did not read 639 answers three times over. The audit above says how far to trust that, and the noise figure says how much of a gap it takes to mean anything. Both are published for exactly that reason.

The rounds are not one fetch. Kerf was fetched again as it improved, over 10 rounds, and each page's answers come from the latest round in which its extraction changed. Firecrawl was fetched once and its documents never changed; its answers were regenerated beside each Kerf round so the two columns on any page always come from the same sitting. This dataset was also used during Kerf development, and as such, treat it as a transparent development benchmark rather than a held-out generalisation test. We will run the final implementation unchanged against a fresh frozen set.

One chunker, one retriever. 1,200-character chunks cut at headings, and BM25 over them, are a common production default and not the only one. We checked the shaping at 500 and 2,000 characters too and the direction held, but a pipeline with a different chunker or an embedding retriever will see a different size of gap.

Kerf was called with links off. That is what a retrieval caller sends, since addresses match nothing anyone asks, and it is the setting the option is meant to sit beside. Firecrawl was called at its defaults, which keep links.

The shaping trades structure for retrieval. Fewer headings and no Markdown tables come back with the option on. On the structure measures our other study scores, the same Markdown does worse, which is why the option is off by default and this page shows both columns.