Kerf Online against TinyFish, on 76 addresses
We called both providers on the same fixed set of pages, saved every answer before scoring any of it, and scored against each page's own HTML rather than asking a model. Here is the whole run: why it matters, what we compared, how it was measured, every per-address result, and what the numbers do not show.
Quorum Technologies · run 12 September 2026
01
A table that arrives as a paragraph has lost the thing that made it a table
Whatever reads a page next, a model or an index, only ever sees what the extractor handed it. Everything after that is downstream of one decision nobody watches.
A pricing table is a grid before it is text. Every figure in it means what it means because of the row and the column it sits in. Flatten it into prose and the numbers survive while the thing that made them answerable does not: ask which tier includes SSO and the model is now guessing from a list of words in a row.
The same is true of a heading. A heading is not decoration, it is the address of a section. Drop it and a retrieval index has nothing to cite and a reader has nothing to jump to, while the prose underneath still reads perfectly well, so nobody notices until an answer comes back wrong.
Why not just measure how clean the output is? Because a cleanliness score rewards throwing things away. An extractor that returns the first three paragraphs of every page scores beautifully on noise and is useless. Anything worth measuring has to count what arrived as well as what did not, which is why the two figures below it are headings kept and headings in order.
02
What we compared
Two products that turn a URL into Markdown. Each was called through its own public API at its own defaults, with no tuning on either side and no retries.
| Provider | Called as | How it works |
|---|---|---|
| Kerf Online | POST /v1/render | Ours. Renders the page, measures the boxes the browser drew, and reads them by layout rather than by source order. Reads most pages straight from the markup and only opens a browser when the page needs one. |
| TinyFish | Fetch API | Browser-backed rendering. Loads pages in a real browser, strips non-content elements and returns Markdown. Described here as they describe themselves. |
We picked TinyFish because they publish their own evaluation of this exact problem, with their method and their URL list in the open. That is rarer than it should be, and it makes a comparison possible at all. The 1,000-character floor below is theirs, kept so the coverage figure means the same thing in both places.
03
The test set
76 addresses across ten kinds of page, plus five behind a bot wall that are reported on their own.
Every address was chosen to still resolve to the same document next year: versioned documentation, published standards, licences, frozen encyclopaedia subjects, and essays nobody edits after publishing. A set built from dated news articles ages out within months, and a benchmark that cannot be run again says nothing about whether anything improved.
| Category | n | What is in it |
|---|---|---|
| documentation | 10 | API references and guides. Dense prose, code samples, deep heading trees. |
| reference | 8 | Encyclopaedic pages. Long, heavily sectioned, full of tables and footnotes. |
| standards | 8 | Published specifications. Never edited once issued, and numbered throughout. |
| pricing | 8 | Tiers and feature grids. The shape a flattening reader destroys. |
| repositories | 8 | READMEs inside heavy application chrome. |
| longform | 12 | Essays nobody edits after publishing. |
| spa | 8 | Drawn entirely in JavaScript. The pages a fetch without a browser loses. |
| tables | 6 | Pages whose whole point is a table. |
| legal | 6 | Licences and terms. Structured, numbered, never rewritten. |
| simple | 2 | Almost no markup at all. The floor case. |
Counts are the addresses that were scored. The five walled addresses are listed separately below.
Were the addresses chosen before anyone was called? Yes, and the harness enforces it rather than asking you to take our word. A freeze step checks every address answers and writes the list to a file with a timestamp; the runner reads only that file. An address cannot be swapped once somebody has seen a result.
04
How we score
Against each page's own HTML, so reading order and structure are measured rather than judged. No model marks this homework, which is what lets anyone run it twice and get the same answer.
| Measure | What it counts |
|---|---|
| Pages returned | At least 1,000 characters came back. Anything shorter is a stub, not an extraction. |
| Headings kept | How many of the page's own headings arrived at all. This is the measure that punishes throwing content away. |
| Headings in order | Of every pair of headings the page put in an order, how many came back in that order. |
| Tables kept | On pages whose source holds a table, whether a table came back rather than prose. |
| Code kept | On pages whose source holds a <pre> block, whether code came back as code. |
| Characters | How much came back. Reported beside the noise count, because either alone can be gamed. |
| Time | Wall clock as the caller sees it, including the render where one was needed. |
Why 1,000 characters? It is TinyFish's own floor, taken from their published evaluation and kept so that the coverage figure means the same thing in both places. It exists because some providers report success while returning a truncated excerpt.
Was any threshold chosen to flatter anybody? No. The floor is inherited, and every other measure is a count with no threshold to tune. The one judgement call, that only a <pre> block counts as code and inline <code> does not, was made against our own interest: it was found because the first scoring run showed us losing on code retention, and looking at why showed the other provider wrapping single words like title in fenced blocks and being credited for it.
05
What came back
Every figure below is produced by the scorer in the kerf repository and nothing on this page is typed by hand.
| Measure | Kerf | TinyFish | What it means |
|---|---|---|---|
| Pages returned | ● 76/76 | 75/76 | At least 1,000 characters came back. |
| Headings kept | ● 90% | 76% | Share of the page's own headings that arrived at all. |
| Headings in order | 100% | 100% | Of every pair the page ordered, how many survived in that order. |
| Tables kept as tables | ● 28/29 | 8/29 | Of the pages whose source holds a table. |
| Code kept as code | ● 38/38 | 36/38 | Of the pages whose source holds a <pre> block. |
| Median characters | ● 19,654 | 18,008 | More of the document came back, at the same measured noise. |
| Median time a page | ● 587ms | 1,724ms | Wall clock, as the caller sees it. Less is better. |
Median figures across 76 scored addresses. ● marks the better of the two.
05b
By kind of page
Pages returned, per category. The set is built so no one kind of page can carry the result.
| Category | Kerf | TinyFish | What is in it |
|---|---|---|---|
| documentation | 10/10 | 10/10 | API references and guides. Dense prose, code samples, deep heading trees. |
| reference | 8/8 | 8/8 | Encyclopaedic pages. Long, heavily sectioned, full of tables and footnotes. |
| standards | 8/8 | 8/8 | Published specifications. Never edited once issued, and numbered throughout. |
| pricing | 8/8 | 8/8 | Tiers and feature grids. The shape a flattening reader destroys. |
| repositories | 8/8 | 8/8 | READMEs inside heavy application chrome. |
| longform | 12/12 | 12/12 | Essays nobody edits after publishing. |
| spa | 8/8 | 8/8 | Drawn entirely in JavaScript. The pages a fetch without a browser loses. |
| tables | 6/6 | 5/6 | Pages whose whole point is a table. |
| legal | 6/6 | 6/6 | Licences and terms. Structured, numbered, never rewritten. |
| simple | 2/2 | 2/2 | Almost no markup at all. The floor case. |
06
One page in detail
A summary table can hide what a difference actually looks like, so here is a single address, start to finish.
RFC 9110, the HTTP semantics specification. A long, heavily numbered document: the kind of page an agent is asked about precisely because nobody wants to read it.
| rfc-editor.org/rfc/rfc9110.html | Kerf | TinyFish |
|---|---|---|
| Characters returned | 442,587 | 5,325 |
| Headings kept | 100% | 1% |
| Table kept as a table | ● | ○ |
| Code kept as code | ● | ○ |
| Time | 1,306ms | 3,814ms |
One address, both providers, from the same run.
Both fetches count as successful. Both clear the character floor, and a coverage-only scoreboard would record one point each. But one of them contains the specification and the other contains a fragment of it: the shorter answer is section 6.4 and its two subsections, and nothing before or after them.
That is what “headings kept” is for. Ask either result which request methods are idempotent, which the specification answers in section 9.2.2, and only one of them contains that section. Nothing about the character count on its own would have warned you which.
07
Every address, and what each one returned
The whole dataset. Characters returned, time taken, whether the page cleared the floor, how much of its heading structure survived, and whether a table stayed a table.
● kept · ○ lost · – the source has none. Order is blank where too few headings came back to say anything.
08
The five behind a wall
These answer 403 to anyone. They are reported here and counted nowhere: a provider that returns nothing because a site said no has not done anything wrong, and one that returns a paywall preview has not done anything right.
| Address | Kerf | TinyFish | Kerf, asked again 12 September 2026 |
|---|---|---|---|
| www.nytimes.com/2026/04/28/opinion/ezra-klein.html | nothing | nothing | 424 site_refused |
| www.bloomberg.com/markets | 12,040 | 283 | 11,581 characters |
| www.glassdoor.com/Reviews/index.htm | nothing | 9,642 | 424 site_refused |
| www.indeed.com/career-advice | 11,347 | nothing | 11,347 characters |
| www.crunchbase.com/organization/anthropic | nothing | nothing | 424 site_refused |
Characters returned in the run, and what Kerf answers now. No column feeds any figure above.
In the run, three of Kerf's five came back as an HTTP 502 from our own edge. The site had refused, but the edge reported that as its own failure, and behind the CDN the refusal never reached the caller. That was our error, and the run records it as it happened. We fixed it and asked Kerf again on 12 September 2026: the same three now answer 424 site_refused, which names the site's answer instead of hiding it, and the other two still extract. Nothing in this table feeds any figure above, then or now.
09
What this does not show
Three steps, kept separate so that no result can change the set and no threshold can be chosen after the fact.
01
Freeze
Every address is checked and written to a file with a timestamp. The runner reads only that file, so an address cannot be swapped once somebody has seen a result.
02
Run
Each provider is called on each address and what comes back is saved verbatim, next to the page’s own HTML. Nothing is scored at this stage.
03
Score
The saved answers are read and measured. Thresholds are a judgement call, so scoring re-runs against the same files as often as the argument needs.
Everything is scored against each page's own HTML, so reading order and structure are measured rather than judged. No model marks this homework, which is what lets anyone run it twice and get the same answer.
One address was dropped. A page is removed only when the whole document, markup included, is smaller than the 1,000-character floor. example.com is 559 bytes end to end, so no extractor alive can pass it and both providers fail it however good they are. The rule keys on a property of the page rather than on anything either provider did. It was added after this run rather than before it, which is the weaker position, and it removed the same single row from both.
The run asks Kerf to skip its robots.txt check. Honouring robots.txt is the default on every other request Kerf makes. The other provider evidently does not honour it, and leaving ours on scored our own compliance as our failure on the pages that disallow a crawler. This measures extraction against extraction.
Heading order is only scored where at least three headings came back. Fewer than that and there are not enough pairs to say anything. That slightly flatters a provider which drops headings, because dropping them can take a page below the threshold instead of scoring it badly. Headings kept is the measure that catches exactly that, which is why both are reported and neither summarises the other.
The address list is new. An earlier run of this comparison was lost with the machine it lived on, so these figures are a fresh measurement rather than a continuation of it.
10
Do it yourself
The harness, the frozen address list and the scores are in the kerf repository.
node bench/headtohead/freeze.mjs KERF_TOKEN=… TINYFISH_KEY=… node bench/headtohead/run.mjs node bench/headtohead/score.mjs
Run 12 September 2026 against a set frozen 12 September 2026. TinyFish is called through its own public API at its own defaults; we have tuned nothing on either side and there were no retries. Naming another product is not a claim about anything except what it returned on these addresses on this day.