Public benchmark

What it measures ​

The benchmark quantifies what each conformance level of the index-ai ladder (SPEC §13.1) buys an agent: token consumption per query and correct citation rate, across five benchmark query types. For every site, the harness consumes exactly the payloads the protocol requires an agent to read at that level — the full HTML at L0, the AI Manifest at L1, the Agent Index plus a targeted fetch at L2a/L2b, the query response at L3 — then measures the token cost (SPEC §9.3 heuristic — NFC characters ÷ 4) and whether the payload contains the ground-truth answer (a deterministic citation check).

Every number above is a fact from the published run: 250 synthetic sites, 1,250 queries, regenerated deterministically and locked by integrity tests.

The shape of the result is stable across levels: reading the manifest instead of the HTML cuts token consumption ~6.6×, Levels 2a/2b reach 100 % citation at ~15 % of the L0 cost, and Level 3 keeps 100 % citation at 22 %. Level 1 remains a scoped contract — it answers who publishes a site and when it changed (100 %), and by design carries no content, which is why it cites 40 % and not more. That is the ladder working as specified.

Corpus ​

The corpus is synthetic and deterministic (benchmark/corpus.mjs): generateCorpus({ seed }) always produces the same sites, and the seed is recorded in every results file — anyone can regenerate the exact dataset. Synthetic sites are used because external implementations of index-ai are still "none yet" (SPEC §13.5): the benchmark measures the mechanism of each level on identical, realistic content (hotels, blogs, docs, e-commerce, news verticals), not real-world adoption.

Methodology ​

For each (site, query) pair the harness (benchmark/run.mjs) consumes exactly the payload the protocol requires, then measures it:

L2a and L2b consume the same payloads on this corpus (single-page graphs — the difference appears on paginated graphs, tracked as RFC-006). Full protocol, metrics, and governance: benchmark/README.md.

Reproducibility ​

# reproduce the published dataset exactly
pnpm benchmark        # = node benchmark/run.mjs --sites-per-level 50 --seed 20260813 --status Published

# verify integrity (meta, aggregates, token order, citation matrix, identical regeneration)
pnpm vitest run tests/benchmark-full.spec.ts

The seed, the token rule, the status, and the level count are recorded in the meta block of every results file. A change of methodology or seed is a new results series, never an edit to a published one (immutability policy §17.2).

Limits — what these numbers do not claim ​

• Synthetic corpus: the run measures the mechanism of each level, not real-world adoption or cost savings on the live web.
• Citation by containment: the answer is checked as a substring, not by semantic equivalence — an agent that answers without the exact substring counts as a miss.
• English-only token heuristic (§9.3): chars ÷ 4 is calibrated for English; per-language rules are not exercised.
• L3 is a projection, not a live server: the query service exercises the §8 contract shape deterministically.
• Node selection is modeled: the two-phase relevance heuristic is a stand-in for a real retrieval step.
• No network variance: the corpus is served from localhost.

These numbers validate the ladder's efficiency in mechanism — they do not estimate implementation effort, real tokenizer values, or non-English content.