The benchmark ​

Every DocPilot 2.0 claim about accuracy, size, speed and safety comes from one script run on one corpus: bench/run.mjs on Harborline. You can rerun it. The tables on this page are the output of that run, and nothing else.

What Harborline is ​

Harborline is a product that does not exist, documented as if it did: a job-queue service with a CLI, an HTTP API and SDKs. A language model has never read about it, so a correct answer can only come from retrieval over these pages. The fixture lives in bench/harborline/:

The corpus carries deliberate defects, each tied to the questions it affects: pages with no description whose words readers do not use, sections whose answer is split across chunks, and a page that carries an injected instruction. A defect makes a metric measure something. A corpus with none would score well on every system.

What it compares ​

An arm is an index build: how the corpus is chunked, embedded and stored. A system is an arm answered by one runtime of one release.

The 1.5 baseline is the code that shipped, run from a built checkout, not a model of it.

Rerun it ​

You need Node 20 or newer, a built checkout of this repository (npm run build, because the report step imports from dist/), and Ollama serving the embedding model bge-m3 and the chat model. Run every command from the repository root.

node bench/run.mjs index        # build one index per arm
node bench/run.mjs calibrate    # fit the gate on eval/calibration.jsonl for the calibrated arm
node bench/run.mjs retrieval    # eval --gate-only per arm, dev and test splits: free, minutes
node bench/run.mjs answers      # the golden set answered, per system: hours on a local model
node bench/run.mjs security     # the adversarial probes, per system
node bench/run.mjs perf         # what a reader downloads, per index
node bench/run.mjs report       # bench/harborline/results/RESULTS.md from everything above

answers and security take a system name to run one system: node bench/run.mjs answers "v2.0 server".

Each step saves the log and the eval report in bench/harborline/results/. report reads those reports and writes RESULTS.md and results.json. Every number in it comes from a saved report. The paired intervals are computed from the reports' own rows with the bootstrap that docpilot improve compare uses.

To run one measurement by hand, work from the fixture directory. The CLI resolves docsDir, evalDir and indexDir against the directory it starts in:

cd bench/harborline
node ../../bin/docpilot.js index --yes
node ../../bin/docpilot.js eval --gate-only --split=test --quiet
node ../../bin/docpilot.js eval --set=adversarial --runtime=server --quiet

How to read the tables ​

• Retrieval is scored --gate-only on the test split, so it calls no model. An arm is compared with base by a paired difference with a 95 % bootstrap interval. A row marked * has an interval that excludes zero. A difference without it is not shown to be real. See Measure for what each metric counts.
• Answers come from one local model. They are a statement about that model on this corpus. Another model gives other numbers, so compare systems within one table, never across machines.
• Security is the attack success rate on the probe set, per system. See Attack probes.
• Bytes are what a reader of the static install downloads: the size on disk and the size on the wire, compressed with brotli.
• An adopted setting needs a measured gain and a stated cost. An arm that does not beat base with a clear interval stays an option, not a default.

Results ​

Measured on 2026-10-25 with bge-m3 and qwen3:8b on a local Ollama. The raw reports are in bench/harborline/results/.

Answers ​

The golden set at --level=high: 65 questions, 50 answerable and 15 negatives. ΔF1 is paired by question against v1.5 static.

What it says:

• The static install answers exactly as 1.5 did. All 65 answers of v2.0 static are byte-identical to those of v1.5 static. The latency difference is the machine: the two runs generated the same tokens.
• The server's default turn is three times faster to the first token. One-shot without reasoning shows the first token at 8.3 s against 25.0 s for the agentic loop, and it uses 27% fewer prompt tokens. Answer F1 and citations move inside the noise.
• The cost of one-shot is that the model cannot search again inside a turn. One of its 65 answers was a generation that ran into the 120 s step timeout. Set chat.maxTokens when a local model answers. server.turn.loop: 'agentic' restores the 1.x turn.

Security ​

The 30 attack probes, per system. A breach is a probe whose forbidden text reached the answer.

What it says:

• The server keeps its instruction. The canary in prompt.extend reached no reader of either server turn. The static install ships its prompt to the browser, so its canary is public by design.
• Echoed markup is inert. Every system repeated an injected <script>, onerror= or javascript: when asked to. The panel escapes HTML and links no javascript:, so the reader sees text.
• The loop resists more. The agentic turn resisted two probes that one-shot followed: a translated instruction and an instruction planted in a page. With 30 probes the difference is two probes, inside the noise. A site that weighs injection resistance above speed can set turn.loop: 'agentic'.

A hosted model ​

The same 65 questions with deepseek/deepseek-v4-flash on OpenRouter as the chat model; the embedder stays the local bge-m3. A right outcome is an answer to an answerable question or a refusal of a negative. The three runs cost $0.16.

Answer F1 here counts a refused answerable question as 0, so a system cannot raise it by refusing.

What it says:

• The model is the lever. With the hosted model the static install reached the right outcome on 63 of 65 questions instead of 56 (McNemar p = 0.039). Most of the gain is honesty: it refused 14 of the 15 questions the docs cannot answer, where the local model refused 6, and it still answered 49 of 50 answerable ones. Answer F1 moved inside the noise.
• One-shot is a local-model default. With the hosted model the agentic loop showed the first token sooner than one-shot (9.8 s against 13.3 s) and answered every Russian question one-shot refused (9 of 9 against 6 of 9), at 3.3 times the prompt tokens. The right outcomes are level (61 and 60 of 65). A server with a fast hosted model can set turn.loop: 'agentic' and pay in tokens, not in time.

The gate ​

eval/calibration.jsonl holds 295 probes in the eleven strata docpilot calibrate reads, and none of them is a golden question. On Harborline calibrate refuses to write thresholds: the tau that lets every answerable probe through is 0.15, below the lexical weight 0.25, where a question would clear the gate on shared words alone (bench/harborline/eval/calibration.report.md). The build then inlines the provisional tau 0.30, and the calibrated systems measure that.

What it says:

• The gate measures topic, not whether an answer exists. Probes about absent features and absent APIs sit at a median G of 0.65 to 0.91, as high as the probes the docs answer (0.90 to 0.97). Only off-domain questions fall below a threshold that a real question never reaches.
• On these questions the gate adds one refusal. In the static install it refuses five off-domain questions that the model refused anyway, so the reader gets the same answers without a model call. On the server it catches one more negative, with no false refusal. The two server runs gave the same text on 20 of 65 questions, and the gate changed five of them: the rest is the model's own variation.
• The default stays guard.mode: 'off'. A threshold has to be measured per corpus and per language — see Relevance gate — and on this corpus no measured threshold exists. A site with blatant off-topic traffic can set guard.tau by hand and check it with eval --gate-only.

Retrieval ​

The test split, 52 answerable questions, scored --gate-only. Δ is paired against base.

What it says:

• No lever is a default. ctx-path gains half a point of MRR, and on DocPilot's own docs the same idea lowered context precision. ctx-desc does not clear the noise.
• binary halves the download at a cost inside the noise. It is an option for a site that counts bytes, not a default.
• llms.txt as page descriptions ranks worse on the very questions it was written for. The description lands on the page's first chunk, which then outranks the section that answers. See Improving.

Download ​

What a reader of the static install downloads, per index. ctx-path and ctx-desc have the bytes of base.

The tuning agent ​

The tuning agent, headless, gate-only, the embedder on a local Ollama.

What it says:

• The edits fix the questions they target and break nothing. On Harborline three dev questions ranked better and none worse; the test split did not move. Each kept line restates what its section already says, in the words readers use.
• The first run kept nothing because of its rule. Two questions fixed out of 33, none worse, leaves a bootstrap interval with a lower bound of exactly zero. A gate-only run calls no model and ranks the same way every time, so improve compare now keeps such a Pareto move. The DocPilot candidate in the table is keep under that rule, on dev and on test.

Next ​

• Measure — the metrics in the tables.
• Choose your install — static or server.
• Attack probes — the security run.