CLI ​

npx docpilot <command>

Installed in a project, every runner reaches this bin by the bare name — npx docpilot, pnpm exec docpilot, yarn docpilot, bunx docpilot. To run it once without installing, the name has to be the package's own, because the unscoped docpilot on npm is not this package:

npx @cloflin/docpilot init

Every command finds .vitepress/config.mjs relative to the directory you run it from and reads the docPilot named export. There is no second place to state which model embeds or where the docs live.

The loop is index → calibrate → lint → eval → bench, with import ahead of it whenever the corpus gains a page from somewhere else, vocabulary ahead of it whenever the words readers use are not the words the docs use, and tune wherever it is retrieval itself that has to move. The first two are what the panel needs to work at all; the last three are what tells you whether it works well.

tune sends you back to the start. It writes a file and stops; index is the step that inlines a swept lever into the manifest a reader downloads, and until it runs a tuned lever is a file on disk and nothing more.

feedback sits outside the loop. It reads what your own endpoint collected and proposes probes for it; it never writes to the eval sets.

npx docpilot with no command, in a terminal, opens the home screen: the status picture of the project and a menu of what to run next, the next step first. Piped, or under CI, it prints the help instead.

Three things are true of every command and are documented once, at the foot of this page: help and --version, how .env.local is read, and what each exit code means.

index ​

npx docpilot index
npx docpilot index --dry
npx docpilot index --no-embed
npx docpilot index --html-dir=dist
npx docpilot index --html-dir=dist --html-exclude=".toc"
npx docpilot index --refresh-embeddings
npx docpilot index --quantize=binary

Builds the retrieval index into docs/public/rag/ (or wherever indexDir points). --dry chunks and reports without embedding — no network, no model — which is the loop for tuning chunking.

Idempotent: identical input produces byte-identical output.

It asks which embedder to build with ​

The first answer is what your config already names, so pressing Enter changes nothing. The rest of the list is every provider your environment carries a key for — named by the variable, so you can check your own .env.local — plus a local Ollama when one is answering, plus lexical-only. A row it cannot run here says so instead of building until it reaches a 401.

The question needs a terminal. npx --yes, CI and a Dockerfile never see it; they get one line naming the embedder and the build proceeds. To answer it up front:

npx docpilot doctor --embed prints the same list without asking, with the exact command that picks each row.

An override writes to a directory of its own by default, and that is not caution for its own sake. The index is bound to the embedder that built it. If your config names a model, embedderMatchesIndex in the panel compares the two, logs the mismatch and drops retrieval to lexical-only — loud, and degraded. If your config leaves the model to a free pool or to 'auto' there is no name to compare, the only remaining check is vector width, and two 1024-dimensional models pass it identically: the panel then scores queries against a foreign vector space with nothing anywhere reporting it. So an override prints the embed: block to paste, with the model spelled out, and leaves the index your site is serving alone.

It asks the provider which embedding models it serves, when the model was not one you wrote down — you named a provider and stopped, or you named neither. The configured name is tried first and the provider's own answers line up behind it:

  embedders 2 to try — custom offers 1, and BAAI/bge-m3 is configured
  warn  BAAI/bge-m3 is not answering (HTTP 404); trying the next embedder
  embedder  acme/gte-large-v2 · 1024d — chosen from 2 candidate(s)

One extra request at most, and none at all when you named the model or when a free pool already stands behind the provider. A name you wrote is used as given and a wrong one fails loudly rather than being replaced. See Asking the provider.

The vectors you already bought are not bought twice. A chunk's embedding is cached under a key made of the model, the provider, the host, the prefix that was applied and the chunk's own text — so editing one page costs one embedding request rather than one per 32 chunks of the whole corpus. The cache lives in ${evalDir}/embed-cache/, is gitignored by init, and is safe by construction rather than by expiry: a different model, provider or host is a different namespace, so a cache hit is always a vector from the space the index is being built in. --refresh-embeddings skips reading it — never writing it, so a refresh also repairs a file that went bad — for the one case a key cannot see, which is a provider that changed under a stable model name.

--quantize=auto|binary|int8 chooses how the chunk vectors are stored. binary keeps one sign bit per dimension — an eighth of int8 — and scores it against the question's full-precision vector; int8 is the one-byte format earlier builds wrote. auto, the default, measures the 1-bit error and takes the bit when it is at most 0.015 mean |Δcos|, which in practice means a wide embedder (4096 dimensions) gets the bit and a 1024-dimension one keeps the byte. Each format has its own build-time error gate, 0.03 and 0.01. See Vectors are quantised for what the bit costs and when to keep the byte.

--no-embed writes a real index with no vectors in it: no embedding calls, no vectors.<hash>.bin, and retrieval by BM25 alone. It is the one-off form of embed: false — the config key is what a deployment sets, because the browser has to be told as well.

The flag alone is not a deployable state, and this is the part to know before reaching for it: a vectorless index under a config that still names an embedder is a readiness FAILURE, not a warning. npx docpilot doctor reports it, and a site built in that state ships {enabled: false} — no panel at all. Use the flag to look at what the mode produces; set embed: false to run on it.

What that mode costs was measured on a 1191-chunk corpus: recall@8 fell from 0.97 to 0.41, retrieval F1 from 0.35 to 0.18, and 11 of 44 answerable questions were refused outright. The lexical channel also scores zero for a question asked in a language your corpus is not written in — there is no overlap to score. Measure your own corpus with eval --gate-only --lexical before you throw its vectors away — that needs a written ${evalDir}/golden.jsonl, so docpilot init and the gold answers come first.

It also inlines the calibrated guard. If ${evalDir}/calibration.json is missing, is for a different corpus, or was measured with a different embedding model, the build warns and inlines a provisional guard rather than failing — documentation must stay publishable when a threshold is stale.

And it inlines the tuned retrieval levers, which is the half of tune people miss: tuning.json reaches a reader only through this command. A missing one is silent, not warned — the shipped levers are measured values rather than placeholders, so a site that never runs tune is not misconfigured. A tuning.json that is present but stale is a warning, and the build falls back to those same shipped values.

--html-dir — a site that is already built ​

npx docpilot index --html-dir=dist
npx docpilot index --html-dir=public --html-select="article.doc"
npx docpilot index --html-dir=dist --html-select="article.doc" --html-exclude=".feedback"
npx docpilot index --html-dir=dist --sitemap=dist/sitemap.xml

Your markdown is walked as it always was, and every .html under --html-dir is walked too. That is the whole answer for a generator this package has no adapter for — Hugo, Jekyll, MkDocs, Astro, Next, a Blade or Twig template, a help centre somebody else runs. It has already produced HTML; that HTML is a corpus.

A built page is a page of your site. It is cited by its route, exactly like a markdown page: dist/guide/install.html is /guide/install, and an index.html collapses to its directory, which is the same rule docs/guide/install.md follows. No sources.allow entry is needed and no origin is written, because nothing left your machine — the difference from import, which fetches somebody else's page and must therefore say whose.

Markdown wins. An HTML file whose route a markdown page already claims is skipped, and the count is reported. Pointing this at the dist/ of the very docs site you are indexing is therefore a no-op rather than a duplicate corpus:

  pages            465
  built pages      0 (412 shadowed by markdown) from dist

Which part of each page is the documentation is decided in code, by the same extractor import uses: <main>, [role=main] or <article> when the page has one, otherwise the block holding the most prose. Navigation, footers, banners and share widgets are dropped; tab panels and accordions are kept, each panel under its label; code keeps its line breaks and its language; a table keeps its merged cells. --html-select overrides the choice of body with a CSS selector of your own — and a selector that matches nothing warns and skips the page rather than quietly falling back, because a typo in a flag should not index a different subtree than the one you asked for. --html-exclude removes one more selector from inside that body on every page: --html-exclude=".toc", or a CSS selector list such as ".toc, .feedback" when one rule is not enough.

The same two settings can be written once in the config, as html, and narrowed per directory there. The order is: the flags, then the longest matching html.sources key, then the project default, and content and exclude are each taken from the most specific level that sets them. A level that sets exclude replaces the lower list rather than adding to it.

One bad page does not stop the run. A page that cannot be read is skipped, warned about by name and counted, and the build goes on with the rest: a noindex page, one with fewer than 24 characters left after extraction, one over 8 MiB, one that will not parse, or one the selector does not match. index stops only when it attempted at least one page and extracted none — then the warnings above the error say why. To see what the extractor keeps and drops over a directory without building, doctor prints the same table import does.

The corpus hash carries the extractor. A corpus that holds --html-dir pages hashes the extractor's version in with them, so a release that reads a page differently is a changed corpus and the calibration behind it reads as stale, instead of a different text under an unchanged hash. A corpus of markdown alone has no extractor in it, and its hash does not move.

--sitemap reads a local sitemap.xml and indexes only the routes it declares. Nothing is fetched: the file is the list of pages your site publishes, which is exactly the filter that removes the 404 page, the draft a generator left behind, and the per-tag listing pages. --html-base=https://example.com only resolves relative links found inside the pages.

The HTML parser ships with the package, so there is nothing to install first, and it never reaches the browser bundle.

DOCPILOT_SPLIT_IDENTIFIERS — searching an identifier by its parts ​

DOCPILOT_SPLIT_IDENTIFIERS=1 npx docpilot index

docPilot.sources.allow is one token, so a reader who types sources allow matches nothing. getUserName is one token, so user name matches nothing. In technical documentation the identifier is the most common thing a reader searches for and the least likely thing they spell the way the code does.

With this set, a token that looks like an identifier — a separator inside it, or an internal capital — also contributes its parts, and keeps its whole form so an exact query loses no weight. Ordinary prose contributes nothing: documentation has neither a separator nor an internal capital.

It is off by default and it is an environment variable rather than a setting, on purpose. It moves every lexical score in your corpus, and whether it moves them the right way is a question only eval can answer, on your corpus. Sweep it:

npx docpilot eval                                   # the number you have
DOCPILOT_SPLIT_IDENTIFIERS=1 npx docpilot index
npx docpilot calibrate && npx docpilot eval         # the number it becomes

On this project's own 470-chunk corpus it takes the vocabulary from 4,710 types to 4,998 — +6.1% — and changes no chunk.

The flag is written into manifest.tokenizer, so the reader's browser tokenises the way the index was built; and it is folded into vocabHash, so a calibration measured without it is reported stale by the guard that already exists rather than used silently.

import ​

npx docpilot import https://example.com/product
npx docpilot import https://example.com/product --dry-run
npx docpilot import https://example.com/product --html page.html --out product

Turns an allowlisted external page into a page of the corpus, written to importDir with its provenance attached — see Imported pages.

Six steps, in this order:

1. The allowlist runs before the network. A URL outside sources.allow is refused without being fetched, and the line that would widen the boundary is printed rather than applied.
2. Fetch, identifying itself as docpilot-import and asking for --lang (default en), with a 30-second deadline. A page behind a bot wall or built by JavaScript will not fetch: render it yourself with html.fetcher, which the command calls instead of its own request, or open it in a browser, save the DOM, and pass --html <file> — or --html - to pipe it in. --html wins over the fetcher. The fetcher is called with the URL and an AbortSignal, and returns the HTML or { html, url, headers }; the URL you asked for and the url it returns both pass sources.allow, exactly as a redirect does, and x-robots-tag in headers is honoured.
3. Look for markdown the site already publishes, and use it if there is any — see below. Steps 4 and 5 are what happens when there is none. --no-alternate skips the search.
4. Extract, in code: headings, prose, lists, data tables, code samples and tooltips are kept; navigation, footers, cookie banners, share rails and anything whose meaning is a button are dropped. Every drop is named on stdout, and so is the subtree that was read. --html-select names the body yourself and --html-exclude removes selectors from inside it; both override what html in the config says for the source (flags, then the longest html.sources key, then the project default). The command then prints the extraction report. A page that cannot be extracted is skipped for one of these reasons, and because import handles one page, a skipped page is a command that imported nothing: it prints nothing to import: <reason> and exits 1.
5. Convert to markdown, with title, description, source, importedAt and extractor frontmatter — and lang and dateModified when the page declares them — and the attribution block. extractor is the version of the rules that wrote the page; index and doctor name a page an older one wrote, with the command that refreshes it. Relative links are made absolute against the page's <base href>, then its canonical address, then the URL you gave; a link that cannot be resolved, or that names javascript: or data:, is written as its text, and images are never carried over.
6. Annotate, with the model chat points at. This pass may add <llm-only> and <llm-exclude> and nothing else — the output is stripped of those tags and compared to the input character for character, and a model that reworded, reordered or dropped anything has its whole answer discarded. --no-annotate skips it.

The page's own markdown wins ​

If the site publishes page.md beside page, that file is the import. It is not a better conversion of the page — it is what the page was built from, so there is nothing to reconstruct and nothing to lose to a theme.

It is found two ways, because sites do both:

• declared — <link rel="alternate" type="text/markdown" href="…">, or any rel="alternate" pointing at a .md;
• derived — from <link rel="canonical"> when there is one, otherwise from the URL you asked for: /a/b → /a/b.md, then /a/b/index.md; a trailing slash tries index.md first. Canonical is what the guess is built from because that is the address without the tracking parameters and the trailing slash.

A derived URL is a guess and is treated as one. It goes through the same allowlist as the page — it is a URL the tool decided to request — and the response has to come back 200, non-empty and not HTML. A single-page app answers every unknown path with its own shell, so the status code alone proves nothing.

The frontmatter of the published file supplies title and description when it has them, including YAML block scalars (description: >-). Its # heading is dropped — the imported page has one already — and its relative links are made absolute against the source site, because they name routes that exist there and not here.

Whichever path ran, the report says so:

  read         https://vitepress.dev/guide/what-is-vitepress.md
               the page's OWN markdown — nothing was converted or dropped

  read         <main#app_main> — nothing outside it was read
               no .md at https://example.com/plugin/index.md (404)

The extraction report ​

Every run prints what the extractor kept and dropped, so the size of a loss is visible before the corpus changes. A silent extraction reads as "it took everything", and this table is how a selector that took too little or too much shows itself:

source                       pages  kept  dropped  empty  link failures
https://example.com/product      1  4210      812      0              0

pages is the pages extracted, kept and dropped count characters of the text — Unicode code points, whitespace collapsed, plain digits with no locale grouping, so two machines print the same table — empty counts pages skipped for having almost nothing left, and link failures counts links that could not be resolved and were written as text. Skipped pages are named under the row, the first ten of them, then a count of the rest:

source  pages   kept  dropped  empty  link failures
dist        2  14100     2760      1              2
  skipped /404: near-empty (11 characters)

A skipped page carries one reason: noindex (a robots meta tag, or an X-Robots-Tag header), near-empty (N characters), too large (N bytes), parse failed, invalid selector "…", content selector "…" matched nothing, or extraction failed. import exits 1 on any of them, because it had one page and imported none. The report is human-readable only; there is no JSON form.

The command does not index. An import changes the corpus hash, so what follows it is index --dry, index, lint, eval --gate-only and calibrate — the command prints that list when it writes.

vocabulary ​

npx docpilot vocabulary
npx docpilot vocabulary --dry
npx docpilot vocabulary --languages=ru,de --limit=30
npx docpilot vocabulary --replace

Reads every title and heading in docsDir, asks the configured chat model which words readers are likely to use for the terms it finds, and writes ${evalDir}/vocabulary.json. index inlines that file into the manifest, and vocabulary in the config file overrides it per term.

It proposes and never decides. The file is committed and edited like the golden set: which word a reader types is a judgement about your product, and a model's guess at it is a draft. A re-run merges — a term already in the file keeps what the file says — so a hand-edited list survives the next run.

It reads markdown, not the index, so it runs before there is one — which is why it sits at the front of the loop rather than inside it.

It changes what every lexical score means. The index hash is over chunk text and cannot see a tokenizer change, so the manifest carries a vocabHash beside it and index reports a stale guard when the two disagree. The command prints the sequence when it writes:

npx docpilot index && npx docpilot calibrate --refresh && npx docpilot index

A deployment with chat: false has no model to ask and the command says so rather than writing an empty file.

calibrate ​

npx docpilot calibrate
npx docpilot calibrate --sweep-only
npx docpilot calibrate --refresh
npx docpilot calibrate --limit=40

Measures the refusal thresholds against ${evalDir}/calibration.jsonl and writes ${evalDir}/calibration.json. A run that cannot satisfy its bounds fails and leaves the previous file untouched rather than shipping a regression.

--sweep-only re-runs the threshold search over ${evalDir}/calibration.raw.jsonl — the per-probe scores the last run cached — which makes trying a different selection rule free. A probe that is not in that file stops the run with the file named rather than being scored as a miss; run once without the flag to populate it. --refresh ignores the cache and re-embeds.

--limit=N is the short loop while you are authoring probes, and it is a head-slice of the file rather than a sample of it. A limited run still writes calibration.json, so the thresholds it measures are the ones the next index inlines — which is what makes it an authoring aid and not a measurement. Re-run it whole before you rebuild.

The embedder is not a flag on this command either: it comes from docPilot.embed through the same resolver index uses, so a calibration cannot be measured in a vector space the index was not built in. DOCPILOT_EMBED_URL, DOCPILOT_EMBED_PROVIDER and DOCPILOT_EMBED_KEY still override it, for a sweep against a second endpoint.

--transfer does not weaken that. It runs against the target index, embeds against the target's own endpoint, and writes a document stamped with the target's embedModel — what comes from elsewhere is the calibration file, and what crosses from it is tau in normalised space. The window is re-fitted here, on this embedder's cosines.

It refuses rather than resolving: a source measured on another corpus hash, another vocabulary, the other denseMode, a vectorless index, the same embedder as the target, a tau at or below wLexical, or an anchor draw below the n its own strata's bounds need. It refuses again if no window in the grid carries the inherited tau without over-refusing past what the source measured. A refused transfer leaves the previous calibration untouched, exactly as a failed run does.

The result is stamped source: "transferred-window" and carries overRefusalUB95 and gatePrecision as null — no full-set sweep produced them, and an anchor-scale figure in those two fields would ride into the manifest and every feedback record reading as the corpus's. transferCheck carries the anchor numbers with their own n instead.

There is no --no-embed here — that is index's flag, and this command reads its consequences off the manifest. On an index built without vectors it calibrates the lexical threshold only: tauLexical is swept and held to the same refusal floor, and tau — the hybrid threshold — is left null in calibration.json and reported as unmeasurable rather than filled in with a number no probe produced.

The null stops there. The next npx docpilot index inlines the measured tauLexical and puts the provisional 0.3 in the tau slot, stamped source: "calibrated-reduced-lexical" so the number is never read as measured — the retriever asserts a numeric tau at every init, and a null there is a panel that cannot open. Nothing on a vectorless index consults it: every verdict is G = L against tauLexical.

Run it again after a corpus change, an embedder change, or a prompt override.

lint ​

npx docpilot lint
npx docpilot lint --file=<path>
npx docpilot lint --json

Checks the golden set against the index it claims to measure. A gold_chunks entry naming a page that has since been renamed never matches, so the record reports a flat 0 that reads as a retrieval regression — this turns that into an error with the id named.

--json puts one object on stdout instead of the report: the counts, warnings, errors and ok. The exit code is the same one the prose form returns, and the sentence explaining a failure still goes to stderr.

Run it before every eval, and after every index. Run it after upgrading this package, too: the second and later parts of a split section are now #anchor~2, #anchor~3 where they used to be #anchor-2, #anchor-3, so every gold entry pinned at a continuation part matches nothing until it is repointed. -N now means only "the Nth heading with this title", which is what VitePress means by it. See Building the index.

feedback ​

npx docpilot feedback pull   --from ./export.jsonl
npx docpilot feedback report --from ./export.jsonl
npx docpilot feedback pull   --from https://example.com/feedback --since 2026-07-01
npx docpilot feedback report --from ./export.jsonl --out ./reports/votes.md

Turns the votes your readers cast into candidates for the eval sets.

Both spellings work — --from ./export.jsonl and --from=./export.jsonl — and --help prints the same table.

--out moves the one file each mode writes: ${evalDir}/candidates.jsonl for pull, ${evalDir}/reports/feedback-latest.md for report. It does not change what is written. A pull to a custom path still merges into whatever is already there, so pointing two runs at one file accumulates rather than replaces, and pointing one run at a fresh path starts a fresh review.

This package ships no database driver, and will not. The panel POSTs to an endpoint you run, into storage you chose; the way back out is yours too. --from ./export.jsonl reads anything one-JSON-object-per-line — a psql -c … > export.jsonl, a mongoexport, a Supabase CSV converted to JSONL, or what window.__docPilot.exportFeedback() prints in the console when you run no endpoint at all. A JSON array works as well; the shape is sniffed.

--from https://… is the convenience on top: if you already serve a receiver, add a GET beside it. It sends Authorization: Bearer $DOCPILOT_FEEDBACK_TOKEN when that variable is set, and follows a {items, cursor} pagination until the cursor runs out or --max-pages (default 50) — which it tells you about rather than reporting a total that looks complete. The token comes from the environment only, never from the config file, on the same rule provider keys already follow.

pull writes ${evalDir}/candidates.jsonl: one row per distinct question, with how often it was asked, the down-rate, a reason histogram, the sentences readers wrote, the gate's numbers across every ask — and a suggested stratum with the reasoning for it.

It writes nothing to calibration.jsonl or golden.jsonl, on purpose. A stratum is a judgement about what a question was for, and a feedback record only carries signals about what the gate did. Where the signals cannot separate two strata the row carries stratumOptions and no answer, because a single guess is false precision — and false precision in a file that is meant to be reviewed gets rubber-stamped. Every row also carries a target, since a good many candidates belong in the golden set or in a documentation backlog rather than in the calibration set at all:

A re-run merges: your edits to stratum, target and expect, and any row you marked promoted, survive it — counts and gate statistics are refreshed. A row you reviewed for a question nobody asked this time is kept rather than dropped.

report writes ${evalDir}/reports/feedback-latest.md: helpfulness, the reason histogram, refusal causes, the worst questions by down-rate at a minimum of three votes, and what readers wrote. It opens with what the sample cannot tell you — see below.

Read the bias note before you promote anything. Votes are not turns: a satisfied reader usually presses nothing, so this is a sample of people who felt strongly, filtered again by whichever verdicts feedback.send lets through. Under feedback.send: 'down' it is a purely negative sample, and calibrating tau against one moves the gate toward refusing every reader — the exact failure the stratified design in calibrate exists to prevent. That is why every candidate arrives as needsReview: true.

--level= ​

eval, bench emit and tune all take one, and all three read it out of the same module, so a bench and an eval at the same tier start from the same records.

A golden record declares the pool it enters at:

{"id": "q-01", "question": "How do I get started?", "level": "low", "expect": "answer", …}

Six tiers, smallest pool first:

low · medium · high · xhigh · max · ultra

They are cumulative. --level=medium runs low + medium. Every larger pool contains every smaller one, and that is the whole point: a regression in the smoke pool is a regression in the full set too, so the quick run and the full run can never disagree about which questions matter. --limit=N — still there, still a head-slice of whatever the author happened to write first — cannot give you that. Where both are passed, the level picks the population and --limit then takes the head of that tier.

Two defaults, and between them they land tiers on a golden file that already exists without moving a single number:

• a record with no level reads as high. high is defined as roughly the set that exists today, so a file written before tiers existed scores identically under --level=high and under no flag at all. Nobody has to backfill sixty records to keep their history.
• a run with no --level is ultra — everything, which is exactly what every run did before.

An unknown tier is refused rather than defaulted, by every command that takes the flag: a typo that fell through to ultra would print a pool nobody asked for and be read as the tier the author thought they ran. In the file, the rule is inverted — a record whose level is not one of the six falls into every pool, ultra included, because a typo may never delete a question from a run. Turning that into an error is lint's job: it warns on an absent level, errors on an unrecognised one with the id named, and prints the pool size each tier yields.

A bare flag is the same promise, and it is now kept. These parsers read --name=value and nothing else, so --level low left low as a stray positional and the flag read as absent — which means ultra. That was silent and destructive rather than merely wrong: the run scored the whole set, stamped meta.level: 'ultra', and — because ultra adds no segment — wrote itself over the full-set baseline report and then diffed itself against it, with the header line that names the pool suppressed for exactly that tier. Every value-taking flag on eval, bench and tune now refuses instead, naming itself and the form it wants:

  FAIL  --level takes a value: --level=low

The boolean flags — --gate-only, --lexical, --resume, --dry — are unaffected: for them the bare form is the form.

A filtered run is only ever compared against its own level. eval files a narrowed run apart as report-<hash>-<model>-lvl-<level>-<prompt>.json, and the report refuses to pair two runs across levels — --level=low scores a different population, not a smaller sample of the same one, so a delta between the two is arithmetic on unrelated numbers. ultra adds no segment at all, which is what keeps every unfiltered report on disk pairing with its successor. bench emit follows the same rule with a .<level> segment in its default task path, so a smoke emit cannot overwrite the full task file the last comparison was scored on.

latest.json is the last unfiltered run ​

Beside the report it writes under that name, eval copies every run to ${evalDir}/reports/latest.json — the fixed path anything outside this tool reads. A narrowed run is filed next to it as latest.<level>.json, on the same asymmetry as everything above: ultra adds no segment, so a path that is already hard-coded keeps pointing at exactly what it always did.

It is the one artefact levels did not partition, and it is the one most likely to be read by something that is not looking. Unpartitioned, npx docpilot eval followed by npx docpilot eval --level=low left that path holding a ten-question smoke score where the project's number was the full-set one over sixty — meta.level inside the document either way, and nobody reading it.

eval ​

npx docpilot eval
npx docpilot eval --gate-only
npx docpilot eval --gate-only --lexical
npx docpilot eval --models=qwen3:8b,phi4:14b
npx docpilot eval --level=medium --limit=5
npx docpilot eval --resume
npx docpilot eval --split=dev
npx docpilot eval --runtime=server
npx docpilot eval --endpoint=http://localhost:8787/ai/turn
npx docpilot eval --set=adversarial --runtime=server
DOCPILOT_BASE_URL=https://api.openai.com npx docpilot eval --provider=openai --models=gpt-4o-mini

Runs ${evalDir}/golden.jsonl and writes a report to ${evalDir}/reports/.

--level= scores one tier of the set — see --level= above. The tier is stamped into the report's metadata on every run, filtered or not, and a report written before tiers existed reads as ultra, so the history survives.

--gate-only measures retrieval and refusal without calling a model — fast, free, and the right loop while tuning. --lexical additionally disables the dense channel, which is how you measure what the embedder is actually worth on your corpus.

On an index built without vectors the flag is implied — there is no dense channel to disable — and the report's metadata says the run was lexical-only, so a number from it is never mistaken for a hybrid one.

Reports are named by index hash, model and prompt hash. Two reports built from different instructions are not comparable, and the tool says so instead of diffing them. docPilot.product, prompt.override and prompt.extend all move that hash, because all three change what is sent.

Flags ​

A deliberate ranking change is reviewed as the diff of the baseline file.

--provider= names the same three wire adapters the panel runs on, so an eval measures the transport your readers will actually get. It is not the fourteen provider names the config takes: every OpenAI-compatible service — OpenRouter, Groq, DeepSeek — is openai here, with DOCPILOT_BASE_URL pointing at it and the key in the environment. A key goes only to the host that owns it: DOCPILOT_API_KEY is sent as given; without it, eval sends the key of the provider whose own address DOCPILOT_BASE_URL is — OPENAI_API_KEY only to api.openai.com, OPENROUTER_API_KEY only to openrouter.ai, and so on for every provider. A LAN vLLM box, a local Ollama or another gateway gets no key unless you set DOCPILOT_API_KEY. Nothing is stored, and no report carries a credential. Spell it carefully: an unrecognised id falls back to the Ollama adapter rather than being refused, and the header line each model prints names the transport but not the adapter. The report records the string you typed as meta.provider, which is where a typo shows up — after the run.

The embedder is not this flag. It is resolved from docPilot.embed by the same resolver index uses, because the two halves are configured separately — Anthropic answers and cannot embed — and deriving one from the other embedded a hosted corpus's queries against a local Ollama, scoring them in a foreign vector space and reporting the result as this corpus's retrieval quality. DOCPILOT_EMBED_PROVIDER, DOCPILOT_EMBED_URL and DOCPILOT_EMBED_KEY override it; a reply of the wrong dimensionality stops the run. When either of the first two moves the embedder, the configured key stays behind: the run sends DOCPILOT_EMBED_KEY, or the key of the provider that owns the new host, or none. calibrate, tune and answer-bench use the same rule.

--fallback= decides whether the run uses native tool calling or the no-tools transport. auto asks each model what it supports — the server's capability list where there is one, a behavioural probe where there is not — which is the only way a model advertised as a tool caller and shipped without tool support is measured on the transport it will actually run on. on and off force it, which is how you measure the same model both ways.

--max-iterations= defaults to what the product ships, so an eval with no flags measures the configuration a reader gets rather than a laboratory one. It is written into the report's metadata.

--num-ctx= is pinned because Ollama's server default is 4096 tokens and a primed turn is routinely larger. Unpinned, each model in a matrix truncates at a different point and the run measures three different prompts. 8192 is the default; 16384 is the largest value every model in the reference matrix can honour. On openai and anthropic the value is not sent at all and the report records numCtx: null.

--resume is what makes a matrix restartable. For each model it looks for the report this run would write — index hash, model, -lexical, -novec, tier and prompt hash all have to match — and loads that instead of calling the model again; a model with no such report still runs. Because the name carries all six, --level=low will not read the full-set file, and a re-indexed corpus resumes nothing. The name does not carry the knobs, so --resume also compares the ones in the report's metadata — levers, seed, temperature, num_ctx, maxIterations and the A/B arm — and re-runs a model whose saved report used other values. It does nothing under --gate-only, which calls no model and has nothing to skip.

Every value-taking flag here refuses a bare form. --level low, --limit 5 and --num-ctx 4096 are all read as absent by a parser that matches --name= and nothing else, so the run would use the default and report it as fact — see --level=.

The split ​

A golden record may carry "split": "dev" or "split": "test". --split=dev scores only the first half and --split=test only the second; a record without the field is assigned by a hash of its id, the same way on every machine, so a set written before the field existed splits with no edit. Propose documentation edits from dev failures — that is what improve reads — and prove them on test: an edit written against the questions it is scored on measures how well it was fitted to them. The split is part of the report's name and metadata, so a dev run never diffs against a test one.

The server runtime ​

--runtime=server answers every record through the server turn — @cloflin/docpilot/server, created in-process over this project's index and settings and driven through its own HTTP handler, so the bytes scored are the bytes a panel would read. A conversation is replayed the way the panel holds one: each prior question is asked first and its answer travels back in history. The chat model is the one your config resolves, as the deployed server's is; --model= overrides it. The rows add what only the server measures: time to the first answer token, and where the time went (embed, retrieve, model). --endpoint= measures a server that is already running, over real HTTP; only what the wire carries is scored, so the retrieval columns are blank there. --gate-only has no server form — the server turn always ends in a model call — and is refused.

Security probes ​

--set=adversarial runs ${evalDir}/adversarial.jsonl instead of the golden set — prompt injection, attempts to extract the instructions, requests for keys, off-topic tasks, markup to echo back. Each probe names what must not appear in the reply (forbid, forbid_regex) and whether the right outcome is a refusal (expect: "refuse") or a safe answer ("answer-safely"). The report's security block gives the attack success rate — lower is better — beside refusal accuracy and the safe-answer rate, per kind. The reasoning is read as well as the answer, because the panel shows both.

bench ​

npx docpilot bench emit  --config=base
npx docpilot bench emit  --config=swept        # with the levers set in the environment
npx docpilot bench emit  --config=base --level=low
npx docpilot bench shard --tasks=base.tasks.jsonl --shards=10
npx docpilot bench score --tasks=a,b --answers=a,b
npx docpilot bench runs  --tasks=a,b --runs-a=a1,a2,a3 --runs-b=b1,b2,b3
npx docpilot bench judge-emit  --tasks=a,b --answers=a,b
npx docpilot bench judge-score --verdicts=v.jsonl --key=judge.key.jsonl

Compares two retrieval configurations on answer quality, with no API key. Retrieval, the gate and the primed observation run in-process from the production modules; the forced final call is handed to agents through a checked-in protocol.

What it measures is the difference between two configurations. What it does not measure — and must never be quoted as — is the agent loop, tool calls, iterations, latency or tokens per answer.

Artefacts go to ${evalDir}/bench/, which docpilot init gitignores: the filenames are stable across runs, so a fresh run overwrites the previous one's answers.

emit takes --level= — see --level=. It is the only mode that does: the tier chooses the population, and score and the judge modes then work on whatever task and answer files you hand them. A narrowed emit writes <config>.<level>.tasks.jsonl rather than <config>.tasks.jsonl, which is what stops a ten-task smoke emit from silently replacing the sixty-task file the last comparison was scored on. Within the pool, emit measures only records whose expect is answer, so its count reads "answer tasks within the tier", never "the tier".

Three runs, not one. A single-run difference between two configurations is usually inside their own run-to-run spread. runs is the mode that measures that spread, so the difference can be read against it.

The six modes ​

shard exists for the withholding. A shard line carries the id, the prompt and the citable id set and nothing else — gold_answer, identifiers and gold_chunks stay behind in the task file, because an answerer that can see the gold copies it and every metric downstream becomes a measurement of that copy. Shards are written as <taskfile-basename>.shardNN.jsonl.

judge-emit assigns sides by a hash of the record id, so neither config is systematically "left", and writes the key to a separate file the judge never reads. Each side is judged against its own excerpts, since the differing evidence is the thing under test. The judge is advisory and may never gate: a model inside the instrument measuring the model is not a measurement.

Flags ​

--history= is not optional in practice for a set with follow-ups. Without it a follow-up record's prompt is missing the turn it follows, and the record measures a different question than the one it names. It names one file and that file accumulates: each pass emits the priming turns the last one made answerable — <id>#prev1, <id>#prev2, oldest first — and their answers are appended to it before the next emit. A record with N prior questions therefore takes N + 1 passes, and how far it has got is read off the answers rather than off the flag, so a partly answered set advances the records it can and re-emits the turns it cannot.

A bare flag is refused. Every flag in that table is checked at start-up for the --name value form, and named with the shape it wants, because a parser matching --name= alone reads a bare one as absent: --level falls through to ultra, --out and --history fall through to a default path, and --tasks/--answers fall through to an empty list. On emit that combination writes sixty tasks over the file the last comparison was scored on.

tune ​

npx docpilot tune
npx docpilot tune --level=medium          # narrowed: report only, no tuning.json
npx docpilot tune --lambda=0.5:1.0:0.05
npx docpilot tune --k=4:12
npx docpilot tune --dry

Sweeps the two retrieval levers — MMR_LAMBDA × GATE_K — against ${evalDir}/golden.jsonl and writes the winning cell to ${evalDir}/tuning.json, with a report of the whole grid beside it. The whole set, or no tuning.json at all — see What it writes.

It needs an index to sweep against and a golden set to sweep with, so it belongs after index and after lint. Stage A calls the embed endpoint — resolved from your config by the same resolver index uses, and asked for the model the manifest was built with; a reply of the wrong dimensionality stops the run rather than scoring this corpus in a vector space it was never built in. A run where no positive record at the chosen tier carries gold_chunks inside its own scope fails outright rather than reporting a grid of nulls: there is nothing for retrieval F1 to be measured against, and lint is what verifies those ids still resolve.

Both of those numbers used to be literals in retriever.js, arrived at by a hand-run loop — export a DOCPILOT_* variable, re-run eval --gate-only, read four figures off the summary, repeat — performed by eyeball, once, against one corpus, and then shipped to every consumer's bundle. This is that loop with the eyeball removed, and tuning.json is where the answer gets to live instead of in a code comment.

What it sweeps, and what it leaves alone ​

GATE_K is the k that matters, because it is the k every retrieval metric in eval is measured at: the gate primes the turn with exactly that many excerpts, and retrieval F1 is scored on exactly that set.

A tuned GATE_K above 8 is legal but partial, and the default axis runs to 12 so you will meet one. It widens the excerpts that prime the turn, which is exactly what retrieval F1 is measured on, so the gain the grid reports is real. What it cannot widen is a search_docs call the model makes for itself: that k is clamped to 1..8, so the model can never reach a k of 10 on its own however the lever is set. Read a win above 8 as a win on the primed turn — worth having on a corpus whose answers routinely span several sections, worth less on one where the model searches its way to the answer over several tool calls. At 8 or below the two agree and the distinction does not arise. The report calls this out under the grid, and again on the chosen cell when the winner crosses it.

The k axis is capped at the resolved FUSED for a different reason: FUSED is the pool the re-rank picks from, so any k above it selects the whole pool and every larger k measures an identical cell.

What it costs ​

One embedding per record — plus one more per follow-up, for the composed query — and then a pure in-process grid. No chat model is contacted, there is no LLM judge, and nothing is re-embedded between cells, so roughly a hundred cells cost one embedding pass rather than a hundred.

That is possible because the gate is invariant under this grid. MMR_LAMBDA and GATE_K reach only the re-rank and the excerpt slice — which passages are handed over once a turn has already been admitted. The gate's own verdict is computed from the best cosine over the whole scope and the top lexical hits, neither of which either lever touches. So pure retrieval metrics are sufficient here, and no cell can buy F1 by refusing the questions it is bad at. That is a property of today's code rather than a law, so every cell measures the over-refusal count anyway and the report prints it as a sanity row — the day it stops being constant, the table says so and tells you not to inline the result.

The winner is argmax mean retrieval F1, then recall@8, then MRR, then proximity to the levers already in force. That last tie-break is not cosmetic: a tie decided by float noise would churn a committed file, a rebuilt index and a redeployed bundle for a difference of zero.

Flags ​

A bare lo pins an axis to one value, which is how you sweep one lever and hold the other. A malformed range throws and never falls back to the default: a silent fallback would run the sweep over a grid nobody asked for and write its winner to a file that gets inlined into a bundle, with the typo still on screen. A range yielding more than 200 points is refused as well. --limit is checked the same way and for the same reason — it decides whether this run may write tuning.json at all, so --limit=ten is refused rather than read as "no limit".

Every flag is checked before the index is loaded and long before stage A embeds anything: a run that is going to be refused has to be refused while it is still free.

An axis pinned in the environment refuses the run ​

docpilot tune will not start while DOCPILOT_MMR_LAMBDA or DOCPILOT_GATE_K is set — including from .env.local, which is loaded into the environment before the check runs.

The reason is the precedence below working correctly. Env outranks the tuning object, and the tuning object is precisely what this command varies per cell: a pinned axis makes all ninety-nine cells measure the identical retrieval, all three metrics tie everywhere, and the winner falls out of a tie-break rather than a measurement — a value nothing on the grid scored, on its way into a committed file and from there into every reader's bundle. That is a shell variable shipping, against the rule that env never ships.

The variable is named and the run stops, rather than being unset for you: the environment is yours, and this process shares it with whatever launched it. The remedy is unset, or pinning the axis where a pin is honest — a bare lo on the grid:

unset DOCPILOT_GATE_K
npx docpilot tune --k=5          # the same pin, on the axis, in the report

The other six levers are reported rather than refused. A DOCPILOT_FUSED=20 widens the pool the sweep selects from, which is a real thing to want to measure — but tuning.json records only λ and GATE_K, so the answer was measured under a pool the file does not mention, and the run says so.

What it writes ​

${evalDir}/tuning.report.md, always — --dry exists to produce exactly that file, and a sweep you cannot read is a sweep you have to run again. It carries the chosen cell against the baseline, the full λ × k grid, the gate-invariance sanity row, and the ten records that moved most in each direction, which is where a headline of +0.02 mean F1 turns out to be one record going 0 → 1 and nine going nowhere.

A narrowed run files its report apart, and only a narrowed one gains a segment — a full run keeps the name it has always had, so no report on disk is orphaned:

${evalDir}/tuning.json, from a full-pool run only, and not under --dry. It carries the two measured levers and nothing else — the other six lever names are deliberately absent rather than written at their current values, because a key in that file is a claim that the number was measured on this corpus, and freezing six unmeasured constants into a consumer's manifest would make a later change to the shipped defaults invisible.

So --level and --limit make the run report-only, and that is a different decision from the suffix above rather than an inconsistency with it. The two artefacts are different kinds of thing. A report is reading material: a narrowed one is useful, it just must not overwrite the full-set one, and a suffix is exactly enough. tuning.json has one purpose — to be read back by docpilot index from one fixed path and inlined into every reader's bundle. So a tuning-lvl-low.json has only two possible fates, and both are worse than not writing it: nothing ever reads it, or something does and the smoke-pool answer ships anyway. Writing it to the fixed path is worse still, and is the defect this rule exists for: ten records replacing levers that took the whole golden file to earn, waved through by every check the build makes, because the version, the index hash and the embedding model all still match.

The run says which it did, on the way in and on the way out:

  narrowed pool — REPORT ONLY, docpilot/tuning.json will not be written

Re-run with no --level and no --limit to write it.

It is written even when the sweep picks the levers already in force. That is the point rather than an oversight: an unwritten winner survives only as a literal in the package, and the next release that moves that literal would move your corpus without anybody deciding to. Pinning it is the difference between "measured and unchanged" and "never measured".

The step everyone forgets ​

tuning.json does nothing until the next npx docpilot index. tune writes the file and stops. It is the build that reads it back, validates it, and inlines the levers into the manifest, from where they reach every retrieval in the browser and in the eval. Until then, the file is a measurement nobody is running on.

The build drops it — loudly, and back to the shipped defaults — when it is a version this build does not read, when it names a different index hash, or when it was measured with a different embedding model. That last one has no other way of being caught: the index hash is taken over chunk text, so swapping the embedder leaves it identical, and a λ measured in one embedder's cosine space describes nothing about where another one puts its cosines.

On an index built without vectors there are no cosines at all, so what is really being swept is BM25 order plus GATE_K. The run still means something, the report says so on every page, and embedModel is written as null — which fails against every embedder while still matching the vectorless build it came from.

What it may never touch ​

tau, tauLexical, wDense and wLexical are the guard's, set by calibrate and by nothing else. Levers are tune's; thresholds are calibrate's. Should one of those four find its way into tuning.json — by hand, by a merge, by a future writer being helpful — the build names it and drops it rather than letting it ride into the manifest the guard also rides in.

The build's allowlist is narrower than that, and narrower than "a lever". Only MMR_LAMBDA and GATE_K may travel through tuning.json into a bundle: the two tune measures, and no others. The remaining six — RRF_K, W_LEXICAL_RRF, W_DENSE_RRF, CANDIDATES, FUSED, EXPAND_BELOW_TOKENS — are dropped with a line saying they were never measured on this corpus, even though every one of them resolves fine at runtime.

That is the point rather than an oversight. tuning.json is a file you commit and may hand-edit, and the gate reads its lexical evidence off the top three of a candidate list CANDIDATES sizes — so a hand-written CANDIDATES: 1 starves that evidence and turns an answerable question into a refusal, with no threshold named, no model called and nothing printed. A refusal verdict is calibrate's to move; anything that can move one hits a wall here. Set the six for an exploratory run in the environment, where they belong and where they cannot ship.

One thing does outrank a tuned lever: a topK you set yourself is GATE_K under its documented name, and an author's number beats a measured one. The full order is a DOCPILOT_* variable, then your config's topK, then manifest.tuning, then the package default — with one implementation of that rule, so the report can never name a lever the retrieval did not run on. A DOCPILOT_* that is not a number does not count as set: it leaves the lever where it was rather than pinning the corpus to garbage.

That first layer is read when a lever is resolved, not when the module loads — which is what makes .env.local work at all. Every CLI entry point loads that file into the environment after its imports, so a layer answering out of import-time constants would be reading a snapshot taken before the file existed: DOCPILOT_GATE_K=9 in .env.local counted as set, resolved to the package literal, and discarded both its own value and manifest.tuning on the way past. All eight lever variables now take effect from .env.local, including the six that were silently dropped on that path. In the browser there is no environment at all — globalThis.process is undefined — so the rule collapses to manifest.tuning then the package default, and nothing set in a shell can ship.

doctor ​

npx docpilot doctor
npx docpilot doctor --json
npx docpilot doctor --proxy
npx docpilot doctor --embed
npx docpilot doctor --models

Prints what was resolved and either confirms the panel will render or lists what is missing, each with the command or variable that fixes it. Exits non-zero when not ready — the build never fails for these, so this is the opt-in place to gate CI.

It also prints the provider chain in full, with the member that answered marked — unconditionally, whether the provider was named in your config or chosen by the environment:

[docpilot] chain     auto → openai
                     ✓ openai      OPENAI_API_KEY         ←
                     · gemini      GEMINI_API_KEY
                     · mistral     MISTRAL_API_KEY
                     …
                     ✓ ollama      no key needed

The build log stays quiet about this when a provider is named, because a line restating your config file is noise in a block people read at every start. doctor is the opposite: it is run precisely when the question is why is it talking to that, and which variables are set is not visible anywhere else. Only the name of a variable is ever printed, never its value.

It runs without a config file at all, on the shipped defaults and your environment. That is the zero-config install, and a command that exited there could not check it.

--json puts one object on stdout and nothing else — the same facts the rows carry, plus the installed version, and ready as a boolean. The diagnosis still goes to stderr, so npx docpilot doctor --json | jq .ready gives the script its answer while the operator still sees the report. The exit code is unchanged: ready and the code are one verdict, and two ways of reading one answer is one way for them to disagree. No key value appears in the object, the same rule the prose obeys.

{
  "version": "1.1.0",
  "configPath": "docs/.vitepress/config.mjs",
  "indexDir": "docs/public/rag",
  "chat": { "provider": "openrouter", "model": null, "providerAuto": false, "chain": ["openrouter"] },
  "embed": { "provider": "openrouter", "borrowed": false },
  "ready": true,
  "missing": [],
  "notes": ["…"]
}

When docPilot.html.sources names a directory of built pages, doctor also re-runs the HTML extraction over it, offline, with the options that directory's key gives it, and prints the extraction report — one row per directory, skipped pages named — under an html heading. A key written as a URL is never walked, and a directory that does not exist is named in one line. When importDir holds pages that an older extractor wrote, an imports row lists every one, each with the command that re-imports it (index names at most five; doctor names them all):

[docpilot] html      extraction, re-run offline over docPilot.html.sources
                     source  pages  kept  dropped  empty  link failures
                     dist        2  14100     2760      1              2
                       skipped /404: near-empty (11 characters)
[docpilot] imports   2 written by an older extractor (this build's is 2)
                     npx docpilot import https://example.com/a --out a --force
                     npx docpilot import https://example.com/b --out b --force

Both rows are information. They make no network call, they are left out of --json, and neither moves the exit code, which stays the readiness verdict.

--proxy additionally prints the contract a production reverse proxy has to satisfy: the exact paths, the upstream for each, and the name of the header the key goes in. The key value is never printed. See Production for what to do with it.

--embed lists every embedder this project could build with — what your config names, what your environment has a key for, what is running locally, and lexical-only — each with the source of the row and the exact npx docpilot index … line that picks it. It is the non-interactive half of the question index asks, and it is what an agent driving this package should read before proposing a rebuild. It touches the network only to ask the Ollama on this machine whether it is running; no hosted provider is contacted and no metered request is spent.

--models also asks whether the chat-only claim in the provider table still holds. anthropic, deepseek, groq, xai and cerebras are recorded as serving no embeddings endpoint, which is a claim rather than a law — the same table said that of OpenRouter for months after it stopped being true, and the cost of it going stale is a second key and the text of your whole corpus posted to a third party at build time. So the endpoint is knocked on, with a candidate from the provider's own catalogue, and the answer is printed when it has changed:

[docpilot] embed?    groq answers /v1/embeddings after all — nomic-embed-text-v1.5
                     embed: {provider: 'groq'} drops the borrowed openrouter key

Silent otherwise, which is the expected case. It reports and never acts: the reverse proxy carrying /ai/v1/embeddings is written from your config at build time, so a build that moved itself would send every reader's query vector to the wrong upstream. Two candidates at most, so this cannot become a survey of somebody's catalogue. anthropic is skipped without a request — its API has no embeddings path to knock on.

--models asks the provider's live free catalogue whether the model list in force is still being served — the chat half and the embed half separately, and for each it is the free pool this package ships when the model is left unnamed, or your own chat.models where you wrote one. It prints the pool size against the catalogue size, then RETIRED: for every id upstream no longer lists and up to six that are new upstream. This is the one question a baked list cannot answer for itself, and free ids are retired weekly: a pool whose members have all been retired fails in the least legible way available, 404ing model by model until the reader is told the last one's name.

It is a flag rather than part of the default run because it is the only thing in doctor that touches the network. doctor is the command CI gates on, and a check that fails when a third party is slow is a check that gets removed — which takes the readiness gate with it.

It checks only a half that has a pool and a catalogue to check it against. On a provider that publishes neither, the line reads no catalogue to check them against rather than reporting every id as retired, and an unreachable catalogue is printed as unreachable rather than turned into a verdict. The exit code stays the readiness one either way — a retired id is news, not a broken configuration. See budget.

status ​

npx docpilot status
npx docpilot status --json
npx docpilot status --url=http://localhost:8787/ai/turn

One picture of the project, and the one command to run next. It reads the index, the calibration, the eval sets and the reports on disk, and it re-chunks the docs to tell a fresh index from a stale one, which on a documentation site takes well under a second. No network unless --url names a server, or the project runs runtime: 'server', whose endpoint it reaches on http://localhost:8787, where serve puts it.

An illustration — the figures are an example project's, not this site's:

  index       ✔ fresh  1a2b3c4d · 412 chunks · 38 pages
              bge-m3 · 1024d · int8 · 1.9 MB on disk · ≈ 1.1 MB brotli
  guard       ● provisional  tau 0.3 · tauLexical 0.3 · source provisional
              mode off — scored on every turn, refuses nothing
  …
  next        ❯ npx docpilot eval --gate-only
                nothing has measured index 1a2b3c4d yet — a gate-only run calls no model, costs nothing and takes seconds

Fresh is the hash the build would write. The index version is sha256 over every chunk's id and text, with no embedding in it, so the docs are chunked here exactly as index chunks them and the two hashes compared. Three sources are not re-read — pages under importDir, OpenAPI specs, and a site built with --html-dir — and an index that holds any of them is reported unverified rather than stale, because rebuilding on a false "stale" could drop the pages this check cannot see.

The delta and the trend use two rules on purpose. A delta is against the run eval itself compared this one with — same index, same model, same tier, same channels, same embedder — so an arrow here is the arrow the report recorded. The trend drops the index: a history that restarted at every rebuild would never be longer than two points. A metric moving the good way is green, whichever way that is: up for recall, down for latency.

The next rules, first match wins:

--json prints one object on stdout and nothing else, {"kind": "docpilot.status", "v": 1, …}, with exactly these keys: project, index, guard, evalSets, lastEval, server, next. next.rule is the name in the table, so a script can branch on it without reading a sentence, and next.argv is the command to run. It is the one object an agent driving this package should read first.

{
  "kind": "docpilot.status",
  "v": 1,
  "project": { "config": "docs/.vitepress/config.mjs", "runtime": "browser", "docsDir": "docs" },
  "index": { "exists": true, "hash": "1a2b3c4d", "fresh": true, "chunks": 412, "quant": "int8" },
  "guard": { "mode": "off", "verdict": "provisional", "tau": 0.3, "source": "provisional" },
  "evalSets": { "golden": { "records": 40, "byExpect": { "answer": 30 }, "byLevel": { "low": 8 } } },
  "lastEval": { "model": "qwen3:8b", "level": "ultra", "metrics": [{ "key": "recall8", "value": 0.9, "delta": 0.025 }] },
  "server": { "configured": false },
  "next": { "rule": "no-report", "argv": ["eval", "--gate-only"], "command": "npx docpilot eval --gate-only" }
}

(Abridged: every block carries more fields than shown.)

It always exits 0 when it ran. A stale index is a finding, not a failure of this command; doctor is the gate a CI job exits on.

The home screen ​

npx docpilot with no command, when stdin and stdout are both a terminal and CI is unset, prints the picture above and a menu of the commands that make sense now — the next rule's first, so Enter runs it. ↑ and ↓ (or j and k) move, a digit picks, Esc or q backs out, and Ctrl-C or Ctrl-D exits 130. Each entry runs the real command in your terminal, with its own questions and its own exit code. Anything else — a pipe, CI, --help — prints the help, exactly as it always did.

init ​

npx docpilot init

In a terminal, and only when nothing has already answered them, it asks up to five questions — which agent tool gets the skills, whether they go in this repository or in your home directory, which install (the turn in the reader's browser — the static install — or on your server), where the button goes, and what shape the panel is. It prints the config block for the last three, and for the server install the commands that run it. It never edits your config: the block is yours to paste.

The trigger question offers the four words; ui.trigger also takes an array written out, which is something you put in your own config rather than answer at a prompt.

Any flag, a missing TTY, or --yes skips the questions — which is what makes npx --yes, CI and a Dockerfile work. The placement questions are also skipped in a directory with no VitePress config, because they are about a file that does not exist yet; the install questions are not, because a bare directory is exactly where somebody wants to be asked which tool they use. The two placement values are validated by the same resolver the build and the browser run, so an unusable value is reported once, in the same words, wherever it was typed. See ui for what the two settings mean.

Where the skills can go ​

Agent Skills is an open standard — a directory per skill, a SKILL.md inside it — and four tools read the same shape from four different places. --target names the tool; --scope names which of its two directories.

agents is the vendor-neutral spelling; Cursor and Copilot both read it, Claude Code does not. --skills-dir= is the escape hatch for a tool this table has never heard of, and it writes no slash commands unless --commands-dir= says where they go — the whole point of the flag is that we do not know what is reading it.

A slash command per command ​

Every command on this page is also generated as /docpilot-<command> — /docpilot-index, /docpilot-eval, /docpilot-update, all eleven. Each one carries that command's own flag list, rendered from the same table this reference is checked against, so a flag cannot exist in one and not the other.

They are generated, never authored, which is why npx docpilot update rewrites them all: a command file left at the previous release describes flags the CLI has stopped taking, and an agent runs what it reads.

Scaffolds the whole loop, and never overwrites:

The two starter golden records that expect an answer enter at low and medium, and the negative — the one that must be refused — enters at low, so the smallest pool cannot be passed by answering everything. See --level=.

<skills-dir> and <commands-dir> are whatever --target and --scope resolved to — .claude/skills/ and .claude/commands/ by default. The skills are copied rather than left in the package because no tool discovers a .claude/, .codex/, .cursor/ or .github/ directory inside node_modules — copying is the only way they reach anyone. See Skills.

.gitignore is the one exception to never overwrites, and it is an append rather than a write: your project almost certainly has one, and skipping it is how the ignore rule ended up documented and never implemented. The index is megabytes of quantised vectors rewritten whole by every npx docpilot index, so a repository that commits it grows by that much per rebuild. Delete the line if you would rather commit it — a deploy that ships the index makes no API requests of its own, which is what this repository does. Running init twice adds the entry once.

Every file is reported as written or kept, so running it twice is safe and running it in an existing project is honest.

Upgrading, and what init will not do ​

The rule that makes init safe to re-run is the rule that makes it useless for an upgrade. It writes a file only where nothing is, and that check never looks at what the existing file says — so a project that ran init once keeps its copy of the skills across every package upgrade that rewrites them. Re-running init prints kept … (already there) for each one and moves on.

That is still true, and it is still right for golden.jsonl and calibration.jsonl: those are your work, and quietly replacing either would be far worse than a stale skill. The skills are not your work — they are documentation this package ships — so they got a command of their own. See update.

update ​

npx docpilot update

Refreshes the skills and the /docpilot-* slash commands that init copied, from the package that is installed now. It is what you run after npm install @cloflin/docpilot@latest, not instead of it.

With no flags it finds the installs rather than being told about them: it looks in every directory in the target table, in both scopes, and refreshes each one that already holds a DocPilot skill. That is what makes npx docpilot update in one project refresh a skill you installed into ~/.claude/skills/ from another. Nothing is created by discovery — a machine with no Cursor does not grow a .cursor/ because you ran this. Only an explicit --target= installs somewhere new.

What happens to a file you edited ​

Beside every installed skill sits .docpilot.json, holding the release that wrote it and a sha256 per file. That turns "you edited this" from a guess into a fact, and the two cases are treated differently:

[docpilot] REPLACED .claude/skills/docs-rag/judge-protocol.md   (your copy kept as judge-protocol.md.bak)

  1 file you had edited was replaced. Your version is beside it:

    .claude/skills/docs-rag/judge-protocol.md.bak

  Diff them before deleting them — a local edit is usually a local reason.

A second update writes .bak.2 rather than overwriting the first backup. A file the package has stopped shipping is deleted when it is still exactly as we left it, and left alone with a left row when it is not.

The manifest records null — unknown provenance — for every file init found already in place and kept. That is deliberate: recording the package's own hash for a file we did not write would let a later update decide your copy was ours and replace it with no backup.

An install made before manifests existed is still found, by the skill directory alone, and every file in it is treated as unknown provenance. So the first update after upgrading past this release backs up everything it replaces, which is the conservative half of the trade.

--check is the CI form: same report, no writes, exit 1 when anything would change. It is how a repository holds itself to shipping the skills its installed DocPilot actually carries.

serve ​

npx docpilot serve --log=docpilot/turns
npx docpilot serve --dir=docs/.vitepress/dist --port=8787 --log=docpilot/turns --retention-days=30

Runs the server runtime: the turn — openers, embedding, the gate, the tool loop and the prompt — on your machine instead of the reader's browser. Needs runtime: 'server' in the settings and refuses to start without it.

It reads the server.* operator settings — limits, deadlines, caches, the metrics route — see Server operations. It is still a convenience, not a production server: no TLS and no compression. Put it behind the proxy that already serves your site and route server.endpoint to it.

monitor ​

npx docpilot monitor
npx docpilot monitor --url=https://docs.example.com/ai/turn --token=$DOCPILOT_METRICS_TOKEN
npx docpilot monitor --once
npx docpilot monitor --interval=10s >> metrics.jsonl

A live view of a running server runtime. It polls GET {endpoint}/metrics?format=json and redraws one block in place: turns per minute with a sparkline of the last thirty minutes, latency p50/p95/p99, first-token time, the time each stage took, a bar per outcome, errors and rate-limited turns, tokens and the share of the prompt served from the provider's cache, and the model and index the server is running. q quits.

Off a terminal it prints one JSON line per poll, the snapshot exactly as the server sent it, so the same command feeds a log. A poll that fails says why on stderr, and the next one goes on.

The snapshot is a {"kind": "docpilot.metrics", "v": 1, …} object: turns (with byOutcome), latencyMs and ttftMs percentiles, stagesMs, requests, tokens, cache, limits, and perMinute, the last thirty one-minute buckets, oldest first. Any field may be missing or null — a server that has answered no turn yet has no p95 — and each one is drawn as —. A server with no /metrics route — server.metrics: false, or a version from before the route existed — answers 404, and monitor says metrics unavailable rather than failing; one that wants a token answers 401, and off loopback without one 403, and for both it says which variable to set.

traffic ​

npx docpilot traffic report --from=docpilot/turns --feedback=./votes.jsonl
npx docpilot traffic candidates --from=docpilot/turns --since=2026-10-01 --limit=20

Reads the turn log the server runtime writes. report finds content gaps, vocabulary misses, pages retrieved and never cited, pages never retrieved, failure by language, and outcomes per prompt and index hash — each with the command that acts on it. candidates writes ${evalDir}/traffic-candidates.jsonl: failed real questions in the golden record's shape, needsReview: true, capped per cluster, never a gold_answer, never into golden.jsonl. See Improving answers from traffic.

A malformed line is counted and named, never dropped in silence.

dataset ​

npx docpilot dataset kto --from=docpilot/turns --feedback=./votes.jsonl
npx docpilot dataset sft --from=docpilot/turns --feedback=./votes.jsonl --include-unvoted --holdout=0.1

Training sets from your readers' turns: sft (chat messages), kto (TRL prompt/completion/label — one thumb, one row) and dpo (TRL prompt/chosen/rejected, where one question has both a liked and a disliked answer). See Fine-tuning.

llms ​

npx docpilot llms lint --site=https://docs.example.com/
npx docpilot llms lint --file=https://docs.example.com/llms.txt --json
npx docpilot llms build --site=https://docs.example.com/ --md

lint checks an llms.txt against the llmstxt.org format and against your index: one H1 first, a blockquote summary, H2 sections of - [name](url): notes, no duplicate or relative links — and which indexed pages it never links, and which links point at no indexed page. Exits 1 on an error.

build writes one from the manifest: pages grouped by their sidebar group, notes from each page's description, the changelog under ## Optional. On VitePress prefer vitepress-plugin-llms (llms.txt); build says so when it finds it and writes anyway.

improve ​

npx docpilot improve
npx docpilot improve plan --report=docpilot/reports/report-….json
npx docpilot improve compare --before=before.json --after=after.json
npx docpilot improve check --since=HEAD

The improvement loop's deterministic half: it reads what eval wrote, never calls a model and never edits a page.

plan (the default) diagnoses every failing record of a report — the newest dev-split report unless --report= names one — and names ONE cause for each, in the order a turn spends its evidence: a vocabulary gap (the gold is not in the ranked eight and most of the question's words never occur in the gold text), ranking (the words are there, another page wins), not primed (in the eight, not in the five the model reads), structure (the answer is split across chunks the turn did not all see), generation, false refusal or an answered negative. Failures are grouped into one item per cause and page — five questions that miss a page for want of one word are one edit — with the file to change and the records that will prove it, written to ${evalDir}/improve/plan.md and plan.json. The levers it proposes are the page's description in llms.txt and its frontmatter, vocabulary.json aliases, a heading that names what the section answers, an <llm-only> summary, and docpilot tune. Generation, refusals and answered negatives are marked investigate: they are the prompt's, the model's or the guard's, not the corpus's. It never proposes product behaviour the docs do not already state.

compare pairs two reports of the same records by id — before and after an edit — and gives the verdict the skill has always stated: revert when any metric falls more than two points, keep when something rose beyond noise (a paired bootstrap interval clear of zero, or McNemar on the per-record outcome), neutral otherwise. It does not refuse a changed index hash, because a corpus edit changes it by definition; what must match is the population — the same golden set, level and split.

check guards the edit against the two ways a loop cheats: text added to the docs since --since= (default HEAD) that copies a run of eight words from a test-split gold answer, and any change to golden.jsonl, calibration.jsonl, chains.jsonl or adversarial.jsonl. It exits 1 on either.

model ​

npx docpilot model fingerprint ./checkpoints/step-2000 --out=cand-manifest.json
npx docpilot model verify --manifest=cand-manifest.json
npx docpilot model compare --base=base.json --candidate=cand.json --candidate-manifest=cand-manifest.json
npx docpilot model contamination --train=train.jsonl

For a model you train. Nothing here loads weights or trains anything — see Fine-tuning.

• fingerprint hashes every weight, adapter, config and tokenizer file (sha256, streamed) and reads the safetensors and GGUF headers without loading tensors: tensor count, dtypes, parameter count, general.* metadata, a LoRA adapter's base model.
• verify re-hashes against a manifest and exits 1 on any missing, changed or extra file.
• compare reads two docpilot eval reports — the base and the candidate, each run against an endpoint serving those weights — pairs them record by record and tests the difference: McNemar exact on pass/fail, a paired bootstrap on answer F1, citation precision and recall, recall@8, latency and tokens. It refuses when the index, prompt or golden set differ, and exits 1 on a regression.
• contamination finds golden records whose 13-grams appear in a training set — an eval that measures memory. Exits 1 on any.

Help, and the version ​

npx docpilot --help            # every command, and what the loop is
npx docpilot <command> --help  # that command's flags, what each does, what it costs
npx docpilot help <command>    # the same thing, spelled the other way
npx docpilot --version         # the installed version, and nothing else
npx docpilot                   # in a terminal, the home screen; piped, the help

The help groups the commands the way a project goes through them — Set up, Build, Measure, Improve, Operate, Train — and on a terminal it is coloured. The words are the same either way; NO_COLOR switches the colour off and FORCE_COLOR keeps it through a pipe, here and in every command.

Help costs nothing and needs nothing: no config file, no key, no network, no built index. That is deliberate — it used to run the command, and on index, eval, calibrate and vocabulary that made --help a purchase order. Every command's flag list is rendered from the same table that validates the flags, so a flag that exists is a flag the help names, including its short spelling where it has one (--yes, -y).

The environment, and who wins ​

.env and .env.local are read once, by the launcher, the way the VitePress build reads them — and the existing environment wins. A variable already set in your shell or by CI is never overwritten by the file, so a one-off OPENROUTER_API_KEY=… npx docpilot eval beats a checked-in .env, which is the case the rule exists for.

Every command sees the result, including bench and lint, which used to read the file not at all while this CLI's own help told you to put your key in it. A project without VitePress installed simply has no file to read, and the shell stands alone.

Exit codes ​

Every command returns one of four, and a script can act on the difference.

Two of these are new in 1.1.0 and both used to be 0 or 1: cancelling a prompt returned 0, and a typed flag returned the same 1 as an outage. There is no code for "succeeded with warnings" — a warning goes to stderr and the command still returns 0.

Diagnostics go to stderr, the product goes to stdout. That includes the progress counters, which redraw in place only when stderr is a terminal — there index, eval, calibrate and tune draw a bar with a count and an ETA — and piped to a file they print whole lines instead of one line holding every value the counter ever had. docpilot doctor --json | jq .ready therefore works with the diagnosis still visible on the terminal.

Errors are one line, prefixed [docpilot] . Set DOCPILOT_DEBUG=1 to get the stack trace behind one.