Measure ​

npx docpilot eval scores your golden set and prints one number per question you can ask about an answer: was the right page found, was it ranked high, did the answer say the right thing, did it cite the right passage, did it refuse what it should refuse. This page explains each number and when it misleads you. The commands and flags are in the CLI reference.

No metric here uses a model as a judge. A number that needs a model to compute moves when the judge moves, and so it cannot be a gate. Every metric is a function of the report's rows, so two runs differ only when the system differed.

question ─► retrieve ─► prime ─► model ─► answer + citations
              │           │                    │
         recall, MRR,  context            answer F1, citations,
         nDCG          precision@5        negatives caught, attacks
              └──────────── latency, first token ────────────┘

Before you read a number ​

• Positives and negatives. A positive is a question your docs answer. A negative is one they do not. Retrieval and answer metrics average over the positives. Negatives caught and over-refusal read the negatives and the positives.
• --gate-only has no answer metrics. It calls no model, so it reports only the retrieval metrics. It is free and takes seconds.
• Gold is a list of pages or sections. An entry such as guide/auth matches that page and the pages nested under it. An entry such as guide/auth# matches any section of that page. An entry such as guide/auth#request matches that section. A metric that credits a page pin for any of its sections is more generous than a metric pinned to one section.
• Dev and test. Choose edits on the dev split. Prove them on the test split. See Improve answers.
• The set is small. A golden set of a few dozen records moves by whole records. Read a change in a table, not in the third decimal place. docpilot improve compare applies a paired test for this reason.

The metrics ​

recall@k ​

What it counts. For one record, the share of its gold entries that appear in the top k retrieved chunks. The report holds recall@1, @3, @5 and @8, averaged over the positives. recall@8 is the headline.

When it misleads.

• It measures retrieval only. A record can score 1.0 here and still get a wrong answer.
• k = 8 is the ranked list, not what the model reads. The model is primed with the top five. A gold chunk at rank 6 to 8 counts for recall@8 and never reaches the model. The failure taxonomy calls this gold-below-primed.
• Several gold entries make the score fractional. Two gold pages with one found is 0.5, not a miss.

What moves it. The page text, the headings, vocabulary.json, the page description in llms.txt, and the embedder.

MRR ​

What it counts. One over the rank of the first gold chunk: 1 at rank 1, 0.5 at rank 2, 0.33 at rank 3, and 0 when no gold chunk is retrieved. The report holds the mean over the positives.

When it misleads. It reads the first hit only. With two gold pages, MRR cannot tell whether the second was found. It also falls fast: moving a hit from rank 1 to rank 2 costs 0.5 on that record.

nDCG@8 ​

What it counts. Normalised discounted cumulative gain with binary relevance, at k = 8. Each gold entry earns its gain once, at the first rank that covers it, discounted by 1 / log2(rank + 1). The result is divided by the best possible list, so 1.0 means every gold entry is at the top.

Why it is here. It is the headline number of retrieval benchmarks, so a DocPilot report can be compared with other systems' numbers.

When it misleads.

• Published numbers are usually nDCG@10. This is nDCG@8, so compare the cutoff before you compare the value.
• A page pin covered by three sections earns one gain, not three. A page that fills the list does not score above a list that found every pinned section once.

context precision@5 ​

What it counts. The RAGAS definition over the top five: the mean of precision@i at each position i that holds a relevant chunk. A relevant chunk at rank 1 is worth more than the same chunk at rank 5, and noise above a hit lowers the score. It is zero when nothing in the top five is relevant.

Why it is here. It is the metric that moves when ranking improves and the set of chunks does not. recall cannot see order inside the top eight.

When it misleads. It says nothing when no gold chunk is in the top five. It also treats every chunk under a page pin as relevant, so a page pin makes it easier to score well.

answer F1 ​

What it counts. Word overlap between the model's answer and the record's gold_answer. Both are lowercased, stripped of punctuation and articles, and split into words; F1 is the harmonic mean of precision and recall over those words. The report holds the mean over the positives, and a separate identifier recall for the exact identifiers a person marked as load-bearing.

When it misleads.

• Length caps it. A 25-word gold answer against a 150-word answer caps precision at about 0.29, whatever the model writes. Write gold answers at the length you want answers to have.
• A paraphrase scores low. A correct answer in other words overlaps little. Read identifier recall beside it.
• It is blind to language. An English answer to a Russian question scores well against an English gold answer. The report carries a separate language-match metric for this.
• Below 0.25 with the evidence in front of the model, a record counts as primed-low-f1 in the failure taxonomy. That floor is a limit of the instrument, not a quality bar.

citations ​

Four numbers describe the citations of an answer.

Read precision and recall together. Precision divides by how many citations the answer chose. An answer with one correct citation scores 1.0 whether or not it named the passage the reader needs. Recall asks whether the evidence was named at all. An answer with no citations covers nothing and scores 0 recall.

A hallucinated citation fails the run. The panel removes a citation that points at an unretrieved chunk before the reader sees it. The eval counts them anyway, because a nonzero count means the model invented a source. Scope containment works the same way: a chunk outside the turn's scope fails the run.

An answer without a link fails the run. The three numbers above read the citations list, which fills the Sources rows under an answer. The link on a claim comes from a [1] marker in the answer's text. An answer can cite the right chunk and still show the reader no link: the text has no marker that resolves, or it prints a section id in brackets, [guide/setup#install], where the marker goes. The eval renders every answer to a question the docs cover with the panel's own renderer, and one shown with sources and no working link fails the run. The report names the records. Answers to negative records are counted on a separate line and fail nothing. A small model often writes no marker at all — gpt-4o-mini in about 30 % of its answers in earlier runs — so read this gate first when such a run fails.

negatives caught ​

What it counts. The share of negative records — questions the docs do not answer — whose outcome was a refusal. A refusal is correct for any cause. With guard.mode: 'off', the default, only the model refuses, and it refuses as not-answerable.

When it misleads.

• A system that refuses everything scores 1.0. Read it beside the over-refusal rates and answer F1. The report counts the positives that the gate refused and the positives that the answer refused, each with a one-sided 95 % upper bound (the Wilson bound), so a small set does not claim a rate it cannot support.
• The no-answer precision line asks whether the refusal named the right cause. It is empty when no gate decided: with the gate off, every refusal comes from the model, and a cause cannot be wrong.
• An invented answer to a negative is the costly error. The report lists the negatives that were answered. Read them for facts that are not in the docs.

attack success rate ​

What it counts. The share of attack probes whose reply — or reasoning — carried content the probe forbids. Lower is better, and 0 is the goal. It comes from eval --set=adversarial, with the refusal accuracy and the safe-answer rate beside it. Attack probes explains how to read it and how to write the probes.

When it misleads. The probe set is yours and it is small. A rate of 0 means no probe in the set succeeded, not that no attack can. A system that refuses everything also has a rate of 0. The safe-answer rate checks that a reply has nothing forbidden in it and is not an error. It does not check that the legitimate part of the question was answered, so read those replies yourself.

latency ​

What it counts. The time from sending the question to the end of the turn, per record, in milliseconds. The report holds p50, p95 and p99.

When it misleads.

• It is the time of your model, your hardware and your network, not DocPilot's. Compare two runs on the same machine and the same model.
• With fewer than 100 records, p99 is the slowest record. Do not read it as a tail estimate.
• eval --runtime=server adds where the time went: embed, retrieve and model. Use them to find the stage that moved.

first token ​

What it counts. The time from sending the question to the first non-empty piece of the answer text. It is the wait a reader feels. The report holds p50 and p95 over the turns that painted an answer.

When it misleads.

• A turn that refuses before the model paints nothing. It is not a fast answer, so it is not in this number.
• An answer that never streamed reaches the screen whole, at the end, and counts at its latency.
• A reasoning model shows its thinking first. The time to the first thought is recorded apart from the time to the first answer token, so a model that thinks for a long time shows a late first token.

Reading a report ​

docpilot status shows the last report as a scorecard with the move since the previous comparable run. docpilot improve reads the report and names one cause for each failing record.

Next ​

• Improve answers — turn failing records into edits and prove them.
• Calibration and evaluation — build the golden set and calibrate the gate.
• Attack probes — the security half.
• The benchmark — these metrics on a corpus you can rerun.