A support answer can cite a document and still make a claim that the document never makes. Build the evaluation set around the individual claims an agent would approve, then preserve the passages that justify them. This gives the reviewer something concrete to inspect when a fluent answer gets the policy wrong.
This guide draws on published evaluation documentation and the AI Support Desk planning brief. The product remains an unimplemented blueprint. The worksheet contains proposed checks, with every result marked Not run. It contains no customer conversations or measured accuracy.
Start with questions your team can label
Ask the support owner for a small, approved sample of real questions. Remove personal information and record which document version was available when each question was asked. Keep the original question wording where permission allows. A rewritten question may remove the ambiguity that made the original difficult.
For every question, assign a workspace, an expected response type and the specific evidence an authorized agent can use. A useful reference answer lists the necessary claims and their source passages. Avoid requiring one exact sentence when several accurate phrasings would satisfy the request.
Ragas separates several checks that are useful here. Faithfulness evaluates whether response claims are supported by retrieved context. Context recall asks how much relevant information was retrieved and needs a reference for comparison. Context precision examines whether relevant retrieved material ranks ahead of irrelevant material. These measure different parts of the process. Faithfulness, context recall and context precision.
Include questions that should stop an answer
Use the following categories as proposed sampling rules. Their proportions should follow the support team's actual work. They are not a ready-made benchmark or evidence of the frequency of any failure.
| Question class | Evidence prepared by the reviewer | Expected behavior |
|---|---|---|
| Supported | Current authorized passage | Answer within its scope |
| Missing information | Approved collection lacks the answer | State the gap and request human help |
| Conflicting sources | Two applicable passages disagree | Expose the conflict for review |
| Restricted information | Relevant passage belongs to another workspace | Do not retrieve or disclose it |
| Instruction attack | Source or question contains an unauthorized instruction | Preserve the configured task boundary |
The restricted case needs an authenticated test identity. Azure AI Search documents a security-filter pattern that compares principal strings, while explicitly distinguishing that filtering from authentication and authorization. A passing retrieval filter alone therefore cannot establish that the requester was entitled to those principal values. Azure security filtering.
Prompt attacks also deserve their own cases. Amazon Bedrock distinguishes instruction override attempts and attempts to extract confidential configuration. Use those categories to prepare clearly labeled synthetic questions alongside the approved real questions. Synthetic cases should retain their origin label when the set is exported. Bedrock prompt attack categories.
Inspect the answer claim by claim
For a supported question, save the retrieved passage identifiers, their order and the final draft. Break the draft into statements that could independently be wrong. Attach the passage that supports each statement. A link to the right manual is insufficient when the quoted section addresses a different feature or an older policy.
Have the reviewer mark each claim as supported, unsupported or unresolved. Record whether the answer omitted a required qualification. Keep access failures as blocking findings even if every visible statement happens to be correct. An average score should never hide a cross-workspace disclosure.
For missing information, score the intended handoff separately. The draft should explain which information is missing without inventing the missing policy. A useful review note specifies which team owns the next decision and which source would make a later answer possible.
Keep comparisons reproducible
Freeze the source versions, retrieval settings and question set for each comparison. Record the model and prompt revision, but avoid placing credentials or private source text in a public report. When a reviewer changes a reference answer, retain the reason and rerun affected cases before comparing scores.
The download provides fields for these decisions and blank evidence cells. Bring the approved document collection, representative questions and handoff owner to the first review. Download the RAG evaluation worksheet.
For a scoped internal support workflow, review the AI Support Desk blueprint. Its proposed release includes source-linked drafts and human review; implementation and evaluation remain to be agreed.
