Evaluate an AI support assistant with representative questions, approved source material and an expected answer or handoff for each case. Review what the answer claims and what supports it. Fluency alone does not show that the assistant gives the right advice or recognises a request it cannot handle.
A useful evaluation set makes the release decision specific: which support workload has been checked, which failures remain and what evidence supports the next bounded step? It should help the team improve the system without disguising uncertainty behind a single accuracy score.
Define the support workload the assistant may handle
Separate public information questions from account-specific requests and actions. Explaining a published delivery policy differs from confirming the status of a particular order or changing an address. Each may require different sources, identity checks and authority.
Choose an initial boundary that support staff can describe. List the topics the assistant may answer, the topics it should route elsewhere and the actions it must not infer permission to perform. A broad instruction to “help customers” is difficult to evaluate consistently.
Include the intended audience and channel. A short website response and an internal support draft may have different review arrangements. If a person checks every response before sending, record that dependency rather than treating the assistant as independently customer-facing.
Atlassian's support-agent evaluation documentation illustrates using questions and inspecting batch results. The useful principle is a repeatable set of cases. A vendor's workflow or score should not automatically become the acceptance rule for a different assistant.
Choose representative questions and approved evidence
Build cases around user intent rather than a list of product terms. Include a direct question, an ordinary paraphrase and a question missing information the assistant needs. Choose the mix because it represents the planned workload, not because it produces reassuring demonstrations.
Use fictional or properly sanitised questions where possible. Preserve the meaningful ambiguity without copying unnecessary personal details from support conversations. Have a support owner confirm that the cases still resemble real work.
For each case, identify the approved source and its version. If a policy changes, reviewers need to know which answer was expected at the time of the run. A link to a continually edited page without a retained reference can make an old result difficult to interpret.
Separate development cases from a held-back review set where practical. If every adjustment is judged only on the examples used to make it, improvement may be narrower than the team assumes. The size and composition of the set should follow the scope and risks, not an invented universal minimum.
Describe the expected answer or escalation
Record the claims the answer may make, the source supporting them and the limits it must communicate. Do not require an exact phrase when several clear answers would satisfy the same intent. Require the essential facts and behaviour instead.
Include unanswerable and conflicting-source cases
A plausible invented policy can be more damaging than an explicit handoff. Include questions for which the approved evidence has no answer. Also include conflicting or outdated source material and define whether the assistant should ask a question, withhold a claim or refer the case to a person.
The following dataset excerpt is hypothetical. It describes evaluation requirements, not measured performance.
| Question class | Expected outcome | Reviewer checks |
|---|---|---|
| Published policy question | Explain the applicable policy from the named approved version | Correct claim, relevant source and important qualification |
| Policy absent from the sources | State the limit and follow the agreed handoff | No invented exception or unsupported promise |
| Account-specific change request | Use the identity and authority route for that action | No unauthorised action inferred from a conversational request |
Make handoff useful. Specify what the customer should do next and which context can be passed appropriately to the support team. “Contact support” may be insufficient if the product provides no practical route or leaves the user repeating the entire problem.
Review answers by failure category
Distinguish retrieval failure, unsupported claims, missing qualifications, irrelevant answers and incorrect escalation. These categories point to different changes. Adding more documents will not necessarily correct a failure to respect an action boundary.
Microsoft's RAG evaluation documentation distinguishes dimensions of grounded-answer evaluation. Use separate evidence for whether relevant material was found and whether the answer stayed within its support. Neither dimension alone establishes overall readiness.
Review citations too. A source can be real but irrelevant to the particular statement beside it. A correct sentence linked to the wrong policy version remains a defect in the evidence trail.
Record consequence as well as category. A missing minor detail and an invented contractual promise should not disappear into an undifferentiated average. Have the responsible business owner decide which failures prevent the planned release and which require a bounded correction.
Manage reviewer disagreement and repeatable changes
Give reviewers the same question, source version and criteria. Ask them to record why they disagree. The disagreement may reveal an unclear policy or expected outcome rather than inconsistent judgment.
Include source changes and handoff failures within the agreed software acceptance testing.
Keep a run record containing the evaluation set version, knowledge-source version, relevant configuration and observed answers. When the system changes, compare like-for-like cases and inspect important regressions. Re-running a different question set cannot cleanly explain the effect of one change.
Automated scoring can help organise a larger review, but define what it measures and check important cases with people who understand the support domain. A model-generated score should not silently decide whether an unsupported promise is acceptable.
Retain failed examples after correcting them. They become regression cases. Add new cases when the source collection or authorised workload changes, while preserving enough history to explain why an earlier decision no longer applies.
Add a paired example to the dataset. One fictional question asks for the published returns process; a second asks whether an exception applies to an individual account. The first may be answerable from the approved policy, while the second may require information or authority the assistant lacks. Review whether a small wording change causes the assistant to invent an exception. Preserve the expected distinction in the record so a future improvement to answer fluency does not remove the handoff boundary.
Record reviewer confidence separately from the assistant's wording. If the policy owner cannot settle the expected result, mark the case as an unresolved source or business decision. Do not force a pass or fail merely to complete the dataset.
Use the evidence to choose a bounded next step
The next decision might be to improve the sources, narrow the assistant's scope or run a supervised pilot. Describe that decision with its conditions and owner. Avoid turning a successful demonstration into a claim that all support questions are covered.
Use the evaluation evidence to inform AI model selection, rather than choosing a model from a demo alone.
A useful review leaves support staff able to explain the allowed workload, the observed failures and the escalation route. It also identifies what has not been evaluated, including languages, account conditions or actions outside the selected cases.
Discuss the support workload and acceptance evidence for support chatbot development with ZakCodeX.
