FAQ support bot with source boundaries and human escalation
A client needed repetitive pre-sale and support questions handled consistently without allowing the AI to invent delivery, refund or account-specific policy.
Starting problem
The chatbot could answer simple product questions, but policy answers were inconsistent, fallback behavior was weak and difficult conversations did not reliably reach a human. There was also no reusable regression test set.
Constraints
- Only approved FAQ, product and policy material could be treated as authoritative.
- Refund disputes, complaints and account-specific issues needed human escalation.
- The existing website/chat platform had to remain in place.
- Private payment credentials and unnecessary customer account data stayed outside chatbot context.
A→Z implementation
- Discovery and scopeMapped frequent intents, excluded topics, tone, response length and escalation triggers using examples of weak conversations.
- Knowledge cleanupOrganized approved delivery, returns, product and account-help material and flagged conflicting or incomplete source information.
- Instruction designRebuilt the system instructions around source boundaries, uncertainty handling, clarification questions and a rule not to invent policy.
- Fallback and escalationDefined specific behavior for missing evidence, complaints, sensitive topics and requests requiring customer-specific decisions.
- QA and regression testingTested normal FAQs, ambiguous wording, multi-part questions, hostile prompts, outdated information and unsupported requests.
- Refinement and handoverCorrected failure patterns, froze a baseline configuration and delivered a maintenance/retesting process.
Artifacts delivered
Acceptance criteria
- Approved FAQ scenarios followed the intended tone and structure.
- Unsupported policy questions triggered clarification or escalation instead of invention.
- Complaint and account-specific cases followed the human handoff path.
- Core scenarios could be rerun after future prompt or content updates.