AI support pilot: how to run one and judge the result
How to test an AI support agent over two weeks: gather your real questions, set hand-over rules, define a resolved question and decide whether to keep it.
A demo shows an AI support agent answering questions someone chose in advance. A pilot shows it answering yours, in your channels, from your content. Two weeks is enough to tell whether it will work, provided you prepare first and decide how you will judge the result before the pilot starts.
Prepare first, fix in week one, count in week two, then decide.
Before day one
- Gather your last 100 conversations
- Write the correct answers
- Mark fixed answers
- Set hand-over rules
Week one: fix
- Read every conversation, 10 minutes a day
- Add the missing answer or rule
- Keep a log of fixes
Week two: count
- Stop changing the setup
- Real questions and resolved
- Hand-overs and wrong answers
- Time to first reply
Decide
- Resolved ÷ real questions
- Our pilot line: at least half
Before day one: gather your real questions
Go through your last 100 customer conversations from email, WhatsApp, chat or wherever most arrive. Group them by question. The same few questions usually repeat many times: delivery, prices, opening hours, refund rules, how to book, where an order is.
For each group, write the correct answer in one or two plain sentences, or note which page on your site holds it. The agent answers from what you give it, so any gap in this list becomes a gap in its answers.
Name one owner
Give the pilot to one person in your team. That person reads the conversations, fixes the content and decides when something goes to a colleague. A pilot that belongs to everyone usually gets reviewed by no one.
Fix the answers that must never change
Prices, opening hours and policy wording should come out exactly as you wrote them. Mark these as fixed answers so the agent quotes them and does not reword them. Leave the agent free to phrase everything else in its own words.
Decide what goes to a person
Write down the questions the agent should hand over: angry customers, refund requests outside your policy, anything legal, medical or about safety, and large orders. Set the message the customer sees and name who receives the alert. The guide to hand-over rules shows how to write them.
Start with one or two channels
Pick the channel with the most messages, usually WhatsApp, website chat or email. A pilot across every channel gives you several sets of problems at once and makes the results harder to read.
Define a resolved question before you start
A question counts as resolved when the customer received a correct answer and no person had to step in. Leave out spam, wrong numbers and messages that were never questions. A hand-over that your rules called for is a correct outcome, so count it separately and do not treat it as a failure.
Divide resolved questions by real customer questions. Our own pilot terms put the line at 50%: if the agent does not resolve at least half of your real questions during the pilot, you owe nothing.
Week one: read every conversation
Spend ten minutes a day reading what the agent said. When an answer is wrong, find the cause. Either your content did not contain the answer or a rule was missing. Add the answer or the rule and note the fix in a log. Do not judge the agent yet, because this week is for fixing.
Week two: stop changing things and count
Leave the setup alone and measure. Record how many real questions came in, how many the agent resolved, how many it handed over, how many answers were wrong and how quickly the first reply arrived. Write down any customer complaint about the agent.
If you receive few messages
A small business may see only a handful of questions a day, which makes two weeks a thin sample. In that case collect the questions customers asked in the past and send them to the agent yourself as test messages, then count those results separately from live ones. Do not blend the two numbers.
What a two-week pilot cannot tell you
It cannot show how the agent copes with a seasonal rush or a product launch. It also cannot show how your content ages as prices and policies change. Plan a monthly review after the pilot so the content stays current.
Deciding what to do next
If the agent resolved at least half and gave few wrong answers, keep it and extend it to the next channel. If it fell short, look at why. A low score caused by missing content improves quickly once you fill the gaps. A low score because most of your questions need a person means automation suits your business less than you hoped, and two weeks is a cheap way to find that out.
Our free pilot runs for two weeks and needs no card. You can book a demo to see how it would handle your questions, or look at pricing for what comes after. If you are comparing costs first, how much AI customer support costs in 2026 explains the numbers.
