Demo store: Alder & Ash is fictional; the model runs are real.
Fixes and fresh-ticket tests
Three fresh rounds, each on 356 new tickets, after an initial test. We fixed, then re-tested on new tickets nobody had seen, three times. Round 0 was the original published test on 223 tickets kept out of the build; rounds 1 to 3 each used 356 new tickets, generated after that round’s fix. The same AI model and the same two graders (code checks plus an AI grader) were used throughout. Every round is listed.
| Round | Fix being tested | Real resolution | False resolution | Missed escalations |
|---|---|---|---|---|
| 0initial test, published 12 Nov 2026, 223 tickets | Cleaned knowledge base, order and refund actions, routing rules | 73.1%was 22.4% | 1.2%was 47.4% | 0.0%was 15.5% |
| 1356 new tickets | Refund amounts and bank timing come from the order record. Every dollar amount and day range is checked in code; a reply that fails is rewritten once, then replaced by a safe template or a handoff | 72.5%was 23.3% | 0.4%was 47.5% | 0.0%was 19.4% |
| 2356 new tickets | Four help-center lines the policy does not support were removed (pausing, gift-card delivery, support hours, promo-code guess) | 71.1%was 23.0% | 1.6%was 44.6% | 0.0%was 12.9% |
| 3final fresh round, 356 new tickets | If the AI fails to reply, the ticket goes to a person. Every shipping time states its unit, and a number range without one is caught | 73.3%was 22.5% | 0.0%was 45.2% | 0.0%was 19.4% |
“was” is the default bot on the same tickets. False resolution: of the tickets the bot closed itself, the share that were wrong. Missed escalations: of the tickets that needed a person, the share the bot kept.
What each round’s failures were
- Round 0: two refund replies quoted the order total instead of the refund (item price), and one said “3 to 5 days” instead of 5-10.
- Round 1: the bot said Ash Club “can’t be paused”. The cleaned help center said so, but the policy does not, so the claim had no support.
- Round 2: three replies were empty because the AI service failed and the bot sent nothing. One reply said “Canada is 5-8.” with no unit.
- Round 3 (final fresh round):
- Resolved: 261 of the 263 tickets a bot can resolve. Wrong answers: 0. Missed escalations: 0 of 93.
- 2 tickets went to a person when the bot could have answered (safe misses): a promo-code question, which the default bot had answered, and a double charge the bot sent to billing because it read the ticket as a bank dispute.
- The ceiling is 73.9%. The rest must go to a person.
How we measured
Each ticket is one customer message, answered once, for one store. Same AI model (Gemini 3.8 Flash) for the bots and the grader in every round. On your help desk we add multi-turn chats and repeat runs, on your own exported tickets.
Final round: 0 wrong answers in 261 resolved tickets, on 356 tickets the bot had never seen. At 95% confidence, the true rate of wrong answers is under 1.5%.
Run records
One template sentence was removed from the published after file, after the run. Our handoff template ended 94 handoff replies with “Our team replies Monday to Friday, 9am-6pm ET.” The store’s policy sets no support hours, so we deleted that sentence from the template and from those 94 replies in the published file. The sentence came from our template code, not from the AI model. No model output was changed, and no outcome or grade changed.
- Original after run, as recorded 632 KB · JSON
- Published after run, with the sentence removed 628 KB · JSON
- Before run, the default bot 291 KB · JSON
For engineers file names, model id, hashes
Final round files: 20261112T114257Z_before.json, 20261112T114738Z_after.json and 20261112T114738Z_nohours_after.json. The third is the second with the support-hours sentence stripped from 94 template handoffs by runner/strip_hours.py; the reply text of those 94 is the only difference. Model gemini-3.8-flash-high(low) for the bots and the grader. 1,401 model calls and 1,326,450 tokens in the final round, ticket wording included.
Hashes: tickets 888dca7a0b18, knowledge base before 815c0b3a1757, after fdb71af21a5f, rescue config ec3eb4186926 (as run; the template sentence above is the only edit since), graders 544e05239f40. Errors: before 0, after 0; grader errors 0 / 0. Ticket sets are generated from random seeds, kept in the run records.