We scored a store’s AI support agent on 356 help-desk tickets the way a support lead would: the right answer, the right handoff, nothing invented. Then we ran the rescue we sell and re-scored it on tickets we never looked at while fixing it.
Alder & Ash is a fictional outdoor apparel store. The tickets are synthetic. The bot replies and grades are real model runs.
Vendor dashboards count every chat the bot didn’t hand off as resolved. We count a ticket as resolved only when the reply is correct and complete against the store’s real policy and order records.
For context: Intercom advertises a 76% resolution rate for Fin. Independent production tests report 38-53%, and those tests come from vendors selling alternatives, so treat both as directional (getmacha, Builts.ai, CloneDesk). The numbers on this page are our own runs on our own fictional store.
Every reply below came from the same model. Pick a ticket to see the customer’s message, the true order record, both replies, the code checks and the judge’s verdict.
Three layers, in the order we work: find why tickets fail on a dev set, fix the knowledge, give the bot the actions and routing it was missing. The held-out tickets stayed closed until the end.
order_lookupreadFinds the order by number, or the signed-in customer’s latest order. Returns status, carrier, tracking and dates. Refuses orders that belong to someone else. Flags a parcel as lost when tracking has stalled 7 days (US) or 10 (international).
order_lookup(order_id?, email) → {status, tracking, delivered, lost}
refund_checkwrite, gatedApplies the return policy in code: 30-day window, final-sale items, 90-day defects. Starts the return or issues the refund itself, up to $150. Above that it opens an approval request and hands the customer to a person.
charge_lookupread + refundReads the customer’s charges for an order. Tells a pending authorization (drops off in 3-5 business days) from a posted duplicate, and refunds the duplicate under the same $150 gate.
charge_lookup(email, order_id?) → pending | duplicate_refunded
A planner reads every ticket, and a code rule backs it up, so a missed flag still reaches a person. Data-driven handoffs (lost parcel, address change before shipping, refunds over $150, warranty after 90 days) are decided by the actions, not by the model.
It is our own demo on our own fictional store. Read these before you quote any number from it.
Your numbers, one formula. We make no claim about the lift you will get; that is what the scorecard measures.
$0per month
Export from Intercom, Zendesk or HubSpot, strip names and e-mails, send it over. You get this page for your bot: real resolution next to the dashboard number, false resolutions, missed escalations, by intent, with the transcripts.
$3,500fixed, two weeks
from $1,500per month
Resolution Scorecard by KH Labs. Alder & Ash is a fictional company; every customer, order, charge and ticket on this page is synthetic. Intercom, Zendesk and HubSpot are named only as platforms we work on; their bots were not tested here.