Testing method · 8 steps
How to Test an AI Sales Agent on Real Pipeline Outcomes
A demo is one run on the vendor’s best example. To know what an AI sales agent will do for your pipeline, run it on your own accounts several times, score it against facts you already know, and compare its pipeline with a group it never touched.
- Pick two or three tasks you actually need doneChoose real, repeatable jobs, such as finding ICP accounts that hired a first RevOps lead, qualifying last week's signups, or flagging stalled deals. Write each one down exactly as you would brief a new hire, with your ICP and your rules. Every agent gets the same brief.
- Build an answer key before any agent runsTake 20 to 50 accounts you know well. Record the facts the agent should find: who holds the role now, who left, which accounts are customers or open deals, which contacts opted out. Seed in a few traps: a contact who changed jobs, a customer that looks like a prospect, an address on your suppression list.
- Run each task at least five timesSame brief, same inputs, separate runs. Export every output. One run tells you what the agent can do; five tell you what it will do. Keep each run's outputs side by side.
- Score every run against the keyFor each run, count: facts wrong (old titles, wrong companies, bad emails), rules broken (a customer, open deal or suppressed address included), steps skipped, and runs that stopped early. Then look at the spread across runs, not only the average. An agent that is right four times and badly wrong once is a risk at scale.
- Measure what review costsHave the people who would review output in production do it during the trial. Record how many drafts they approve unchanged, edit or reject, and how long it takes. This is the real cost of the agent next to its price.
- Run a holdout against pipelineSplit comparable accounts or leads into two groups before you start. The agent works one group; your current process works the other, over the same weeks. Compare replies, meetings held and qualified opportunities created, and agree the definition of a qualified opportunity in advance.
- Give it long enoughMeetings and opportunities take weeks to appear. Run the pipeline comparison for at least one full sales cycle stage you can measure, typically several weeks for meetings held, and keep the repeated-run checks going as the vendor ships updates.
- Decide on the evidenceKeep the agent if its error rate across runs is low, it never broke a send-layer rule, review time is acceptable and the agent group produced more qualified pipeline than the holdout. If it only looked good in the demo, you have your answer.
The scorecard
Six numbers per agent, each tied to one of the ranking criteria.
| Measure | How | Criterion |
|---|---|---|
| Facts wrong per run | Count against your answer key | Data freshness |
| Spread across five runs | Best run versus worst run on the same task | Repeated-run reliability |
| Runs that stopped or skipped steps | Out of all runs | Repeated-run reliability |
| Drafts approved unchanged | Share of drafts, and review minutes per draft | Human review |
| Suppressed, customer or open-deal contacts reached | Must be zero | Send-layer safety |
| Qualified opportunities versus holdout | Same period, agreed definition | Pipeline outcomes |
Why repeated runs, and why pipeline
Published research points the same way. Frontrunner’s PipelineBench, which simulates a year of GTM work and reports the mean of 30 runs per model, found that 1 in 10 model and task pairs passed one run and failed the next, that some models stopped working before the end of the year in a large share of runs, and that stale data, more than writing, kept even the best model well short of what perfect information allowed. Its conclusion: whether your data is current matters more than the model, and one good run is not an agent. You can read the method and the full table at usefr.com/pipelinebench.
The same logic applies to your trial. Activity numbers (emails sent, accounts researched) are easy for any agent to produce. Qualified pipeline against a holdout is the number that tells you whether to keep paying.
Before the trial
Agree the tasks, the answer key, the number of runs and the holdout with the vendor up front. The questions to ask vendors cover what to get in writing, and the ranking shows which vendors already publish evidence on each criterion.
Get the next review of the ranking
One email when an agent enters, leaves or moves in the ranking, with what changed and why. Nothing else.