Operations

Measuring an AI sales agent: the four numbers that matter

Response time, qualified rate, quote rate, closed revenue — and the vanity metrics to ignore.

· 5 min read · Editor: Max

Anyone who deploys an automated sales assistant faces the same question within a month: is it working. The dashboards that come with these tools rarely answer it. They report conversations, messages, resolution rates and satisfaction scores, all of which move in the right direction as soon as you switch the thing on, and none of which tell you whether you sold more.

Four numbers do the real work. Everything else is context.

1. Time to first substantive reply

Measure the gap between a lead arriving and a reply that actually engages with what was asked. Not an acknowledgement. Not "thanks, we will be in touch." A reply containing an answer or a specific question.

Use the median, and look at the distribution behind it. Averages are dominated by the tail — the enquiry that arrived on Friday evening and waited until Tuesday. Track the 90th percentile alongside the median to find where the process actually breaks.

This metric matters because it is the one with the best evidence behind it. The Lead Response Management study led by Prof. James Oldroyd found that the odds of qualifying a lead fall sharply once the first contact slips from minutes into hours. The Harvard Business Review article "The Short Life of Online Sales Leads" reported the same pattern and noted how much slower most companies were than they believed.

Before you change anything, measure this for the previous quarter. Without a baseline, every improvement is an anecdote.

2. Qualified rate

Of the leads that arrive, what share reach a state where a salesperson would want to spend time on them? You need a definition that is written down and stable: budget signal present, timeline stated, the request fits something you actually sell.

The reason this number sits second is that it guards against the obvious failure mode of the first. It is trivial to make replies faster by replying to everything instantly with nothing of substance. Speed without qualification is just noise arriving sooner.

Watch the two together. Response time down and qualified rate flat or rising means the system is working. Response time down and qualified rate falling means you have automated the appearance of responsiveness.

Any single metric can be gamed by the process that reports it. Pairs are harder to fake.

3. Quote rate

What share of qualified leads receive a concrete, priced proposal, and how long does that take from first contact?

This is where most small and mid-sized sales operations lose the most money, and it is the least examined step in the funnel. Leads arrive, conversations happen, everyone is busy, and a meaningful share of genuinely interested buyers never receive a document they could act on. Nobody said no. The thread simply stopped.

Quote rate is also the clearest test of whether automation is doing work rather than making conversation. A system that handles enquiries pleasantly but produces no more quotes than before has added activity, not throughput.

Split it by product line and by channel. The variation usually points straight at a specific bottleneck: one product whose pricing needs an engineer, one channel where the handover is broken.

4. Closed revenue attributable to automated first contact

The number that ends the argument. Of revenue closed this period, how much came from leads whose first contact was handled by the system?

Attribution here should be deliberately crude. Do not build a multi-touch model. Tag the lead at first contact, follow it to the close, and report the total. Crude attribution that everyone understands is more useful than a sophisticated model nobody trusts, because the purpose of this number is to settle whether to keep going.

Compare it against the fully loaded cost: subscription, setup, the internal time spent maintaining rules and reviewing conversations. If the comparison is uncomfortably close, that is useful information too.

The vanity metrics

These appear in almost every dashboard and none of them belong in a decision.

  • Number of conversations handled. Rises automatically with traffic and with anything that makes the assistant chattier. Measures volume, not value.
  • Messages sent. Actively misleading. A system that needs six messages to do what a person does in two is performing worse, and this metric rewards it.
  • Resolution rate or containment rate. Borrowed from customer support, where deflection is the goal. In sales, "resolved without a human" is not a win if the human would have closed the deal. Optimising containment optimises against revenue.
  • Customer satisfaction on the chat itself. Polite, fast replies score well regardless of whether anything was sold. A buyer can rate an interaction highly and buy elsewhere.
  • Uptime and availability figures. Operationally relevant, commercially irrelevant. Nobody bought more because the service reported a high availability number.
  • Cost per conversation. Falls as volume rises, which looks like progress when nothing improved.

The common flaw is that all of them can improve while revenue is flat. A metric that cannot move against you is not measuring anything.

Reading the four together

Set them up as a chain, because that is how they behave. Leads arrive, some get a fast substantive reply, some of those qualify, some of those receive a quote, some of those close. A change at any stage shows up downstream with a lag equal to your sales cycle — which is the main reason to be patient before judging.

Two diagnostic patterns are worth knowing. Response time improves but qualified rate drops: the first reply is fast and empty, or it is answering the wrong kind of enquiry. Qualified rate and quote rate both look healthy but closed revenue does not move: the bottleneck is downstream, in follow-up or in pricing, and no amount of front-end automation will fix it.

Review monthly, judge quarterly, and keep the definitions fixed while you do. Redefining "qualified" halfway through is the most common way these reviews end up meaningless.

See what the reporting covers on the product page, or read more on where lead data breaks.

Axelion answers in seconds, around the clock. It replies in your website chat, recommends the right product from your pages, saves every lead and hands complex cases to your team. WhatsApp and Telegram on Business. See it on your own website →

Related articles

CRM data quality starts with the first messageAI chatbot for your website: how to choose one in 2026

← All articles