AI pilot results: worked out examples

You ask what sits behind the sponsor’s 90% headline. The interviewer shares Exhibit 1: the pilot task funnel, covering the same 200 assigned tasks at one review cutoff. “Accepted” means the output meets the agreed task criteria.

The sponsor describes this as “90% acceptance”. Calculate completion, acceptance among completed tasks and accepted delivery across all assigned tasks before deciding what that headline means.

Completion is 120 ÷ 200 = 60%. Acceptance among finished outputs is 108 ÷ 120 = 90%. End-to-end accepted delivery is 108 ÷ 200 = 54%.

These numbers fit together: 60% completion × 90% conditional acceptance = 54% accepted delivery.

The headline is describing the quality of outputs that reached the finish line. The overall workflow delivered accepted results for just over half the assigned workload by the cutoff.

A useful interpretation is: “The finished outputs are often acceptable, but completion is the larger immediate gap.” That statement identifies where to investigate next without discarding the promising part of the pilot.

Eighty assigned tasks did not finish. Twelve finished but were rejected. Together they account for 92 tasks without an accepted output at the cutoff.

I would split the unfinished group by the point where work stopped: intake, access to source material, processing, review or handoff. A missing document needs a different fix from a processing timeout. A task completed manually may represent fallback workload rather than an abandoned customer request.

Also check whether the unfinished tasks are harder than the finished ones. If the system tends to finish easy documents, the 90% figure tells you little about how it handles the remaining work.

Count the delivered outcome the business needs. A promising intermediate response does not establish that the full task succeeded.

Consider a scope variation: suppose investigation finds that 50 of the 200 assigned tasks were ineligible under the pilot’s original rules. Of the 150 eligible tasks, 120 finished and 108 were accepted.

Within the eligible population, completion is 80% and end-to-end accepted delivery is 72%. Conditional acceptance among finished tasks remains 90%.

Report both the eligible-work result and the intake problem. If the proposed service must handle all 200 incoming tasks, the other 50 still need a route and a cost. Relabelling them does not remove them from the customer's workflow.

If the decision is whether to deploy only on the eligible task types, the 72% measure becomes relevant. Make the proposed scope explicit and size the fallback required for the remaining eligible failures as well.

Now return to the original 200-task comparison, without the eligibility adjustment. The interviewer supplies results for a second pilot on that same workload and cutoff: 180 tasks finished and 162 produced accepted outputs.

Its conditional acceptance is also 162 ÷ 180 = 90%. But its accepted delivery across assigned work is 81%, compared with 54% for the first pilot.

The two pilots would look identical under the original headline. On completed useful work, the second delivers 54 more accepted tasks.

You still need cost and timing to choose between them. The exhibit now tells you which comparison to make rather than letting a shared 90% figure conceal a material difference.

Return to the first pilot's original 200-task scope. Suppose its total processing and review cost is £1,200. The remaining 92 tasks can each be completed manually to the required standard for £10, with no additional charges in this exercise.

Total cost of completing the workload is £1,200 + 92 × £10 = £2,120. Completing all 200 manually at £10 would cost £2,000.

On those assumptions, the combined AI-plus-fallback route costs £120 more. Dividing £1,200 by the 120 finished outputs gives £10, but that comparison omits rejected and unfinished work and does not represent the cost of delivering the whole workload.

At 108 accepted AI outputs, the pilot's processing-and-review cost must fall to £1,080 to match the £2,000 manual total, because the £920 fallback bill remains. Alternatively, better accepted delivery reduces the fallback requirement.

This gives the team two measurable improvement paths: reduce the pilot cost or increase accepted completion.

A clear response is:

“The pilot delivers accepted results for 54% of assigned tasks by the cutoff. I would focus the next iteration on the unfinished work before scaling. With the supplied costs, AI plus manual fallback costs £2,120 versus £2,000 manually, so we need either more accepted completions or at least £120 lower pilot cost to match the baseline.”

That is a confident conclusion from the available evidence, with a specific route to improving the proposal.

Try the EV sales-funnel exhibit drill to practise reading stages and denominators in another setting. For another comparison based on accepted output and total cost, continue with the AI inference cost framework.