An AI quality dip: narrow the rollout and rebuild the economics

Suppose a fictional assistant has stable acceptance rates of 98% on routine tasks and 88% on complex tasks. The client requires at least 95% acceptance in each deployed task segment.

The trial initially contained 90% routine tasks and 10% complex tasks. Its aggregate acceptance was:

90% × 98% + 10% × 88% = 97%.

The intended production mix is now 60% routine and 40% complex. At unchanged segment performance, aggregate acceptance becomes:

60% × 98% + 40% × 88% = 94%.

The overall score falls even though neither segment deteriorates. More difficult work has become a larger part of the workload.

There is also an earlier mistake to correct: an aggregate 97% never established that every segment cleared a 95% segment-level requirement. Complex work was already below the floor.

A useful response is:

“I would withdraw the all-task rollout. Routine work meets the acceptance requirement, but complex work does not. I would evaluate a routine-only deployment and separately test improvements on the complex workload.”

This keeps the successful routine result and changes the unsupported broader claim. It also names the next two questions: can routine-only deployment work economically, and what would qualify complex work later?

Do not explain away the complex-task result by pointing back to the earlier average. The production workload and the client’s stated requirement determine the decision.

To assess the smaller rollout, ask for volume, costs and the platform charge. The interviewer supplies Exhibit 1: monthly economics at the intended 60/40 production mix, with 10,000 tasks a month. AI-assisted costs include review and rework, and the platform has a £30,000 monthly fixed charge. Routing costs will be supplied separately.

The 6,000 routine and 4,000 complex tasks come from applying the stated 60/40 mix to 10,000 monthly tasks.

A full rollout would appear to save 6,000 × £6 + 4,000 × £2 = £44,000 before the fixed charge, or £14,000 net. But it fails the complex-task quality requirement.

A routine-only rollout saves 6,000 × £6 = £36,000, less £30,000, leaving £6,000 per month before any additional routing cost. The complex tasks remain on the current process and contribute no saving to this option.

The recommendation has become smaller, and so has its financial benefit.

The interviewer adds that reliably identifying eligible routine tasks will cost £1 for every incoming task, including those ultimately sent to the existing process.

Routing costs 10,000 × £1 = £10,000 per month. The routine-only option now produces £6,000 − £10,000 = −£4,000 net monthly value on the stated costs.

That changes the recommendation again:

“Routine work qualifies on quality, but the scoped deployment does not yet pay after routing. I would renegotiate the fixed charge or reduce routing cost before committing.”

A technically acceptable segment is not automatically an economically attractive deployment. This is the point at which a candidate should show judgment rather than enthusiasm for keeping part of the project alive.

With £36,000 routine-task savings and £10,000 routing cost, the platform charge must be £26,000 or less to break even. A charge below that leaves a positive saving.

Alternatively, if the £30,000 charge remains, routing can cost no more than £6,000 across 10,000 incoming tasks, or £0.60 per task, to break even.

Those thresholds turn “make it cheaper” into a concrete negotiation or design question. They also keep the discussion tied to the actual eligible workload rather than to optimistic full-rollout volume.

To revisit broader deployment, build an evaluation around the kinds of complex tasks production will contain. Check the acceptance outcome, review burden and full task cost on that scope.

A new test dominated by routine tasks would recreate the original mistake. Define the task mix and required outcomes before judging the system.

The test should also examine routing mistakes. If complex tasks are incorrectly classified as routine, a routine-only policy on paper may still send unsuitable work through the assistant. Measure the whole proposed workflow, including the fallback path.

Suppose complex-task acceptance eventually passes the requirement but review costs rise to £11 per task. That would qualify the workflow on quality while making it £1 more expensive than the current process. You would still need another business benefit to justify deployment there.

If costs fall but acceptance remains below the required floor, the segment still does not qualify. The client’s decision uses both conditions.

Measure an eligible task completed at the required quality and full operating cost. Usage alone does not establish the value of that outcome.

“I would pause the full rollout and avoid committing to the current routine-only contract. Routine quality is sufficient, but after routing the scoped option loses £4,000 a month. I would reopen it below a £26,000 platform charge or with routing below £0.60 per incoming task, while testing complex work separately.”

Try Cybersecurity Product Expansion. When new evidence narrows an option, recalculate its value at the new scope. For the weighted-average mechanics, see AI support mix.