An energy supplier rolls out AI support. A month later, the headline resolution rate falls. The director wants to reverse the rollout.
Before recommending that, I would rebuild the headline from the enquiry types underneath it. The key question is whether the service became worse at handling comparable work, or whether it received a different mix of work.
Read both the rates and the counts
In this illustrative exhibit, each month contains 1,000 enquiries. Resolution uses the same service window and definition in both months. The two categories cover all enquiries without overlap.
| Enquiry type | Month 1 count | Month 1 resolution | Month 2 count | Month 2 resolution |
|---|---|---|---|---|
| Routine | 900 | 90% | 600 | 92% |
| Complex | 100 | 60% | 400 | 65% |
The first finding is visible without calculation: routine resolution improves two percentage points and complex resolution improves five. At the same time, complex enquiries rise from 10% to 40% of the workload.
Try calculating the total resolution rate in each month. Do not average the two percentages equally; the groups contain different numbers of enquiries.
Quick Exhibit Drill
A grocery delivery firm launched a new driver app last quarter. On-time performance then fell, and the director wants to withdraw the app.
- Overall on-time rate
- 93% in Q1, 87% in Q2
| Route type | Q1 deliveries, on time | Q2 deliveries, on time |
|---|---|---|
| Urban | 9,000 at 95% | 5,000 at 96% |
| Rural | 1,000 at 75% | 5,000 at 78% |
Which reading of the exhibit should shape your advice?
Reconstruct the headline
Month one resolves 900 × 90% + 100 × 60% = 870 enquiries, giving 87% overall resolution.
Month two resolves 600 × 92% + 400 × 65% = 812 enquiries, giving 81.2% overall resolution.
The total falls 5.8 percentage points, even though both categories improve. The new mix contains far more of the category that remains harder to resolve.
That is a substantive explanation, not a statistical technicality. Customers experienced 58 fewer resolutions. The exhibit tells you that worsening within-category rates do not explain those missing resolutions.
Aggregation can reverse the pattern within groups when the mix changes. Explain which group grew and how its performance affects the total before reaching for a statistical label.
Separate the improvement from the workload shift
Apply month-two resolution rates to the original mix:
90% × 92% + 10% × 65% = 89.3%.
At the earlier mix, the new rates would have lifted overall resolution from 87% to 89.3%, a 2.3-point improvement. Moving from that original mix to the actual current mix takes the result from 89.3% to 81.2%, an 8.1-point decline.
The bridge reconciles: +2.3 − 8.1 = −5.8 points.
This calculation uses a stated sequence: first change rates while holding the old mix, then change the mix at the new rates. Its purpose is to make the competing effects visible. You should not call the 2.3 points the proven effect of AI; other operational changes may also have improved the category rates.
Quick Exhibit Drill
A gym chain began buying paid social-media leads to fill new sites. Trial-to-membership conversion fell, and the board asks whether sales staff got worse.
- Overall conversion
- 39% in month 1, 32% in month 2
| Lead source | Month 1 share of leads | Month 2 share of leads | Month 2 conversion |
|---|---|---|---|
| Referral | 80% | 40% | 50% |
| Paid social | 20% | 60% | 20% |
At month-2 conversion rates, how many points did the shift in lead mix cost?
Look at where unresolved work now sits
Month one leaves 90 routine and 40 complex enquiries unresolved, or 130 in total. Month two leaves 48 routine and 140 complex, or 188.
Complex unresolved enquiries increase by 100, while routine unresolved enquiries fall by 42. Together those changes create the extra 58 unresolved enquiries.
About three-quarters of current unresolved work is now complex: 140 out of 188. That is where I would investigate the next improvement, while preserving the gains already achieved on routine work.
The headline alone might send you towards a full rollback. The segmented exhibit points towards complex-work capacity, routing or resolution instead.
Quantify the improvement needed
Suppose the director's immediate goal is to restore 87% overall resolution at the current 600/400 mix. That requires 870 resolved enquiries.
If routine resolution stays at 92%, it contributes 552. Complex enquiries must therefore contribute 870 − 552 = 318 resolutions. Out of 400 complex enquiries, that requires 79.5% resolution, up from 65%.
This is a 14.5-percentage-point improvement in the complex category. It gives the team a concrete target to assess. A proposed change that adds only five points there would produce 20 extra resolutions and move the overall rate from 81.2% to 83.2%, still short of 87%.
Ask what changed in the work reaching the service
I would ask for the source and routing of the additional complex enquiries. Did a billing event create more difficult cases? Did the AI resolve simple enquiries before they reached this measured queue? Did classification rules change? Did repeat contacts increase?
Each answer points to a different action. If simple enquiries are now resolved upstream, the queue's mix can look harder while the whole customer journey improves. If a product defect created new complex demand, the fix may sit outside the support team.
Keep the service boundary consistent. Measure the entire customer journey when assessing the rollout, then use queue-level data to manage the workload within it.
Give the director a decision
A useful response is:
“I would not reverse the rollout on this headline. Resolution improved within both categories, but complex enquiries expanded from 10% to 40% of the workload and now account for most unresolved cases. I would investigate that shift and test improvements to complex resolution; restoring the old total rate at today's mix requires complex resolution of about 79.5% if routine performance stays unchanged.”
Try the EV sales-funnel exhibit drill for another segmentation problem. Compare the groups before deciding whether the same issue is happening everywhere.
Exhibit analysis drill
What would an MBB interviewer make of your chart analysis?
EV Sales Funnel: Is the Same Problem Happening Everywhere?
Read a business exhibit and explain your conclusion aloud. Get feedback on your interpretation, supporting figures and business implication.
Frequently asked questions
Why the headline fell while both groups improved
How can overall resolution fall when both enquiry types improved?
The overall rate is a weighted average, and the weights moved. Routine resolution rose from 90% to 92% and complex resolution from 60% to 65%. Complex enquiries grew from 100 to 400 of the 1,000. That larger share of the harder category pulls the total from 87% down to 81.2%, a fall of 5.8 percentage points.
Is this Simpson's paradox?
It follows the same pattern: every subgroup improves while the aggregate worsens because the mix changed. Naming it can help. In an interview, though, the convincing step is to rebuild the 870 and 812 resolved enquiries from the categories. That shows the effect directly rather than relying on the label.
Why do I need the enquiry counts as well as the resolution rates?
The rates alone show only that each category improved. The counts reveal that routine volume fell from 900 to 600 while complex volume rose from 100 to 400. Without the counts, you cannot rebuild the headline. You also cannot see that the service received a different workload rather than performing worse on comparable work.
What should I confirm before comparing the two months?
Confirm three things: resolution uses the same definition in both months, it uses the same service window, and the categories cover every enquiry without overlap. The exhibit states all three. If the definition had changed, or cases could sit in both categories, the comparison and any mix bridge built on it would be unreliable.
Splitting performance from workload mix
How much of the 5.8-point fall comes from mix rather than performance?
Apply month-two rates to the month-one mix: 90% × 92% + 10% × 65% gives 89.3%. Better performance alone would have lifted resolution by 2.3 points from 87%. Moving to the actual mix then takes it down to 81.2%, an 8.1-point decline. The bridge reconciles: +2.3 minus 8.1 equals −5.8.
Does it matter which mix I hold constant in the bridge?
It can change the size of each piece, so state your choice. The worked bridge first applies the new rates at the original mix and then moves to the current mix. Holding a different mix constant could split the 5.8 points differently. Any valid bridge should still reconcile to the total change.
Where are the extra unresolved enquiries coming from?
Almost entirely from complex work. Unresolved enquiries rise from 130 to 188, an extra 58. Complex unresolved cases increase by 100, while routine unresolved cases fall by 42. Counting unresolved cases this way shows the operational problem directly instead of leaving it hidden inside an average.
Should the team prioritise routine or complex enquiries next?
Prioritise complex enquiries, because that is where the remaining problem sits. In month two, 140 of the 188 unresolved enquiries are complex, about three-quarters. Routine unresolved cases have already fallen. Further routine gains would still help, but investigation and improvement effort should start with the complex queue.
What complex resolution rate would bring overall resolution back to 87%?
About 79.5%, up from 65%, if routine resolution stays at 92% and the mix stays at 600 routine and 400 complex. Restoring 87% needs 870 resolutions. Routine supplies 552, so complex must supply 318 of its 400 enquiries. That is a 14.5-point improvement and gives the director a concrete target.
What if routine resolution slips while we work on complex cases?
Then the complex target rises above 79.5%. The calculation assumes routine holds at 92% and contributes 552 resolutions, so any routine shortfall must be made up by complex cases. When you quote the target, state the routine assumption and the current mix alongside it. The director then knows what the target depends on.
Investigating the shift and advising the director
Could the AI rollout make the measured queue look worse while customers are better served?
Yes. If the AI now resolves simple enquiries before they reach this measured queue, the remaining mix looks harder. The wider customer experience may still have improved. The queue's headline rate alone therefore cannot judge the rollout. You need to know where enquiries are being resolved before they are counted.
What should I ask about the additional complex enquiries?
Ask where they came from and how they were routed. A billing event may have created more difficult cases, classification rules may have changed, or repeat contacts may have increased. Each explanation points to a different action, such as fixing a billing issue, auditing categories or tackling the causes of repeat contact.
Should the director reverse the AI rollout?
Not on the evidence of this headline. Resolution improved within both categories, so a reversal would respond to a mix shift rather than a performance problem. A better next step is to investigate why complex enquiries grew from 10% to 40% of the workload and to test ways of improving complex resolution.
What does a strong final answer to the director include?
It includes a clear verdict, the reason behind it and a measurable next step. Say you would not reverse the rollout on this headline. Explain that both categories improved while complex work expanded and now holds most unresolved cases. Then give the target: about 79.5% complex resolution restores the old total at today's mix, if routine performance holds.