?According To What

Evidence checklist

How to read an AI productivity claim

A percentage can be accurate and still be almost useless for your team.

By LisandroSeptember 20266 minute read

AI case studies love a large number: 30% faster, 14% more productive, 26 minutes saved a day. Before you copy it into a slide, check what they actually counted.

You do not have to dismiss every vendor study. You do have to keep the method and context attached, even when that makes the headline less exciting.

Ask six questions

  1. What exactly was measured?

    Look for the task, unit, and time period. “Productivity increased” tells you almost nothing. Issues resolved per hour is clearer. So is the median time needed for one client briefing.

  2. Compared with what?

    Was there a baseline, control group, earlier period, or did people simply estimate it? Even a before-and-after result can be affected by workload, practice, or easier tasks.

  3. Who took part?

    Check the job, organization, experience, sample size, and whether participation was voluntary. A customer-support result does not automatically apply to policy work.

  4. How was the number collected?

    System records, observed tasks, reviewer scores, and surveys answer different questions. Self-reported time is someone’s estimate. Treat it that way.

  5. Who gained and who did not?

    An average can hide large differences. Look for results by experience, role, task, and starting performance.

  6. What did the headline leave out?

    Check errors, checking time, failed tasks, privacy, workload, employee experience, and what happened to saved time.

Apply the checklist

The NBER study of 5,179 customer-support agents found a 14% average increase in issues resolved per hour. Newer and lower-skilled workers gained most. That suggests AI may spread useful working methods in this kind of service team. It does not promise every office worker a 14% boost.

The UK government’s Microsoft 365 Copilot experiment reported an average self-reported saving of 26 minutes a day. That tells us how participants experienced the tool at scale. It was not a stopwatch test of every task, and it does not prove public services improved.

Keep the boring but important details

Participants reported saving time. The amount varied, and the central figure was self-reported. Teams should measure task and service outcomes before turning it into a local business claim.

A note you can copy into your next presentation

  • Claim:
  • Original source:
  • Population and task:
  • Measure and comparison:
  • Important variation:
  • Known limitation:
  • What we can test locally:

Sources