Measurement guide
How to measure AI adoption without kidding yourself
Active use is worth knowing. It is also where the useful questions start, not where they end.
Monthly active users answers one real question: did people open the tool this month? It can expose access problems, unused licences, and teams that need help. The trouble starts when someone puts that number on a success slide and calls the job done.
Someone can open Copilot every day and get very little from it. Someone else might use it twice a week and save hours on one difficult task. Your numbers need to show that difference.
Use five simple questions
Access
Who has the right license, permissions, device, data, and support?
Use
Who used AI, how often, and for which work?
Task change
Which steps became faster, easier, new, or more difficult?
Outcome
Did quality, time, capacity, experience, service, or resilience change?
Value choice
What did the team do with the gain?
Measure one workflow at a time
Say 120 service employees get Copilot and 90 use it in the first month. That gives you a neat 75% active-use rate. It still tells you nothing about the service customers received.
Pick one repeated task, such as preparing an answer after a customer call. Measure 20 examples before the change and 20 after. Record preparation time, checking, corrections, repeat contacts, and how much effort it took. Now the usage number has some actual context.
Ask which step changed, how many minutes it took before and after, what checking was added, and what happened to the time.
Do not hide behind the average
An average hides who gained most and who struggled. Split the result by role, experience, task, and starting skill when you can do that responsibly and without exposing people.
The NBER customer-support study followed 5,179 agents. The average gain was 14%, but newer and lower-skilled workers gained more. That difference is more useful for your plan than the average on its own.
Keep the monthly review to one page
- Access: 120 licensed, 116 able to use the tool
- Use: 90 active, 54 used it for the selected service task
- Task change: median preparation time fell from 18 to 12 minutes
- Quality: corrections stayed at 6%
- Service: repeat contacts moved from 14% to 12%
- Experience: 9 of 15 interviewed employees want to keep the method
- Decision: change the checking guide, then test with a second team
This tells you far more than “75% adoption.” It also gives you a sensible next step, which is handy.