Essay 03

Measure agents by outcomes.

Conversation count is easy. Completed work is harder, and much more useful.

Agents make measurement more interesting.

With Copilot, a person often stays clearly in the loop. With agents, the work can move through a conversation, a workflow, a tool call, a handoff, or an escalation. The value can hide in the path.

That means the old “how many people used it?” question gets even weaker.

And yes, I am still excited about agents. Maybe too excited. That is exactly why I want better measurement.

A busy agent can still be a weak agent.

An agent can answer thousands of questions and still fail at the job it was meant to do.

Maybe people keep asking because the first answer is incomplete. Maybe the same question returns again and again because the process behind it is broken. Maybe every useful answer still needs a human to correct it before anything can happen.

High volume can mean demand. It can also mean friction.

I have to remind myself of that too, because a busy agent looks great in a screenshot.

An agent is valuable when it changes the work after the conversation.

Start with the outcome state.

For every agent use case, define the states that matter.

• Resolved by agent
• Resolved after human review
• Escalated to a person
• Abandoned
• Returned because the answer was not good enough
• Blocked by missing data or permissions

This immediately makes the measurement more honest. A session is no longer just a session. It has a result.

That result might be messy. Good. Messy is often where the useful learning is.

Then measure the hidden work.

The common mistake is to count only what the agent did.

But the real cost often sits around the agent: review time, correction time, escalation handling, knowledge maintenance, failed attempts, and support from the team that owns the process.

If the agent saves five minutes for the end user but creates seven minutes of review elsewhere, the value case is not as strong as the demo felt.

Agents punish weak foundations.

This is where boring platform work comes back again.

An agent built on messy knowledge, unclear permissions, missing ownership, or vague policies will not magically become reliable because the interface is friendly.

Data needs ownership. Knowledge needs maintenance. Permissions need care. Policies need to be understandable enough that builders and users can actually follow them.

Otherwise the agent may still work. Sometimes. Until it does something odd in exactly the place where trust matters.

Useful agent metrics

For serious agent measurement, I would start with these:

• completed task rate,
• deflection rate with quality checks,
• escalation rate,
• abandonment rate,
• average review minutes per accepted answer,
• cost per completed outcome,
• repeat contact rate,
• knowledge gap rate.

That is the kind of result I want to read.

It tells me the agent was not just active. It changed the system around it.

Or it tells me the agent is not ready yet. Also useful.