Measuring What Matters: ROI, ROAI, and Business Value in the Agentic Era

By Michelle Lancaster, General Manager and Partner, Global Leader for Business Strategy, AI Business Solutions at Microsoft

Most organizations measure the success of agentic AI the wrong way. They count agents deployed. They track cost savings. They look at efficiency gains.

Those numbers matter. But they’re not the whole story.

Organizations that derive real value from agents are measuring different things. They’re looking at sustained usage. They’re tracking employee satisfaction alongside productivity. They’re measuring improvements in customer experience, not just cost reductions.

And they’re asking a question that many organizations skip: are we building and transforming, or just consuming?

The Agent 365 Dashboard

Here’s what should be on your executive dashboard.

First, how many agents do you have? Not just how many you’ve built, but how many are you actually using? There’s a difference.

Second, what’s your sustained weekly active usage? This is a three-dimensional view: agents deployed, week-over-week usage trends, and whether that usage is sustained over time.

Third, are you building and transforming, or just consuming? Both matter, but they tell different stories about your organization’s maturity.

If you have 100 agents deployed but only 10 are being used, you have agent sprawl. If usage spikes for a week and then drops off, you haven’t solved a real problem. If everyone is consuming pre-built agents, but no one is creating new ones, you’re not building organizational capability.

The three-dimensional view matters: Deployed agents, sustained weekly active usage, and creation activity. Track all three.

Beyond Cost Savings

Many organizations focus on cost savings and efficiency. That makes sense. Dollars are easy to measure.

But business value appears elsewhere.

Take Microsoft’s customer service example. We have the same number of humans processing twice as many cases. That’s measurable efficiency.

But the impact didn’t stop there. Customer satisfaction numbers went up. Customers get help faster, issues are resolved faster. When customers talk to a service person, they get better answers. That’s customer experience-led value that leads to retention and follow-on revenue.

And employee satisfaction improved. People aren’t doing rote work anymore. They’re solving real problems. That’s a capcity for innovation that’s difficult to quantify but critical for long-term success.

When you only measure cost savings, you miss those dimensions.

Personal Productivity vs. Business Transformation

There are different ways people think about value.

Some focus on personal productivity. Can I do my job faster? Can I prepare for meetings more efficiently? Can I get better insights without spending hours in spreadsheets?

Others focus on business transformation. Can we serve more customers without hiring more people? Can we enter new markets faster? Can we innovate in ways we couldn’t before?

Both matter. But they require different measurement approaches.

Personal productivity shows up in individual performance metrics. Meeting prep time. Time to complete tasks. Quality of outputs.

Business transformation shows up in organizational metrics. Revenue per employee. Time to market. Customer acquisition costs. Innovation velocity.

If you’re only measuring one, you’re missing half of the picture.

The First 90 Days vs. Six Months vs. One Year

What you measure should change over time.

In the first 90 days, focus on adoption. Are people using what you’ve deployed? Is usage growing or flat? Are the right people using the right agents?

At six months, shift to sustained usage and business value. Are people still using the agents, or did the novelty wear off? Are you seeing measurable business outcomes? Are people building on top of what exists?

In one year, look at the transformation. Has the way work gets done actually changed? Can you do things you couldn’t do before? Have you built organizational capabilities that will serve you as AI continues to evolve?

Different timeframes require different metrics because you’re looking for different signals.

The Innovation Question

Innovation is one of the hardest things to measure. But it’s one of the most important value dimensions.

When agents handle repetitive work, employees regain capacity. What do they do with that capacity? Are they solving harder problems? Are they coming up with new ideas? Are they focusing on work that creates a competitive advantage?

At Microsoft, we’ve seen employees who were stuck answering the same questions repeatedly now focusing on complex problem-solving. That’s innovation capacity.

We’ve also seen teams use agents to test ideas faster. Instead of spending weeks gathering data and building presentations, they can test and iterate in days. That’s innovation velocity.

Both matter. Both are difficult to measure directly. But you can measure proxies: time spent on creative work, the number of new initiatives launched, and the speed of decision-making.

Quantitative vs. Qualitative Indicators

The best measurement frameworks balance both.

Quantitative metrics give you hard data. Agents deployed. Usage rates. Cost savings. Time saved. Revenue impact. These are essential.

But qualitative indicators tell you whether transformation is actually happening. Employee sentiment. Customer satisfaction. Quality of work outputs. Capability building.

You need both. If your quantitative metrics look great, but your qualitative indicators are weak, you haven’t actually transformed anything. You’ve just automated tasks.

If your qualitative indicators are strong but your quantitative metrics are weak, you’re building capability but not delivering value. Yet.

The organizations that succeed balance both. They track the numbers and pay attention to signals that aren’t easily quantified.

Future-Proofing Your Measurements

AI is evolving rapidly. The agents you deploy today will be different from the agents you deploy next year.

How do you measure success in a way that’s meaningful, regardless of how the technology evolves?

Focus on organizational capabilities, not just technology deployment.

Security posture. Can you deploy AI safely, no matter how it evolves?

Governance structures. Do you know what AI is doing in your organization?

Adoption muscle. Can you get people to use new tools quickly?

Innovation capacity. Can you identify problems and build solutions fast?

These capabilities matter whether you’re using today’s agents or next year’s more advanced models.

The AI Transformation Office

We’re starting to see organizations build centralized functions to manage AI transformation. Think of it as an HR function for AI.

These teams stay on top of trends. They interpret what’s coming for the specific organization. They figure out change management approaches that can be applied company wide.

Having that centralized construct helps organizations stay agile. Line-of-business managers can focus on their numbers while someone else figures out how to pace agent deployments and integrate new capabilities.

Microsoft and many customers have created AI Transformation Offices for exactly this reason. You need someone who is consistently thinking about how to measure success as the technology evolves.

What Gets Measured Gets Managed

This is true for agentic AI just like it’s true for everything else in business.

If you only measure agent deployment, you’ll get many agents that no one uses.

If you only measure cost savings, you’ll automate everything possible without thinking about employee or customer experience.

If you only measure usage, you’ll miss whether that usage is driving business value.

The organizations that get this right measure multiple dimensions. They balance quantitative and qualitative. They adjust what they measure as their initiatives mature. They focus on building capabilities that will serve them regardless of how rapidly the technology evolves.

Because in the end, success isn’t about how many agents you have. It’s about whether those agents help you serve customers better, enable your employees to do more valuable work, and build capabilities that create competitive advantages.

That’s what matters. That’s what you should measure.

Michelle Lancaster is General Manager and Partner, Global Leader for Business Strategy, AI Business Solutions at Microsoft, where she helps organizations navigate digital transformation and AI adoption across industries.

For more information please visit https://www.microsoft.com/en-us/power-platform