Microsoft just put AI agents to work inside the software that runs the world’s finance departments. The tool most companies use to test that software was built for a world that no longer exists — and closing that gap is quietly becoming one of the more interesting problems in enterprise tech.
There’s a quiet little drama playing out inside big companies right now, and almost nobody outside of enterprise IT is watching it. It has all the makings of a good tech story: a decades-old tool, a shiny new kind of AI, and a gap between them that could cost somebody a lot of money.
The software in question is ERP — enterprise resource planning. If that sounds boring, sit with it for a second, because ERP is the software that literally runs companies: the payroll, the invoices, the inventory, the numbers that end up in the annual report. Microsoft’s version is Dynamics 365, used by thousands of organizations to keep the financial lights on. When ERP breaks, people don’t get paid and vendors don’t get their money — so companies test it obsessively before every change.
For years, the go-to tool for that testing job at Microsoft shops has been something called RSAT. And in 2026, Microsoft did something that quietly made RSAT’s job a lot harder: it shipped AI agents that work inside the ERP itself. To see why that’s a problem, you need to know what RSAT actually is.
What RSAT is, in plain English
RSAT stands for Regression Suite Automation Tool. It’s a free tool from Microsoft, and its whole reason for existing is to save humans from the mind-numbing job of re-testing the same processes over and over every time the software updates.
The way it works is beautifully simple. A person clicks through a business process once — say, creating and posting an invoice — while RSAT records every click, like a macro. From then on, it can replay those clicks automatically, as many times as you want, to confirm the process still works after an update. Record the human, replay the human. For straightforward testing, that’s genuinely useful, and it still is.
But notice the assumption baked into that design: there’s a human clicking through a fixed sequence of screens, and the software just responds. RSAT tests software the way you’d test a vending machine — press these buttons in this order, check the right snack comes out. That held for thirty years. It also has well-known limits even for ordinary testing: the recordings tend to break whenever the screens change, which is a big part of why many teams have started looking at modern alternatives to RSAT. Then the agents arrived, and the problem got a whole lot bigger.
The agents don’t click buttons
In its 2026 releases, Microsoft rolled out a wave of AI agents that live inside Dynamics 365 and do real work on their own. There’s an agent that reconciles the books — matching thousands of transactions between different ledgers. There’s one that reads incoming vendor invoices, pulls out the details, and drafts them for approval. Others handle sales orders and chase suppliers about deliveries.
These aren’t chatbots that answer questions. They’re software workers that watch for something to do, decide what to do about it, and do it. And crucially, they don’t interact with the software the way a person does. They don’t open the invoice screen and click through fields in order. They reason about a goal and act on the data directly.
| You built a tool to replay a human clicking buttons. The new worker doesn’t use buttons, doesn’t click, and doesn’t do it the same way twice. |
This is where RSAT hits a wall it can’t climb. Its entire method — record a human’s clicks, replay them — assumes there’s a fixed path of clicks to record. An agent has no fixed path. Ask it to handle the same situation twice and it might reason its way there differently each time, because that flexibility is the whole point of using AI instead of a rigid script. There’s simply nothing for a click-recorder to record.
Testing the decision, not the clicks
So if you can’t test an AI agent by replaying clicks, how do you test it? The answer needs a real shift in what “testing” even means here — and it’s the interesting part of this whole story.
Old-style testing asked: did the software follow the steps? The new question is: did the agent reach the right result? When an agent reconciles the books, you don’t care which path it took — you care whether it matched the correct transactions and didn’t quietly paper over a real discrepancy. When an agent drafts an invoice, you don’t care what it clicked — you care whether it read the amount correctly and coded it to the right account. The screen is irrelevant. The decision is everything.
This is exactly the shift a newer generation of testing tools is built around. Companies like Sofy, which makes AI-powered testing agents for Dynamics 365, don’t try to replay clicks at all. They check the outcome an agent produced — the actual numbers in the data — against what it should have been, and they adapt automatically when the screens change instead of breaking like a recorded script. It’s a different philosophy: testing autonomous ERP agents by validating their decisions rather than their button presses. Which turns out to matter a lot, because of how these agents fail.
An AI agent doesn’t fail like a human. A human data-entry clerk makes random mistakes — a typo here, a wrong number there. An AI agent makes systematic ones. If it misreads a particular vendor’s invoice layout, it’ll misread that layout the same way every single time, across hundreds of invoices, all looking perfectly normal. That’s the kind of error a person skimming an approval queue will never catch — and exactly the kind that checking the outcome is designed to catch.
| An agent that’s wrong isn’t wrong once. It’s wrong the same way, at scale, with a confident smile on its face. |
Why this is everybody’s problem soon
It’s easy to file this under “boring enterprise IT,” but the pattern is about to show up everywhere, and ERP is just the first place it’s getting real. Across the software world, we’re handing AI agents the keys to systems that used to require a human hand on every action. Coding agents write and ship code. Support agents resolve tickets. Finance agents move money. Each time, the same gap opens: the tools we built to check the old, human-driven way of doing things weren’t designed to check an autonomous agent’s decisions.
And the stakes climb with the autonomy. A wrong answer from a chatbot is annoying. A wrong decision from an agent wired into your general ledger is a financial misstatement that might sit there, quietly, for months before anyone notices. The organizations that adopt these agents fastest and most safely won’t be the ones that trust them the most. They’ll be the ones that figured out how to prove the agents are right.
RSAT isn’t going away — for the ordinary, human-driven parts of ERP, replaying clicks still works fine. But for the agents Microsoft just shipped, “beyond RSAT” isn’t a slogan; it’s a description of where testing has to go next: from checking the clicks to checking the decision. If you want to go deeper on how that actually works in practice, Sofy’s team writes about AI-agent testing for enterprise ERP in more detail. The short version: the software started thinking for itself, and our way of testing it has to catch up. Which is honestly one of the more interesting problems in tech right now — even if it’s hiding inside the least glamorous software on earth.
