AI Agents Push SaaS into Embedded Engineering Territory

For two decades, enterprise software optimized for one thing: humans using it. That was a reasonable priority. Humans were the operators. They navigated the interfaces, read the outputs, and decided what to do next.

What nobody fully accounted for is how much work those humans were doing that wasn’t navigation.

SaaS always assumed a human would catch the gaps

One thing that doesn’t appear in any SaaS architecture diagram: the user noticing that something looks wrong.

The ops person who sees a workflow marked “approved” and knows the approval came through while the budget owner was out sick, and the backup approver almost certainly didn’t read the scope document. The account manager, who notices the contract status says “renewed,” but recalls a conversation from three weeks ago suggesting the situation is more complicated than that. The finance analyst sees a technically correct number but flags it anyway because the source system has been unreliable lately.

None of that is encoded anywhere. There’s no field for “contextually suspicious.” No workflow state for “technically complete but someone should check.” These people are running informal validation against the system continuously, every time they use it, and the system was built knowing they would.

For most purposes, in most systems, an inconsistent state gets noticed before it causes real damage. Stale records got corrected before anyone acted on them badly. The architecture didn’t need to be rigorous because the people using it were compensating for the parts that weren’t.

AI agents remove that compensation.

I agents operate on whatever state the system holds

An agent operating a SaaS system on behalf of a user consumes whatever data is in the system and treats it as current. There’s no built-in skepticism about whether a record reflects reality, whether a field was last updated six months ago, or whether the context behind a piece of data ever made it into the system at all. A lot of that context never did – it stayed in conversations, in people’s heads, in the informal knowledge of whoever was handling the account. The agent has none of it, and nothing in the system flags that it’s missing.

This isn’t an AI problem. The agent isn’t hallucinating. The problem is that the system state was never rigorous enough to operate without a human interpreting it, and nobody noticed because a human was always there to interpret it.

Take approval routing. The agent is supposed to route a request to the right approver, wait for a response, and proceed. Routing logic reads a role field. Role field is correct – it was correct when someone set it up. The person in the role left five weeks ago. HR system reflects the departure. Approval configuration doesn’t, because that’s a different system, and the offboarding checklist doesn’t cover it. Agent routes to a stale entry. Notification goes nowhere. Request either hangs indefinitely or hits a timeout and auto-approves, depending on how that edge case was handled – and in a lot of systems, it wasn’t handled, because a person would normally have noticed the approver wasn’t responding.

The agent executed correctly against the available state. The available state was wrong in a way that had always been wrong, and a human had always corrected for it before it caused a problem. There was no human this time.

The same pattern shows up across any SaaS system where agents are being pointed at operational data. An ITSM agent is routing incidents to the wrong team because on-call assignments weren’t updated after a reorg. An ERP agent triggered a purchase order against a supplier whose certification had lapsed, and nobody updated the vendor record. A contract management agent flagging a renewal based on terms that were superseded by an amendment sitting in someone’s email. In each case, the agent ran correctly. The data it ran against was stale, incomplete, or inconsistent in ways a person would have caught.

Embedded engineering never had that assumption.

Systems engineers have been solving this class of problem for a long time, under a different name and with higher stakes.

Embedded controllers, safety systems, industrial automation – this software was never designed around a human operator in the loop. No UI, no dashboard, no person checking whether the output looks reasonable before anything acts on it. Correctness requirements had to be in the software itself because there was nowhere else to put them.

A state machine in an embedded system has defined states and defined transitions. Invalid transitions can’t be reached or are explicitly handled. Not because the engineer felt like specifying it – because leaving it unspecified means the system can get into a state nobody anticipated, with no human available to notice and intervene. Actuator execution – actuator A fired, actuator B didn’t – requires recovery logic. Stale sensor data needs validity windows, not a person who might notice the reading looks old. The audit trail isn’t just a log of actions; it’s a record of inputs, because when something goes wrong in a regulated context, you need to reconstruct why the system did what it did, not just what it did.

It’s standard practice in any domain where the software has to behave correctly without a human watching it closely. SaaS now enters that domain, at least partially, for any setup where agents are doing real work. The human oversight layer is being removed. The correctness requirements that the layer was satisfying now have to be in the software itself.

Most SaaS systems weren’t designed for that.

SaaS now needs the same guarantees embedded systems require

Most SaaS state models are implicit. There’s a status field. There’s code that sets it. What counts as a valid transition, what preconditions are required, what happens on an invalid input – that logic is scattered across handlers, background jobs, and frontend validation, often inconsistently, and nowhere enforced as a single defined model.

When a human operates that system, the implicit state is manageable. They bring context that fills the gaps. When an AI agent operates it, the implicit state is a liability. The agent can’t fill gaps with context it doesn’t have. It acts on the state as it exists, and if the state is wrong, ambiguous or stale, the action will reflect that.

Closing this gap requires the same properties that embedded systems engineers treat as baseline requirements. Data consistency that is guaranteed, not assumed – field values with provenance, timestamps, and validity signals rather than just a current value with no indication of when it was last verified or by what process. State models that are explicit and enforced – defined valid transitions, preconditions that are checked, and behaviour specified for invalid inputs rather than left as undefined. Failure handling that doesn’t leave systems in intermediate states – multi-step agent operations need compensation logic or transactional boundaries, not an assumption that someone will notice if step four completes and step five doesn’t. And audit trails that capture inputs and decision context, not just the resulting action – because when an agent does something unexpected, “the agent did it” is not a useful entry in a log.

For embedded and systems work, this is the starting point. For SaaS, it’s a significant retrofit against systems that have been in production for years, with implicit state models that nobody fully documented because the users were handling the ambiguous cases anyway.