AI Is Exposing How Much Engineering Was Never About Coding

There is a version of a production system that exists in the source code, and a version that exists in the field. The source code captures what was written. It does not capture why certain decisions were made, what failed before those decisions, which hardware behaviours are being compensated for, or which edge cases were found through incidents rather than design. That knowledge lives in the engineers who built the system, in documentation that may or may not exist, and occasionally in comments that nobody fully remembers writing.

When writing code was slow, this gap was partially obscured. Implementation consumed enough time and attention that the surrounding work – understanding the system, tracing dependencies, validating behaviour – was bundled into the same effort. The code and the understanding arrived together.

AI tools are pulling these apart. Code that used to take days can appear in hours. The understanding still accumulates the same way it always did: slowly, through exposure to the system operating and breaking in real conditions. That gap has always existed. It just becomes a lot more visible when one side of it accelerates, and the other does not.

At PerformaCode, working through where AI tooling fits (in our own practice and in the production systems we build with clients) keeps surfacing the same uncomfortable realization: the things that actually made the engineering work were never labelled as engineering. They were just part of doing the job well, invisible precisely because nobody was skipping them. Looking back at the kinds of systems we have worked on, the places where AI-assisted development would introduce the most risk are not where the code is hardest to write; they are where the code is easiest to write, and the context behind it is hardest to recover.

What the Protocol Does Not Say: Medical Device Interoperability

In one project, we were adding SDC protocol support to a client’s medical devices so they could interact with surgical equipment, imaging systems, and clinical software from other vendors in an integrated OR environment. On paper, this looks like the ideal case for AI-assisted implementation –  there is a standard, the parameters are specified, and conformance requirements are documented. Write the code, pass the tests, ship it.

Then it meets the actual OR ecosystem.

Older imaging systems send status updates at intervals the specification permits, but newer devices don’t expect. One vendor’s device interprets a specific command sequence as a mode transition; another treats the same sequence as an error and goes silent. A device certified against an earlier version of the standard handles one message type in a deprecated way that nobody ever enforced, and if we drop support for that behaviour, we lose interoperability with a portion of the installed base we can no longer upgrade. None of this comes from the specification. It comes from integration testing records, from engineers who have worked across device generations, and from connecting real hardware and watching what breaks.

When the client’s team later extends the implementation with AI tooling, the specification is still there. That accumulated knowledge is not, and a conformant implementation is not the same thing as an interoperable one.

The Algorithm Is Not the Product: Cardiac Imaging

One client came to us with a working algorithm – processing optical signals into a real-time cardiac image. That was their IP, their research, their starting point. What they needed was a product a surgeon could actually use in an OR.

Writing code was a small part of what followed. The acquisition hardware runs fast enough that any slack in the pipeline shows up in the image. Keeping that pipeline stable on hospital hardware, under real OR network conditions, with clinical staff who needed to trust what they were seeing – that took a different kind of work. We ran it with surgeons in realistic conditions, watched where it broke, and iterated on things that had nothing to do with the algorithm: timing behaviour, error recovery, the gap between what the system did and what a clinician could interpret under pressure.

Validation took months. When the client’s team uses AI tooling to extend the processing pipeline later, generating code that compiles and fits the architecture is straightforward. Knowing which timing assumptions the code has to preserve is not in any document we handed over.

The Code That Remembers: Firmware Modernization

Firmware modernization is one of the most common things we work on. From the outside, it looks like one of the simplest jobs, too: old code, accumulated debt, clean it up. Exactly the kind of task where AI tooling seems like a natural fit.

The problem is that old firmware carries two very different things that look identical in the code. One is actual debt: inconsistent naming, functions that grew beyond their original scope, logic that nobody would write that way today. The other is constraint: a driver that polls a register with an explicit delay, a boot sequence with a sleep between two seemingly unrelated operations, and buffer sizes that don’t correspond to anything obvious. Debt and constraint look identical in the repository, and the temptation to clean up both is exactly where the risk lives.

That polling delay compensates for behaviour in a specific peripheral revision that shipped in the first production run and is still in the field. The boot sleep preserves a power stabilization window we found after initial release – the hardware needed it, the datasheet never mentioned it. The buffer sizes match the DMA transfer characteristics of the original silicon, which behaves differently from what we use for development today. Nobody wrote any of this down separately. It lived in the heads of the engineers who discovered it, and then it moved into the code.

An AI tool refactoring this codebase will do exactly what a new engineer does: identify the delay, the sleep, the odd buffer sizes, and remove them. The code gets cleaner. The build passes. Development hardware tests pass. The failures show up months later, in field devices running peripheral revisions that were never in the test environment: intermittent, hard to reproduce, and expensive to trace back.

When the System Is the Integration: Banking Platforms

Banking-as-a-Service work is a different kind of engineering problem. The code itself is often the straightforward part – APIs, transaction logic, data models, all well-understood patterns. Their real behaviour was written by incidents, not architects.

We worked on a platform connecting fintech products to banking infrastructure. The API specifications were there. The onboarding documentation was there. And yet in testing, transactions that should have gone through got rejected. We traced it back to a validation rule a partner bank introduced after a fraud incident. Not in any current specification, enforced differently depending on which legacy onboarding flow the transaction came through, and known only to the engineers who were there when it happened. Everyone who joined after inherited a system that behaved a certain way without knowing why.

That’s what these platforms accumulate over time: not technical debt in the usual sense, but behavioural sediment; rules that came from incidents, edge cases hardcoded after something broke in production, compliance requirements that shifted and got absorbed into the system without ever being formally documented. When the team starts using AI tooling to extend or refactor it, they’re working from the specification and the codebase.

The Part That Doesn’t Accelerate

Every system we’ve described here accumulated something that isn’t visible in the code: decisions made after failures, constraints inherited from hardware revisions, and behavioural rules written by incidents rather than design. Production systems carry two kinds of complexity that look identical in the repository. One is accidental – time pressure, decisions that made sense in a moment that has passed. We can remove it safely. The other is load-bearing: it exists because of a hardware behaviour, a field failure, a partner’s undocumented edge case. Remove it, and things break, sometimes immediately, sometimes months later, in a device variant that was never in the test environment. The instinct to simplify, whether it’s done by a new engineer or an AI tool, doesn’t distinguish between them.

When writing code is no longer the slow part, what becomes visible is everything that was always there alongside it: the system knowledge, the constraint history, the understanding of what the code is actually doing inside the device or platform or OR environment it runs in. That’s the part that doesn’t accelerate. And in production systems, it’s usually the part that determines whether a change is safe to make.