Mihály Tari.

Why most agent demos don't survive a real codebase, and what it takes to get one there

In the demo, the agent answers a question. In production it also needs a gateway, a deploy pipeline and an auth service nobody planned for, and those usually decide whether an AI feature survives.

I’ve watched a lot of agent demos, and most of them work well. An agent is an AI program that calls other systems on its own to answer you, for example by querying a database, and a typical demo shows a chat box, a couple of those calls and a clean answer on the first try. The demo is the part everyone remembers, although it’s the easier part of the project.

The harder part starts when the agent has to go into a real company’s systems. The model doesn’t get worse at that point. What changes is that a demo runs in an environment its builder fully controls, while production has network boundaries, other teams’ tools and real users. Below are three obstacles one of my agent projects ran into before it went live. None of the three had anything to do with the agent itself, and each of them decided whether it would ship.

The agent couldn’t reach the data

In the demo, everything sat in one place. The agent, the data it queried and the credentials were all on one machine, so everything could reach everything else.

Production split that apart. The agent ran on a server reachable from the internet, while the data it needed sat behind an internal API, the interface other programs use to ask a system for information. That API refuses calls from the public internet by design, so the first time the agent tried the one task the demo had made look trivial, it got nothing back. The model was fine. It simply had no path to the data.

A better prompt, meaning better instructions to the model, couldn’t have fixed that. I had to set up an API gateway, a controlled door between the two networks, and configure it to authenticate the agent’s requests, expose only the few endpoints the agent was allowed to call, and refuse everything else. That’s infrastructure work, and demos never show it, yet without it the agent can’t answer a single question that needs internal data.

Because setting up the gateway was real work, I didn’t keep it to this one agent. I registered a shared organization on the gateway that other teams could route their own services through, and documented the connector that links the two networks, so the next team that needs the same internal systems doesn’t have to work it all out again.

A manual deploy only works once

The first deploy, the step that puts a new version live, was done by hand. That works once. By the third change it’s easy to skip a step, push the wrong configuration, or release a version nobody can reproduce, and from then on you’re debugging the deploy instead of the feature. The fix is to automate it, because a machine runs the same steps in the same order every time.

The company already had a deploy pipeline for that, an automated release process built by another team. It looked like the easy part, but the tool was hard to use and barely documented. Getting the agent onto it meant finding the people who built it and learning from them how it worked, because almost none of it was written down. Most of my work on automating the deploy turned out to be collecting that knowledge.

That kind of work, fitting into the half-documented platform the rest of the company already runs on, never appears in a demo. Inside a real organization, though, it’s most of what “deploy the agent” means.

The interface had to know who was asking

The last obstacle came up when a chat component went into an application’s interface. It’s a box on a screen, the most demo-like piece of the project. A real interface has real users in front of it, though, so the agent behind it can’t answer anyone who types into it. It has to know who is asking and whether they’re allowed to ask.

That meant the chat component had to prove its identity to the agent, and the agent had to trust that proof. For that I built a small auth service, a separate program that handles identifying users in one place. I made it reusable, so any team connecting its own interface to the same agent can use it instead of writing its own. As with the gateway, the model didn’t change, only the system around it grew.

The agent was the easier part

Taken together, none of the three obstacles is AI work. I shared the gateway so other teams could use it, I had to reverse-engineer the internal deploy pipeline before I could use it, and I built the auth service to be reused. All three are ordinary full-stack systems work about networking, deployment and identity. A demo hides this work on purpose, because its job is to show the 20% that’s impressive and leave out the 80% the whole thing rests on.

That’s why so many good agent demos never turn into anything. A demo proves the model can do the task. It says nothing about whether the task survives a network boundary, a second deploy or a real user. The features that reach production get there because someone treated the agent as one component in a larger system and did the less visible work of fitting it in.

If you’re building one of these, it’s worth judging an agent feature by how much of the rest of your system it touches, more than by how clean the demo looked. The demo tests an assumption, and the work around it shows whether that assumption holds in production.

And if you’re hiring someone to ship AI features, this is the job you’re hiring for. A prompt is easy to copy. The system around it has to be built by someone. When it’s built so the next team can use it too, it’s still running a year later.

If you have something to build or fix, send me a few sentences and I'll reply within two business days.

Get in touch

← All writing