Every application has always needed a place to keep its state. That is not new. What is new is who creates that place, how many get created, and how long they have to survive after everyone has stopped looking.

“Separating durable state from ephemeral compute is the new requirement, and the idle cost trap is only half of it.”

Consider the agents built by Kimi, the AI platform from Moonshot AI. A non-technical user describes an application in plain language. An agent builds the frontend, backend, and database, then deploys them in minutes. The user never thinks about infrastructure. They asked for a tool and received a running one.

Then the agent does something the previous generation of coding assistants never did. It does not hand over code and walk away. It keeps the application running and comes back to maintain it. That last word is where the old assumptions break. Build once and hand off, and state is almost an afterthought. 

Build, deploy, and maintain across tens of millions of applications, most idle most of the time, and persistence stops being a database feature and becomes the central economic problem of the whole system.

I have written before about what happens to a database when its user is an agent rather than a human. Kimi is the clearest production example I have seen, and it surfaces a requirement that the industry has not really been asked to meet: keep state alive cheaply in two very different places at once. The problem is not performance. It is the economics of persistence.

The number that changes the question

When you serve thousands of tenants, the questions are familiar. How fast are queries? How do we fail over? How do we back up? The industry has good answers for all of them. When you serve tens of millions, most created by agents and most idle most of the time, a different question moves to the front, and it is the one nobody planned for: what does it cost to keep something alive that nobody is using right now?

Ask it once, and you are talking about idle databases. Ask it again, one layer up, and you are talking about the agent’s own half-finished work sitting between maintenance sessions. Same question, two surfaces. At a thousand you absorb it. At ten thousand it stings. At tens of millions it is fatal.

“In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly.”

This is the shift agents force. In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly, because agents create far faster and more casually than humans ever did, and most of what they create goes idle immediately.

One principle, two places

There is a single move underneath everything that follows: separate durable state from ephemeral compute. State is the part you must never lose, so it belongs on a shared, cheap, effectively infinite foundation. 

I have argued before that object storage is becoming the new network layer of the data stack, and this is what that looks like, built for agents: durable data at rest costs almost nothing. Compute is the part you should be able to throw away, summoned when work arrives and released when it stops.

Couple the two and you pay to keep compute running for state that is doing nothing. Decouple them, and your cost stops tracking how much you have created and starts tracking how much work is happening. 

Every hard number in the Kimi deployment comes from getting this separation right in the two places an agent-scale product needs it: the databases the agent provisions for its users and the workspace the agent lives in.

The first place: the idle tenant

The instinct most teams start with is the one that has worked for a decade: give each tenant its own database instance. It is clean and easy to reason about. Kimi’s earlier shape was close to this, with single-instance PostgreSQL behind the product. It works beautifully until the tenant count climbs into the tens of thousands, and then the economics invert. The reason is structural. 

A per-tenant instance couples the logical experience of an isolated database to the physical reality of dedicated, always-on compute. The tenant needs the first. The business cannot afford the second at scale. Cost grows with tenants created, not tenants active, and in an agent-driven product those two numbers are worlds apart.

“The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace. The agent does not need a dedicated instance. It needs the experience of one.”

I have started calling this the idle cost trap, because teams walk into it without seeing it, and by the time they feel it, the architecture that created it is load-bearing. The way out is the principle. The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace and isolation guarantees over a shared substrate, with object storage holding durable data and a routing layer sending requests where they belong. The agent does not need a dedicated instance. It needs the experience of one.

There is a catch worth being honest about. If compute is summoned on demand, then when a request arrives for an idle tenant something has to spin up, and if that is slow, you have traded a cost problem for a latency problem. 

Kimi provisions a database in about a second, from a warm pool of pre-initialized resources kept ready and replenished behind each claim, so database setup drops out of the delivery pipeline entirely. That number is not a benchmark to brag about. It is the evidence that ephemeral compute over shared, persistent storage can stand in for the always-on model, because it appears fast enough to feel always-on.

The second place: the agent’s own workspace

The maintenance half of the lifecycle drags the same problem into a place most teams never look: the agent’s own working environment. An agent that builds once can treat its workspace as disposable: spin up a sandbox, write code, ship, discard. An agent that maintains cannot work that way. 

It has to return days or weeks later and pick up where it left off: source code, Git history, checkpoints, the record of what it was in the middle of doing. If that context is lost, the agent does not resume work. It is reconstructing it.

And execution environments are ephemeral on purpose, because keeping millions alive between sessions is the idle cost trap in a different costume. So the environment gets torn down. If the work goes with it, every maintenance session begins by rebuilding state the agent already had: wasted compute, wasted tokens, and a user watching an agent relearn its own project.

The fix is the same separation, one layer up. Kimi uses a persistent filesystem that keeps development state alive independently of the compute that produced it. Source code, Git history, checkpoints, and task progress persist after the environment is destroyed, so the agent resumes rather than starting over. Compute is ephemeral and cheap to discard. State is durable and cheap to keep. Same trade as the idle tenant, seen from the builder’s side.

The database choice becomes a quality input

There is a further effect that surprised me, because it shows up in the quality of what the agent builds. Every infrastructure decision is a chance to get something wrong. If each task forces the agent to reason from scratch about which database to use and how to configure it, it is improvising on every run, and improvisation is where errors enter. 

When the stack is unified, it applies known-good patterns instead. Kimi saw code generation success rates improve from standardizing on a unified data layer, and that is the mechanism: fewer places for the output to go wrong.

“The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.”

This is a version of an argument I keep returning to. Agents need guardrails, and the most effective ones are built into the infrastructure rather than bolted on as instructions. A database that behaves consistently every time is a guardrail. The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.

Four properties that have never been requried together

Step back from Kimi and the shape of the requirement is clear. An agent-scale product needs four things from its data layer at once:

• Tenant isolation: every agent-created database must be logically separate, so millions of tenants never bleed into one another.
• Instant provisioning: a new tenant has to exist in about a second, because an agent will not wait and neither will the user behind it.
• Cost elasticity: an idle tenant, and an idle workspace, has to cost almost nothing, because most things are idle most of the time.
• Persistent state independent of compute: the tenant’s data and the agent’s development state alike must survive the ephemeral environments that produced them.

Each has been solved before in isolation. Databases have offered isolation for decades. Serverless systems provision quickly. Object storage is cheap at rest. What is new is the demand for all four at once, at a scale where any one failing breaks the product. That is the real infrastructure competition of the agent era, and it is not the one the industry is used to having. It is not about who has the fastest single query. It is about who can hold all four together at tens of millions of tenants without one collapsing the others.

The pattern is bigger than one product

I do not think Kimi is a special case. It is an early, unusually clear instance of a pattern that will repeat across every product where agents build for non-technical users at scale. The shape is always the same. One agent, one workspace, one database, repeated millions of times. Each instance feels independent. 

The infrastructure underneath is shared, and it works only because a virtual layer separates the logical experience of isolation from the physical cost of compute, and because durable state is allowed to outlive the compute that produced it.

Kimi is the version where the agent hands a finished application to an end user and then stays on to maintain it, which is harder in two ways at once. The tenant does not go away when the run ends; it persists, idle, waiting, multiplied by tens of millions. And the agent does not go away either, so its own working state has to survive every gap between sessions. Persistence is no longer a property of one component. It is the property the whole system is organized around.

“The database conversation in the agent era is no longer really about speed. Get it wrong, and no amount of model quality will save the margins.”

Teams building in this direction will hit the idle cost trap on both surfaces, whether they plan for it or not. The only choice is whether they see it coming. Design for all four from the start and you scale past the point where the per-tenant-instance model collapses. Do not, and you hit that ceiling at tens of thousands of tenants, exactly where Kimi’s earlier architecture did, and sooner than you expect, because agents fill a tenant table faster than any human-driven product ever has.

The database conversation in the agent era is no longer really about speed. It is about whether the economics of persistence can survive the scale that agents create in both the database and the workspace. Get that right early, and the rest of the product has room to grow. Get it wrong, and no amount of model quality will save the margins.

If you are building a product where agents provision infrastructure for end users and then stay on to maintain it, four things make that economically viable: isolation, instant provisioning, near-zero idle cost, and durable state on a shared substrate. That is what we have built into TiDB’s serverless and agent workloads. It is the pattern these teams keep arriving at from different directions.

The post When agents build, deploy, and maintain, persistence becomes the hard problem appeared first on The New Stack.

Imagine you just finished a two-hour meeting. You were wearing a small AI work companion that promised to capture the conversation, structure it, and let you ask questions about it later. Two hours after the meeting, you open the app and ask for the transcript. You wait. The spinner turns. The whole reason you bought the device was so you’d never have to remember a meeting again, and now the one thing it promised to do well — recall what was said — is the thing that’s making you wait.

This is the failure mode that fascinates me about AI hardware products, because it’s not a model problem. The transcription was perfect. The summarization was good. The thing that frayed was the part nobody markets: getting the right bytes off disk and onto the screen at the moment the user asks. That’s a data problem. And for a product whose entire promise is memory, a data problem is a product problem. 

“For a product whose entire promise is memory, a data problem is a product problem.”

I want to walk through a real version of this because the team that hit it, Plaud, makes the most popular AI notetaker on the market, and the architecture that broke is the architecture almost every team in this category starts with. It feels reasonable at launch. It becomes a liability at scale. And the reasons why are worth understanding before you ship, not after.

First, what was built correctly

It would be easy to tell this story as “they made a mistake.” They didn’t. The original architecture was a sensible set of decisions, and I want to be precise about that before I dissect what went wrong, because the lesson is in the gap between “reasonable” and “right at scale,” not in anyone’s competence.

Plaud’s product generates two very different kinds of data per recording. There’s structured metadata: who recorded it, when, how long, what tags, what state the processing pipeline is in. And there’s the unstructured payload: the audio file and its transcript, which can run tens of megabytes per session. The team did the textbook thing. They put the structured metadata in MySQL, where relational queries and transactions are cheap, and they put the large objects in S3, where storage is cheap and effectively infinite.

If you’ve built systems, you’ve made this exact call. Object storage for blobs, a relational database for everything you need to query. It’s in every architecture diagram. It’s the default. And for a long stretch of the product’s life, it worked fine.

The problem is not that the decision was wrong. The problem is that it contained a hidden assumption: that the metadata and the content could live in separate systems because they would never need to be consistent with each other in real time. For most applications, that assumption holds. For a product whose core interaction is “give me back exactly what I recorded, right now,” it doesn’t. And the gap between those two worlds is where everything started to fail.

The transcript sidecar

In my article on RAG retrieval, I identified an anti-pattern I called the vector sidecar: the habit of standing up a separate vector database alongside your primary store, only to discover that the two systems can’t answer a single query together. The Plaud architecture is the same shape, applied to an AI hardware product. Call it the transcript sidecar.

The transcript sidecar is what you get whenever you split structured metadata from unstructured content across two systems with no shared transaction guarantee. The metadata says a recording exists, is complete, and is ready. The content lives elsewhere, reached via a separate call, with its own latency and failure modes. Nothing ties the two together. There is no transaction boundary that spans “the row that says this transcript is ready” and “the object that contains the transcript.”

This produces three distinct compounding problems.

• Retrieval latency is not a network problem; it’s a data locality problem. S3 is excellent object storage. It is not a database. When you make object storage the primary retrieval path for tens-of-megabytes payloads under concurrent load, you’re trading the consistency and latency guarantees of a database for the simplicity of a blob store. Under light load, the trade is invisible. Under heavy concurrent load, S3 retrieval latency and its variability become the user’s experience of your product.
• Consistency gaps open between two systems that fail independently. The MySQL row can indicate that a recording is ready a moment before the S3 object is durably reachable, or a replica can lag, causing the metadata a user sees and the content they fetch to disagree. With no shared transaction, the application layer inherits the job of papering over the gap, with retries, polling, and reconciliation logic that grows more elaborate every quarter.
• The schema becomes immovable at exactly the wrong time. More on this below, but the short version is that the metadata store hit a scale ceiling where changing the schema required a maintenance window. For a product still evolving its feature set, that’s a database governing the roadmap, rather than the other way around.

The unifying insight is the one I keep coming back to throughout this whole series: any time you split data across systems that can fail independently, you have handed the consistency problem to the application layer, where it is not solved so much as managed, and where it compounds over time.

When the database governs the roadmap

The most interesting failure in the Plaud story isn’t the latency. It’s the schema freeze, because it’s the one that teams least expect and feel most acutely.

With around 300 million rows, the team’s MySQL setup reached a point where schema changes such as adding a column or changing an index — the routine evolution of any growing product — could no longer be performed online without risk. DDL operations on a table that large, on that architecture, meant locking, replication strain, and the real possibility of downtime. So changes had to be batched into maintenance windows.

Sit with what that means. A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations. A feature that needs a new column will wait until the next window. The product’s cadence is set by the operational fragility of the data layer. Most teams don’t notice this inversion because it happens gradually, and then one day a product manager asks why a small change is a two-week project, and the answer is “the database.”

“A maintenance window for a schema change is the time the database allows the product team to ship. The roadmap now bends around the database’s limitations.”

The 300-million-row ceiling is a forcing function that teams hit later than they expect and should hit later than they do. Expect, because single-instance MySQL feels limitless right up until it doesn’t. Should, because the architecture that gets you to 300 million rows is rarely the architecture that takes you past it, and the migration is far more disruptive at 300 million rows than it would have been at 30 million. The ceiling is real; it is predictable, and most teams plan for it only after they’ve hit it.

What changing the architecture actually fixed

Plaud’s resolution was to consolidate the metadata layer into a distributed SQL database (they moved to TiDB), which removed the single-instance ceiling and restored online schema changes. I’m going to use their numbers, but not as a product pitch. I want them as evidence for a claim about where architectural debt goes in a product like this.

Before (dual-store-brain at scale)	After (consolidated metadata layer)
Throughput strained under concurrent load	~10x QPS headroom
Tail latency variable under load	P95 under 10 ms
Schema changes require maintenance windows	Online DDL, no downtime
Single-instance ceiling at hundreds of millions of rows	Horizontal scale across multiple clusters

QPS headroom and tail latency matter, but the first result I’d point to is the online DDL. Restoring the ability to change the schema without a maintenance window is what handed the roadmap back to the product team. The database stopped governing the release cadence. That’s not a performance win you can put on a benchmark chart, but it’s the one the product organization feels every week.

Also notice what the fix did not do: it did not attempt to store 30-megabyte audio files in the database. Large objects can still be stored in object storage. The point was never “put everything in one system for its own sake.” The point was that the metadata layer, the part that has to be consistent, queryable, and evolvable in real time, needed to actually behave like a database at scale, instead of becoming the fragile half of a split-brain architecture.

Where architectural debt goes

Here’s the claim the Plaud numbers are evidence for. When the database is the product, architectural debt does not accumulate in some back-office system that only the platform team feels. It accumulates in the product experience, where every user feels it.

This is the structural difference between an AI note-taker and, say, an internal analytics tool. For a product like this, every meaningful user interaction is a database operation. Recording creates rows and objects. Transcription updates state. Retrieval is a read. Editing is a write. Search is a query. There is no part of the product that isn’t, underneath, the database doing something. Architectural problems surface as product problems, one-for-one. A consistency gap becomes a transcript that is briefly missing. Slow retrieval reads as a product that is slow to remember. A frozen schema means features arrive late.

No user will ever file a bug report that says “your metadata store and your object store lack a shared transaction boundary.” Left unaddressed, this class of debt manifests as a product that feels less dependable precisely when someone depends on it. For a product whose entire value proposition is “trust me to remember,” the debt lands on the promise itself. This is why Plaud’s decision to re-architect when it did matter. The team fixed the data layer before the failure became noticeable to users.

“You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.”

This is why I say the database is the product. Not as a slogan. As a literal description of where the product’s quality is determined. You can have the best transcription model in the category and still ship a product that feels unreliable, because reliability for this kind of product is a data architecture property, not a model property.

The trap is waiting for the whole category

AI hardware is having a moment. Note-takers, wearables, ambient recorders, pendants, badges — each built on the premise that the device will remember so you don’t have to. The category is growing, and the products are getting better at the parts everyone talks about: the models, the form factor, the battery life.

But underneath, almost all of them start with the same architecture Plaud started with, because it’s the reasonable default. Structured metadata in a single-instance relational database. Large content in object storage. No shared transaction boundary. A schema that’s easy to change at ten million rows and frozen at three hundred million. The trap is identical, and it’s waiting at the same place on the growth curve for every team in the category.

The teams that will do well are the ones that recognize this early, understand that, for a memory product, the metadata layer is not back-office plumbing but the spine of the user experience, and plan their data architecture for the scale they’re trying to reach rather than the scale they’re at. Plaud’s migration is the pattern worth studying before you hit 300 million rows, not after. The lesson is cheaper to learn from someone else’s 300 million rows than from your own.

When your database and your product are the same thing, you don’t get to treat the database as someone else’s problem. It is the product. Build it like one.

The post “The database is the product”: What breaks when memory devices scale appeared first on The New Stack.