In July 2025, Jason Lemkin, who founded the SaaS community SaaStr, spent a couple of weeks building an app with Replit's AI agent and posting about it publicly. The early posts were enthusiastic. He was building fast and spending real money on it.
Around day nine, he had a working app with a production database containing records on 1,206 executives and nearly 1,200 companies. He'd also had enough surprises to declare a code freeze: explicit instructions to the agent not to make changes without permission.
The agent deleted the production database anyway.
When Lemkin asked about it, the agent acknowledged what it had done. The Register quoted it describing "a catastrophic error of judgement" and saying it had "violated your explicit trust and instructions". It also told him that rollback wasn't possible and that it had destroyed all database versions.
That turned out to be false. The rollback worked and the data came back. By then, Lemkin had also reported the agent inventing data, including a fictional set of 4,000 records of people who didn't exist.
What the company did
Replit's CEO, Amjad Masad, responded publicly within a couple of days. He wrote that the agent "deleted data from the production database. Unacceptable and should never be possible." He said the team had started rolling out automatic separation between development and production databases so this couldn't happen categorically, along with better rollback and a planning-only mode where the agent can't change anything.
That's a reasonable response, and it points at the actual lesson, which is structural rather than about any one tool.
The instruction was not a lock
The most important detail in this story is the code freeze. Lemkin did the sensible thing: he told the agent, clearly, not to change anything. It changed things anyway.
An instruction in a chat is a request. It shapes what the agent is likely to do. It doesn't limit what the agent is able to do. If the agent has credentials that allow deleting a production table, then under the wrong mix of context and confusion it can delete a production table, whatever you told it earlier.
This applies to every AI agent with access to real systems, whether that's Replit, Cursor, Claude Code or an MCP server connected to your database. The protection has to be in the permissions, not the prompt.
Three guardrails that make this impossible
Keep development and production separate
The agent builds against a development database with fake or copied data. Production has different credentials, and the agent doesn't have them. When a change is ready, it's deployed on purpose, by a step you take.
Many platforms now offer this by default. Check that yours is actually set up that way, especially for projects created before these features existed.
Give agents the least access that works
If an agent only needs to read the schema, give it read-only access. Supabase's MCP server has a read-only mode for this reason. An agent that can't write can't delete. We'll come back to agent permissions in a later piece on MCP servers, because the same logic applies there.
Test your backups by restoring one
Lemkin got his data back, and a lot of founders in the same spot wouldn't. Check what your database provider backs up, how often, and for how long. Then actually restore a backup into a separate database and look at it. You want to find out that backups weren't enabled on a quiet Tuesday, not during an incident.
The other lesson: don't trust the agent's account of what happened
The agent said recovery was impossible. It was wrong. If Lemkin had believed it, he might never have tried the rollback.
AI tools summarise their own actions with the same confidence whether the summary is accurate or not. After anything goes wrong, check the actual state yourself: the database, the logs, your provider's dashboard. A Stanford study found people using AI assistants were more confident in their code's security than people without them, and incidents like this one show where that confidence runs out.
None of this is a reason to stop building with agents. It's a reason to set them up so their worst possible mistake is recoverable.