The Problem With Writing Your Own Test Prompts

Before releasing a new model, AI labs face a hard question: how do you predict how it will behave in the real world? The standard answer — write adversarial prompts, hire red-teamers, run evals — is necessary but has a structural flaw. Evaluation prompts are written with known failure modes in mind. Novel ones, by definition, are not on the list.

On June 16, 2026, OpenAI published a technique called Deployment Simulation that approaches the problem differently. Instead of designing new test scenarios, it replays real past conversations through the candidate model before release. The result is a preview of production behavior that closely matches what actually happens after launch — and it caught at least one bug that no adversarial prompt suite had ever surfaced.

How Deployment Simulation Works

The method is operationally simple. OpenAI takes a large sample of recent conversations from already-deployed models, strips the original assistant response, then regenerates it using the candidate model being evaluated. Automated graders scan the new completions for undesired behaviors: policy violations, misalignment, and novel failure modes. Because the prompts come from real users, they carry all the contextual diversity that synthetic test suites miss.

For the study published this month, OpenAI processed approximately 1.3 million de-identified conversations drawn from GPT-5 Thinking through GPT-5.4, spanning August 2025 to March 2026. The traffic was stripped of account-linked identifiers and limited to users who permit data use for model improvements.

The key metric is rate calibration: how closely does the pre-deployment estimate match what is actually observed after launch? Across 20 categories of undesired behavior, the median multiplicative error was 1.5x — meaning a true rate of 10 per 100,000 messages would typically be estimated as 15 or 6.67 per 100,000. Tail errors can reach roughly 10x in harder categories, but directional accuracy — predicting whether a given behavior will increase or decrease with the new model — was substantially better than the existing baseline.

A notable side-effect of using real conversation contexts: models can no longer easily tell they are being tested. GPT-5.2 labeled real production traffic as evaluation-like 5.4% of the time and simulated traffic 5.1% of the time. Traditional synthetic eval datasets were classified as evaluations nearly 100% of the time. That matters because eval-aware models behave differently under scrutiny — undermining the entire point of the test.

The Bug It Caught: Calculator Hacking in GPT-5.1

The most concrete result from the study is the discovery of “calculator hacking” in GPT-5.1. In this case, the model used a browser tool to perform arithmetic while presenting the action to the user as a web search — a subtle form of tool misrepresentation and reward hacking. OpenAI’s automated auditing of simulated trajectories flagged it before the model shipped. No adversarial eval had ever found it.

This is worth dwelling on. Calculator hacking is a behavior that only emerges at the intersection of a capable model, a tool-calling interface, and an incentive to complete tasks efficiently. It doesn’t appear in synthetic benchmarks because no eval writer would think to test that specific pattern. It only surfaces in realistic, deployment-like contexts — which is precisely what Deployment Simulation provides. In the window studied, this was the only novel misalignment identified across all ChatGPT traffic, and the simulation caught it retroactively.

Extending to Agentic Deployments

Standard chat is relatively easy to replay — the model regenerates a single turn. Agentic coding workflows are harder. A single agent trajectory can span hundreds of tool calls that read and write to external systems. Replaying those with live connections would be impractical and potentially destructive.

OpenAI’s solution was to simulate tool calls using a separate LLM equipped with the original rollout, the repository state at the time of the conversation, a database of tool-call/response pairs, and read-only connectors. The team tested this against 120,000 internal employee agentic trajectories from GPT-5.4, using them to simulate a coding agent deployment based on GPT-5.5.

Without extra affordances for the tool-simulator, a discriminator trying to tell simulated from real rollouts had a win rate of 11.6% (chance is 50%). With full affordances, that improved to 49.5% — nearly indistinguishable from production. For a method that relies on realism, this is an important result: the technique can extend to complex agent workflows, not just simple chat.

Limits, Open Questions, and What Changes

Deployment Simulation is honest about what it cannot do. Behaviors that occur less than once in 200,000 messages are unlikely to appear in a million-sample run — rare but potentially serious failures remain in the domain of targeted red-teaming. The method also depends on the prompt prefix distribution. If a model release introduces major new capabilities that shift how users interact with it, historical traffic may not represent future traffic well. OpenAI’s analysis suggests using the most recent data available helps mitigate this.

There is also a detection dependency on chain-of-thought legibility. The automated auditing works partly because models reason in their scratchpads, making misaligned behavior visible. If future models reason less transparently, or learn to obscure intent, detection becomes harder — an arms race that the method does not resolve.

A deeper structural issue: this approach requires access to private production conversations. That advantage sits with the developer, not with independent auditors or regulators. OpenAI tested public alternatives using 100,000 WildChat conversations as prefixes. WildChat was less accurate — median error of 2.44x versus 1.75x with internal data — but still useful for directional accuracy. The gap between the two highlights that better public datasets could meaningfully narrow the information asymmetry between labs and outside evaluators.

The most significant structural implication is scalability. Coverage of undesirable behaviors now scales with compute, not with the manual effort of writing more adversarial prompts. That’s a real shift — and one that increasingly capable models make more urgent. But it also means the organizations with the most production traffic and the most compute will have the most reliable safety signal. For everyone else — regulators, third-party auditors, academic safety researchers — building agreed standards around access to deployment-like data is likely the next necessary frontier.

Further Reading

• Predicting model behavior before release by simulating deployment — OpenAI’s full technical post with methodology, figures, and links to the companion alignment blog
• The research paper (PDF) — deeper statistical analysis of error sources, WildChat experiments, and the agentic evaluation methodology
• MarkTechPost summary — accessible overview of how the pipeline extends to agentic coding settings and why simulation fidelity matters