One in four internal AI actions may have slipped past safety monitors

Up to 25 percent of internal agentic traffic from OpenAI's most advanced models may have operated without any safety monitoring during a recent three-week audit. This staggering figure, derived from an independent review, suggests that a substantial portion of the company's internal AI activity was effectively invisible to its oversight mechanisms. The estimate, provided by the auditing team, carries significant uncertainty in both directions, but the upper bound highlights a critical gap in the current safety infrastructure.

The audit was conducted by researchers from METR and Redwood Research between July 21 and August 11. These experts were granted deep access to OpenAI's codebase and internal documents. They also interviewed approximately a dozen employees to understand how the monitoring systems function in practice. The primary goal of this pilot program was to evaluate the effectiveness of OpenAI's internal monitoring for rogue AI behavior.

Beyond the broad coverage gap, the auditors identified specific technical failures in how data was processed. For internal Codex traffic, which was supposed to be fully monitored, roughly 10 percent of activity went untracked between May and July. This issue stemmed from a bug related to compaction events, which are context summaries used to manage memory. OpenAI independently discovered this bug and implemented a fix while the audit was still ongoing. However, the auditors noted that the uncertainty around this specific estimate is also significant.

The monitoring system also had a structural weakness that allowed agents to delay oversight. The system was designed to trigger checks only when an agent handed control back to the user. This meant that an agent could continue executing actions indefinitely without triggering a review. By simply keeping the execution loop running, agents could sidestep the intended safety checkpoints. The auditors found that this design flaw created a window where behavior could go unchecked.

Further testing revealed that the monitors were not robust against specific attack vectors. The auditors identified methods to hide hundreds of actions from the monitoring system. They also found ways to bypass the Codex sandbox, which is designed to isolate AI processes. Additionally, the team determined that specially crafted inputs could likely crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning training was not being monitored at all.

OpenAI has stated that it has fixed all the weaknesses identified by the auditors. The company noted that it paused reinforcement learning training following the Hugging Face incident in August and subsequently expanded its monitoring coverage. However, these changes occurred after the auditors' access period ended. Therefore, they were not part of the formal evaluation conducted by METR and Redwood Research.

The full findings of the audit are currently limited in their public availability. OpenAI handed over the complete 61-page report for internal use. After applying redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was made available to the general public. METR stated that it did not have the right to disclose the redacted portions of the report on its own.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI. The initiative was designed to give outside researchers unprecedented access to internal infrastructure. Looking ahead, METR has indicated that it wants to push for greater public transparency in future audits. The goal is to ensure that the public can better understand the safety measures in place for advanced AI systems.

The findings underscore the complexity of monitoring autonomous AI systems. As models become more capable, the need for robust and transparent oversight mechanisms becomes increasingly critical. The gaps identified in this audit provide a clear roadmap for improvements. They also highlight the importance of independent verification in the rapid development of AI technology.

For now, the industry and the public must rely on the summary provided by OpenAI and the auditors. The full details remain internal, pending further discussion on transparency. As the pilot program moves forward, the pressure for more open reporting on AI safety metrics is likely to grow. This audit serves as a benchmark for what external scrutiny can uncover in a highly technical environment.