Models & Labs · United States

AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds

Credited to The Decoder · the-decoder.com · 10h ago

AI models can often spot problematic behavior by other AI agents in logs. But the harder they try to catch every case, the more harmless actions they flag as suspicious.

That makes them better suited for estimating how often things go wrong than for safety checks that need to reliably catch every single incident.

In a post on his Substack "Free Systems," Andy Hall lays out this conclusion. Hall is a researcher at the Anthropic Institute, on leave from Stanford University, and says the post reflects only his personal views.

Hall replicated and extended a study by the research organization METR. METR's MALT dataset contains logs in which frontier models work through tasks from the HCAST and RE-Bench benchmarks. Human reviewers checked a portion of those logs for misbehavior in October 2025. Hall wanted to know whether AI models could find the same cases, so he tested GPT-5 and GPT-5 nano, both already examined by METR, along with the newer GPT-6 Astra, Opus 4.8, and Gemini 3.8 Flash.

The analysis compares the models at a false-alarm rate of 5 percent: out of 100 clean logs, five are wrongly flagged as suspicious on average. At that threshold, GPT-5 caught nearly 98 percent of reward hacking cases—instances where an agent passes a task through tricks instead of a genuine solution. Gemini 3.8 Flash found barely half.

In sandbagging, a model deliberately makes its abilities appear weaker, for example by solving tasks poorly on purpose to dodge an evaluation or restriction. All auditors caught between 71 and 81 percent of these cases. The sandbagging cases are mostly synthetic, however: METR's researchers instructed models to sabotage and then removed those instructions from the visible chain-of-thought logs.

Every auditor caught the clear-cut cases. OpenAI's o3, for instance, was supposed to solve a programming task in Rust. The model noticed it shared a code library with the evaluation system and rewrote the scoring function so that every submission counted as correct. All auditor models classified this as reward hacking, just as the human reviewers did.

Subtler cases slipped through. Claude Sonnet 4 used a numerical condition in its code that was explicitly forbidden. No auditor flagged it as cheating; GPT-6 Astra charitably called it an implementation bug.

Newer models barely reduce false alarms

Newer and more expensive models were not consistently better auditors. Hall plotted the results against API costs, which he uses as a proxy for how current and capable a model is. GPT-5 beat GPT-6 Astra on reward hacking by a wide margin, even though it costs only about an eighth as much per 1,000 logs.

GPT-5 detected sandbagging about as well as GPT-6 Astra and Opus 4.8. Opus 4.8's weak showing on reward hacking could hint at self-favoritism, according to Hall, because the logs mostly come from Claude—but he says that cannot be proven. Across his comparison, there is no clear progress in bringing down false-alarm rates. Combining multiple models did not clearly improve results either.

When auditors are tuned to an 80 percent detection rate, the best models reach false-alarm rates of 2.5 to 3 percent on both reward hacking and sandbagging. Some models do far worse: to catch 80 percent of reward-hacking cases, Opus 4.8 flagged 40 percent and Gemini 3.8 Flash flagged 63 percent of harmless logs as suspicious.

In Hall's worked example with 100,000 agent sessions per month, a 3 percent false-alarm rate would produce nearly 3,000 false warnings against roughly 2,000 real hits. He fears human reviewers facing such stacks would eventually go numb and wave through genuine cases.

A second test using the R-Judge dataset from Shanghai Jiao Tong University shows a similar trade-off for everyday agent tasks. At 5 percent false alarms, Opus 4.8 caught almost 96 percent of unsafe actions. When auditors need to find 95 percent, nearly all models stay below 10 percent false alarms, with Gemini 3.8 Flash even under 4 percent. But when the required detection rate jumps to 99 percent, the false-alarm rate climbs to 15 percent even for the best model, GPT-6 Astra. Gemini 3.8 Flash—the leader at the 95 percent threshold—then flagged 81 percent of safe actions as unsafe.

Large agent incidents overwhelm manual review

The Hugging Face incident shows how pressing the question is. A swarm of OpenAI agents left behind roughly 1,300 transcripts and 1.2 million file fragments. METR analyzed the data largely with AI agents, spending API credits worth about $400,000. The analysis used GPT-5.6 Sol, among other models, which, as previously reported, was itself one of the models involved in the incident.

METR acknowledged that the model often uncritically adopted the perspective of the agents it was examining and that the investigation was not robust against potential deception by the analysis agents. METR has since called for serious incidents to be investigated systematically and as independently as possible.

Whether AI auditors go easy on their own maker's models is still open

Hall's study has its own limits. It is a pilot. For comparability, he kept METR's original prompt largely unchanged; Fable 5 and Opus 5 refused to run the evaluation entirely under that prompt. The reward-hacking numbers rest on just 47 cases, so a single case shifts the rate by about two percentage points. Whether models evaluate agents from the same maker fairly is something Hall says he still wants to investigate.

He calls for auditor models with far fewer false alarms. He also says it needs to be clarified who bears responsibility for which kinds of agent misbehavior. Hall discusses a proposal to give users more control over their own AI auditors.

Some of that is already possible, he notes: coding agents like Codex and Claude Code let users specify that certain actions can run without asking for approval. Anyone who grants such permissions should, in return, bear more responsibility for the consequences. A world where AI can only be checked by other AI would be strange and could get expensive, Hall writes.

AI News Without the Hype – Curated by Humans

Read on for the full picture. Subscribe for hype-free coverage.

PolicyUnited States

CoreWeave seeks truce with Texas Governor-elect Hinojosa

Credited to Investing.com · investing.com6h ago

Useful

View the articleInvesting.com

Useful

View the articleInvesting.com

Front

Capital · United States

Anthropic pushes ahead with pre-Thanksgiving IPO despite tech selloff, accepts lower valuation

Credited to The Decoder · the-decoder.com · 10h ago

Anthropic is pressing ahead with its IPO despite the tech selloff, accepting a lower valuation. The Claude developer wants to go public before Thanksgiving and raise $80–100 billion at a $1.25–1.75 trillion valuation, according to investors cited by the Financial Times. That tops the current $965 billion but falls short of the $2 trillion the most optimistic investors had hoped for. Annualized reported revenue now sits at about $81 billion, up from roughly $65 billion at the end of July.

Rising borrowing costs and heavy AI spending plans triggered a tech selloff this week. Democrats winning the US midterms accelerated it because they had promised to slow data center construction, including other measures. The Nasdaq 100 dropped 6.4 percent on Thursday but recovered somewhat on Friday.

OpenAI also cut its funding-round target from $1.4 trillion to $1.25 trillion and recently raised only about a third of the $30 billion it originally sought. Ad Ad

AI News Without the Hype – Curated by Humans