Models & Labs · United States

Why a new report on Anthropic automating AI development is alarming researchers

Credited to The Verge · theverge.com · 23h ago

AI is dramatically speeding up Anthropic’s research, even as the company calls for slowing down AI development.

Anthropic is using AI to build more powerful AI. And it appears to be working remarkably well, with the company’s models improving roughly twice as quickly as a year ago, with little indication of slowing down.

That’s according to a new report published Thursday by the AI safety research organization METR, following an 11-week investigation inside the company, as part of a report evaluating its AI research and development. The probe is separate from a program offering safety researchers “employee-like” access to systems in the wake of high-profile hacking incidents over the summer.

The findings offer some of the clearest evidence yet of AI accelerating its own development, raising questions about how quickly the technology could advance toward a future in which AI systems can improve themselves with little or no human involvement. They also come at a pivotal time for Anthropic, which is preparing to go public even as its CEO, Dario Amodei, has publicly called for labs to pump the brakes on AI development.

But while the report offers a rare glimpse into the inner workings of the world’s leading AI labs, what it leaves out is just as revealing. Anthropic was permitted to redact portions of the final report as part of its agreement with METR, and it appears to have exercised that power liberally, raising questions about how much the public really knows about the pace — and risks — of frontier AI development.

AI is dramatically accelerating Anthropic’s research and development

One of METR’s most noteworthy findings is that Anthropic’s AI models are improving roughly twice as quickly as they were a year ago, with benchmark scores suggesting “that Anthropic has made roughly two years of historical progress over the last year.” Crucially, METR’s researchers said they “are confident” that the acceleration was not simply due to scaling up the computing power used to train the models, which historically has been a major driver of AI progress.

Instead, the researchers said that the majority of the acceleration as of October “is due to AI,” with its contribution growing over time. That finding was supported by informal Slack polls and interviews with Anthropic employees, which pointed to sizable productivity gains. METR wrote that employees using Anthropic’s most capable internal model on average self-reported being around 12 times as productive at their respective work as they would have been without AI assistance, around double what staff had reported just a few months earlier while using Mythos 5.

For now, much of that acceleration appears to be coming from AI assisting human researchers, rather than AI working independently, though METR warned that “there are some signs that fully autonomous contributions will play a larger role soon.” Notably, the researchers said they found no clear signs of diminishing returns or observable bottlenecks that could prevent “even faster acceleration.”

Three of Anthropic’s newest AI models remain under wraps

Anthropic’s researchers are using more advanced AI models than the rest of us, with a lot of the acceleration and productivity gains described by METR attributable to unreleased internal models.

Over the past six months, Anthropic has finished training three models it has yet to release, according to METR. The pace of development is swift, with the models — codenamed Model 2, Model 3, and Model 4 — completing training in July, August-September, and September-October, respectively. METR also identified an unreleased model dating back to 2022, called Unreleased 2022. “These are the codenames shared with us; we understand they have different internal codenames,” the researchers wrote.

Anthropic is keeping key information secret

Anthropic gave METR access to a host of non-public information, including confidential research, model performance data, and employee interviews as part of the research process. It had no obligation to do so and by METR’s own admission, the AI lab “shared more information than was initially scoped for this project, surpassing our expectations.” This is commendable, but it is worth noting that Anthropic did not grant researchers unrestricted access, nor the kind of “employee-like” access it has promised other METR safety researchers as part of a separate program announced last month. It is an important distinction, given the reputational benefits companies can gain from independent assessments, even when they come with significant restrictions.

And there were significant restrictions. Anthropic had the power to redact information from METR’s final report — and it used that power extensively. “We submitted a ~100-page full report on our assessment to Anthropic. For the public version, Anthropic redacted a majority of the evidence we found most informative,” METR wrote on X.

METR reiterated those concerns throughout the report, highlighting other limitations on its access. “Our pipeline of information was relatively narrow, and a small number of our requests for data were denied.”

METR repeatedly stressed the impact Anthropic’s redaction decisions had on its ability to communicate its findings and for others to assess what’s going on at the company. It noted that much of the redacted evidence could have “multiple plausible interpretations,” making it difficult for the public to assess its findings, understand the uncertainties surrounding them, and judge the evidence themselves. “As a consequence [of the redactions], the public has to rely on METR’s judgment much more than would be ideal, given the weight of the questions answered here,” the report said.

In the report, METR said Anthropic justified the redactions on the grounds that releasing the information could benefit its rivals. However, METR said “most of the redacted information was very high-level,” even though it said it was not in position to comment whether redactions were justified commercially or not.

“We redacted a lot, and I accept that this makes it harder for others to check METR’s work,” Anthropic cofounder Sam McCandlish acknowledged in a lengthy post on X responding to METR’s findings. “But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster,” he said, adding that doing so would make the risks METR is concerned about “worse.”

McCandlish said that it’s a trade-off, but maintained Anthropic had made the “right call” in deciding to redact the information. He said the company plans to publish its own report on the speed of AI progress “in the coming weeks,” as well as new steps it’s taking to strengthen safety.

Anthropic is accelerating the very technology it warns is dangerous and faster than many expected

The publication of METR’s report has provoked a fierce backlash among AI safety researchers and commentators, largely centered on an apparent contradiction at the heart of Anthropic’s approach to AI: The company has been one of the most outspoken advocates for safety and responsible development, yet the report suggests it is helping accelerate the technology toward a threshold many consider dangerous, while seemingly failing to live up to its own commitments to transparency and accountability.

For some, the findings suggest that even many of the field’s most aggressive predictions about AI progress may be too conservative. They indicate that recursive self-improvement — AI systems that can autonomously develop increasingly powerful successor systems without human intervention, potentially accelerating development dramatically — may be closer than previously thought, raising concerns about whether existing safety measures can keep pace. While METR’s report only covers Anthropic, OpenAI said in September that it is “making strong progress” toward building an AI researcher that can “further progress on deep learning and alignment” by March 2028.

The report provides evidence about just how quickly AI is accelerating its own development

Marius Hobbhahn, CEO and founder of Apollo Research, which conducts AI safety testing for several leading AI labs, told The Verge the report suggests a pace of AI-enabled research and development even faster than what was forecast in AI 2027, an influential scenario that envisions a key breakthrough toward self-improvement in early 2027. He said the report has attracted so much attention because it provides evidence about something that has been very hard to pin down: Just how quickly AI is accelerating its own development.

Hobbhahn said the findings were broadly in line with what many in the field had expected, perhaps a little faster. Alan Chan, a senior research fellow at tech policy research center GovAI, told The Verge that the findings should prompt skeptics to reconsider their stance on AI accelerating AI development, and possibly shorten their timelines to when we will have systems that can fully automate tasks like software engineering.

On X, other researchers commented on the rate of acceleration and its implications for AI development. Sydney Von Arx, founder and CEO of AI safety organization The Nightingale Collective and a former METR researcher, also compared the findings to AI 2027 ’s timeline. Ryan Greenblatt, a METR researcher formerly of AI safety organization Redwood Research and one of three external researchers permitted to investigate OpenAI’s hack of Hugging Face, said he “was seriously wrong about RSI speed up; I previously expected much less in 2026.”

Frustration is mounting over Anthropic’s approach to safety

Beyond highlighting the speed of AI development, the report has also fueled a growing sense of frustration over Anthropic’s role in driving that progress. The company has been one of the most vocal proponents of responsible AI development — and no stranger to calling for others to slow down — yet appears to be racing ahead itself. This report only builds on already heightened tensions from its IPO prospectus and future plans released last month.

Anthropic’s decision to heavily redact the report, citing grounds that revealing more could help its rivals accelerate their own development, has fueled accusations of hypocrisy. Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, which published AI 2027, accused Anthropic of “recklessly gambling with our lives,” adding that “it is pretty rich” of the company to cite others going faster as a reason to redact information when it does “not appear to be slowing down much at all.”

Anthropic declined to answer The Verge ’s questions about the report and ensuing criticism, instead directing us to McCandlish’s post.

The redactions have also renewed calls for greater transparency and independent oversight of frontier AI labs, which grew to new heights earlier this year following a spate of incidents involving rogue AI agents. Brianna Rosen, research director for frontier security at the Institute for AI Policy and Strategy, told The Verge that it was concerning that companies, not governments, were effectively calling the shots on automating AI research. “For democratic oversight to work, independent evaluators need the freedom to explain why they reached their conclusions, governments need to assess the evidence themselves, and the public has to trust this process,” Rosen said. “The report was not sufficiently transparent about the evidence.”

Pressure is building for more transparency from AI labs

Hobbhahn echoed Rosen’s concerns, saying that much of the reaction to the report stems from the apparent gap between Anthropic’s public safety stance and its actions. He pointed to the company’s decision to cancel part of the assessment designed to evaluate how much compute it allocates to safety work versus improving capabilities. METR said the company said “this was due to wanting to keep the engagement focused on risk assessment for the time being, and that it would revisit compute allocation verification later on.”

For Hobbhahn, the episode underscores the need for independent safety researchers to be embedded more deeply within AI labs, with significant access to internal systems and evidence. While his organization stands to benefit from such arrangements — Apollo is taking steps toward becoming an embedded evaluator at frontier labs — he is far from alone in calling for stronger oversight of AI development.

Others have called for more direct action. In a direct response to the report, New York state Assemblyman Alex Bores, a key figure shaping the Democrats’ AI agenda, called on Congress to “ban RSI, this year if possible.” With the blue wave of Democrats — many of whom campaigned against AI companies — taking office soon, the prospect of tougher restrictions is a very real possibility.

And with AI research showing little sign of slowing down, the pressure is set to intensify. Hobbhahn said he had another takeaway from the report, too: “Brace for craziness,” he said. “It is going to go very fast,” likely way too fast for society or politics to keep up.

Models & LabsUnited States

AI Safety Advocates Make Gains After $110 Million Midterm Fight

Credited to Bloomberg$ · bloomberg.com21h ago

Useful

Candidates who favor stricter oversight of artificial intelligence racked up wins in key congressional races, aided by a pro-regulation political funding group backed in part by Anthropic PBC at a time when Americans are increasingly uneasy over the rapidly developing technology.

View the article · $Bloomberg

Front