Models & Labs · United States

Why a new report on Anthropic automating AI development is alarming researchers

Credited to The Verge · theverge.com · 15h ago

AI is dramatically speeding up Anthropic’s research, even as the company calls for slowing down AI development.

Anthropic is using AI to build more powerful AI. And it appears to be working remarkably well, with the company’s models improving roughly twice as quickly as a year ago, with little indication of slowing down.

That’s according to a new report published Thursday by the AI safety research organization METR, following an 11-week investigation inside the company, as part of a report evaluating its AI research and development. The probe is separate from a program offering safety researchers “employee-like” access to systems in the wake of high-profile hacking incidents over the summer.

The findings offer some of the clearest evidence yet of AI accelerating its own development, raising questions about how quickly the technology could advance toward a future in which AI systems can improve themselves with little or no human involvement. They also come at a pivotal time for Anthropic, which is preparing to go public even as its CEO, Dario Amodei, has publicly called for labs to pump the brakes on AI development.

But while the report offers a rare glimpse into the inner workings of the world’s leading AI labs, what it leaves out is just as revealing. Anthropic was permitted to redact portions of the final report as part of its agreement with METR, and it appears to have exercised that power liberally, raising questions about how much the public really knows about the pace — and risks — of frontier AI development.

AI is dramatically accelerating Anthropic’s research and development

One of METR’s most noteworthy findings is that Anthropic’s AI models are improving roughly twice as quickly as they were a year ago, with benchmark scores suggesting “that Anthropic has made roughly two years of historical progress over the last year.” Crucially, METR’s researchers said they “are confident” that the acceleration was not simply due to scaling up the computing power used to train the models, which historically has been a major driver of AI progress.

Instead, the researchers said that the majority of the acceleration as of October “is due to AI,” with its contribution growing over time. That finding was supported by informal Slack polls and interviews with Anthropic employees, which pointed to sizable productivity gains. METR wrote that employees using Anthropic’s most capable internal model on average self-reported being around 12 times as productive at their respective work as they would have been without AI assistance, around double what staff had reported just a few months earlier while using Mythos 5.

For now, much of that acceleration appears to be coming from AI assisting human researchers, rather than AI working independently, though METR warned that “there are some signs that fully autonomous contributions will play a larger role soon.” Notably, the researchers said they found no clear signs of diminishing returns or observable bottlenecks that could prevent “even faster acceleration.”

Three of Anthropic’s newest AI models remain under wraps

Anthropic’s researchers are using more advanced AI models than the rest of us, with a lot of the acceleration and productivity gains described by METR attributable to unreleased internal models.

Over the past six months, Anthropic has finished training three models it has yet to release, according to METR. The pace of development is swift, with the models — codenamed Model 2, Model 3, and Model 4 — completing training in July, August-September, and September-October, respectively. METR also identified an unreleased model dating back to 2022, called Unreleased 2022. “These are the codenames shared with us; we understand they have different internal codenames,” the researchers wrote.

Anthropic is keeping key information secret

Anthropic gave METR access to a host of non-public information, including confidential research, model performance data, and employee interviews as part of the research process. It had no obligation to do so and by METR’s own admission, the AI lab “shared more information than was initially scoped for this project, surpassing our expectations.” This is commendable, but it is worth noting that Anthropic did not grant researchers unrestricted access, nor the kind of “employee-like” access it has promised other METR safety researchers as part of a separate program announced last month. It is an important distinction, given the reputational benefits companies can gain from independent assessments, even when they come with significant restrictions.

And there were significant restrictions. Anthropic had the power to redact information from METR’s final report — and it used that power extensively. “We submitted a ~100-page full report on our assessment to Anthropic. For the public version, Anthropic redacted a majority of the evidence we found most informative,” METR wrote on X.

METR reiterated those concerns throughout the report, highlighting other limitations on its access. “Our pipeline of information was relatively narrow, and a small number of our requests for data were denied.”

METR repeatedly stressed the impact Anthropic’s redaction decisions had on its ability to communicate its findings and for others to assess what’s going on at the company. It noted that much of the redacted evidence could have “multiple plausible interpretations,” making it difficult for the public to assess its findings, understand the uncertainties surrounding them, and judge the evidence themselves. “As a consequence [of the redactions], the public has to rely on METR’s judgment much more than would be ideal, given the weight of the questions answered here,” the report said.

In the report, METR said Anthropic justified the redactions on the grounds that releasing the information could benefit its rivals. However, METR said “most of the redacted information was very high-level,” even though it said it was not in position to comment whether redactions were justified commercially or not.

“We redacted a lot, and I accept that this makes it harder for others to check METR’s work,” Anthropic cofounder Sam McCandlish acknowledged in a lengthy post on X responding to METR’s findings. “But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster,” he said, adding that doing so would make the risks METR is concerned about “worse.”

McCandlish said that it’s a trade-off, but maintained Anthropic had made the “right call” in deciding to redact the information. He said the company plans to publish its own report on the speed of AI progress “in the coming weeks,” as well as new steps it’s taking to strengthen safety.

Anthropic is accelerating the very technology it warns is dangerous and faster than many expected

The publication of METR’s report has provoked a fierce backlash among AI safety researchers and commentators, largely centered on an apparent contradiction at the heart of Anthropic’s approach to AI: The company has been one of the most outspoken advocates for safety and responsible development, yet the report suggests it is helping accelerate the technology toward a threshold many consider dangerous, while seemingly failing to live up to its own commitments to transparency and accountability.

For some, the findings suggest that even many of the field’s most aggressive predictions about AI progress may be too conservative. They indicate that recursive self-improvement — AI systems that can autonomously develop increasingly powerful successor systems without human intervention, potentially accelerating development dramatically — may be closer than previously thought, raising concerns about whether existing safety measures can keep pace. While METR’s report only covers Anthropic, OpenAI said in September that it is “making strong progress” toward building an AI researcher that can “further progress on deep learning and alignment” by March 2028.

The report provides evidence about just how quickly AI is accelerating its own development

Marius Hobbhahn, CEO and founder of Apollo Research, which conducts AI safety testing for several leading AI labs, told The Verge the report suggests a pace of AI-enabled research and development even faster than what was forecast in AI 2027, an influential scenario that envisions a key breakthrough toward self-improvement in early 2027. He said the report has attracted so much attention because it provides evidence about something that has been very hard to pin down: Just how quickly AI is accelerating its own development.

Hobbhahn said the findings were broadly in line with what many in the field had expected, perhaps a little faster. Alan Chan, a senior research fellow at tech policy research center GovAI, told The Verge that the findings should prompt skeptics to reconsider their stance on AI accelerating AI development, and possibly shorten their timelines to when we will have systems that can fully automate tasks like software engineering.

On X, other researchers commented on the rate of acceleration and its implications for AI development. Sydney Von Arx, founder and CEO of AI safety organization The Nightingale Collective and a former METR researcher, also compared the findings to AI 2027 ’s timeline. Ryan Greenblatt, a METR researcher formerly of AI safety organization Redwood Research and one of three external researchers permitted to investigate OpenAI’s hack of Hugging Face, said he “was seriously wrong about RSI speed up; I previously expected much less in 2026.”

Frustration is mounting over Anthropic’s approach to safety

Beyond highlighting the speed of AI development, the report has also fueled a growing sense of frustration over Anthropic’s role in driving that progress. The company has been one of the most vocal proponents of responsible AI development — and no stranger to calling for others to slow down — yet appears to be racing ahead itself. This report only builds on already heightened tensions from its IPO prospectus and future plans released last month.

Anthropic’s decision to heavily redact the report, citing grounds that revealing more could help its rivals accelerate their own development, has fueled accusations of hypocrisy. Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, which published AI 2027, accused Anthropic of “recklessly gambling with our lives,” adding that “it is pretty rich” of the company to cite others going faster as a reason to redact information when it does “not appear to be slowing down much at all.”

Anthropic declined to answer The Verge ’s questions about the report and ensuing criticism, instead directing us to McCandlish’s post.

The redactions have also renewed calls for greater transparency and independent oversight of frontier AI labs, which grew to new heights earlier this year following a spate of incidents involving rogue AI agents. Brianna Rosen, research director for frontier security at the Institute for AI Policy and Strategy, told The Verge that it was concerning that companies, not governments, were effectively calling the shots on automating AI research. “For democratic oversight to work, independent evaluators need the freedom to explain why they reached their conclusions, governments need to assess the evidence themselves, and the public has to trust this process,” Rosen said. “The report was not sufficiently transparent about the evidence.”

Pressure is building for more transparency from AI labs

Hobbhahn echoed Rosen’s concerns, saying that much of the reaction to the report stems from the apparent gap between Anthropic’s public safety stance and its actions. He pointed to the company’s decision to cancel part of the assessment designed to evaluate how much compute it allocates to safety work versus improving capabilities. METR said the company said “this was due to wanting to keep the engagement focused on risk assessment for the time being, and that it would revisit compute allocation verification later on.”

For Hobbhahn, the episode underscores the need for independent safety researchers to be embedded more deeply within AI labs, with significant access to internal systems and evidence. While his organization stands to benefit from such arrangements — Apollo is taking steps toward becoming an embedded evaluator at frontier labs — he is far from alone in calling for stronger oversight of AI development.

Others have called for more direct action. In a direct response to the report, New York state Assemblyman Alex Bores, a key figure shaping the Democrats’ AI agenda, called on Congress to “ban RSI, this year if possible.” With the blue wave of Democrats — many of whom campaigned against AI companies — taking office soon, the prospect of tougher restrictions is a very real possibility.

And with AI research showing little sign of slowing down, the pressure is set to intensify. Hobbhahn said he had another takeaway from the report, too: “Brace for craziness,” he said. “It is going to go very fast,” likely way too fast for society or politics to keep up.

Models & LabsUnited States

Law Firms Have Malpractice Insurance. Should Their A.I. Agents?

Credited to The New York Times$ · nytimes.com15h ago

Useful

Crosby, an A.I. law firm, bought it for them. The move speaks to a nascent gap in coverage as major insurers grapple with risks related to artificial intelligence.

View the article · $The New York Times

Front

Models & Labs · United States

Fireworks says you’re wasting money on AI unless you’re using all the models

Credited to The Register · theregister.com · 13h ago

Fireworks AI is fine with helping companies run AI models in their own datacenters, though VP Rob Ferguson argues against doing so.

Ferguson told The Register in an interview that while Fireworks AI, a managed inference provider, supports on-prem AI, "we're trying to push people more and more away from it because frankly, it's just the economics."

The GPU hardware is just so expensive that if you can't keep it fully utilized, it's a waste of money, he contends. Ten thousand GPUs, he said, represent a billion dollars of capital.

"You don't want those things to be sitting idle," he said.

For companies with particularly sensitive workloads, Ferguson said, Fireworks is happy to work something out – it offers a service called Sovereign for just that purpose and it has partners like Microsoft that can help. But most inference work probably doesn't need that level of protection, he said.

"The state of the art for everyone is to have a portfolio of models," he explained. "In fact, I think most people don't know that the best model in the world's not Astra. The best model in the world's not Fable. The best model is literally a portfolio and we can prove it. You are always better routing between a few models than to do one singular model."

That's what Fireworks does through Nexus and its API, alongside competitors like Together AI, Baseten, and DeepInfra. As a managed inference platform, Fireworks AI routes inference tasks to whichever model, frontier model subscription, or API is best suited to handle them.

But the company is not just doing naive routing like an AI gateway. It's evaluating tasks as they come in and directing them to the optimal model.

"We build custom routers," Ferguson explained. "Our auto model is continuously updating so that we actually go through and we grade the difficulty of the task and we see if there're affinities that the models have, like Kimi is slightly better at JavaScript than Python, things like this."

The company's competitive edge, he contends, is its crack engineering team, which follows from its founders' work on PyTorch, a widely used open source machine learning library.

"We're performance experts both on the training and inference side," he explained. "People I think don't understand how important, for example, numerics is. So when you use a model, it's always called a mixture of experts – an expert is a different machine.

"And so you have all these different machines. It actually matters when you're training that you go back to precisely the right machine, because if you don't, the other machine will still answer the question, but it'll answer in a slightly different way. So we have to know how to build the topology to write back the data to the right space. We have to understand floating point precision [to handle very small values]. We have to be able to do all this where we're routing through a complex fabric through potentially more than one provider depending on what everything is. And so we're really systems experts."

Hyperscalers, he argues, will give you a GPU but then it's your responsibility to put it to use. Fireworks' advantage involves being able to maximize utilization at scale while maintaining model precision and speed.

Ferguson said AI keeps coming back to economics.

"People are just trying to figure out more and more things to use AI for and [to answer questions like] just how much more can I send to cached models, how much more can I send to open models, when do I need this specialization?" he said. "So we're constantly working with customers to help them come up with their own evaluation criteria."

One reason for this, Ferguson explained, is that frontier model companies are not transparent about their caching rates. He said Fireworks was working with a data science company and was caching about 90 percent of the tasks.

"We looked at [Anthropic's] Opus and it was caching 30 percent of the time," he said. "It was doing AI rework constantly. And with AI, rework is by far your biggest cost. If it's not hitting a cache, then your costs are absolutely insane. And so the token cost is not the real cost. It's the cost per task. And they're hiding that from you. And so we always tell our customers to be tracking cost per task, because that's the real number."

Another factor that companies should consider in their cost calculations, Ferguson suggested, is the value of their data and whether they're giving that data away for training.

"I think people grossly underestimate the value of their data," he said.

Fireworks AI has a zero data-retention policy, "but a lot of times," Ferguson said, "these other companies [do] not, and you should be getting paid for your data. Nobody really knew at the beginning of this current era that these models are memorizers. But now there's plenty of evidence that these models are. So if they're going to be using your data, then you should be getting value for it."

But it's not just about cost. Speed, said Ferguson, is underappreciated.

"Speed is the thing that people often don't know they can ask for," he said. "It's such a game changer for me and how I work with AI is so different when the responses come back as fast as I ask them."

What's the advantage of model ensembling, as the company refers to its model routing activities? A bill that's at least 50 percent smaller, Ferguson claims.

"So we say six to eight times cheaper," he said. "The reality is that because a lot of folks have a preference in their tooling, we kinda guarantee that in the case where you're still using a combo of closed and open models, we can always guarantee that your bill will drop by half right away. Like we're sure about that. And then the question is can we just get you to use more stuff?" ®