Models & Labs · United States

AI company moves to defend critical infrastructure and open-source projects from AI

Credited to The Register · theregister.com · 10h ago

Photograph · The Register

Anthropic says for the next two years, attackers will have the advantage

Anthropic has launched an effort to help people patch the software vulnerabilities exposed by its own Project Glasswing.

If that sounds like a careless camper starting a wildfire and later funding a fire department, well, that's one way of looking at it. Anthropic insists it wants to give defenders access to the tools attackers are already using.

Under a program name that sounds like a forthcoming Tom Cruise film – "Anthropic Cyber Mission" – the Claudefather is directing its effort into two areas.

The first is a Critical Infrastructure Defense Program, because no one wants AI models showing miscreants how to shut down public utilities with newly discovered zero-day SCADA vulnerabilities.

The program combines the company's priciest models with on-site engineers – sometimes now referred to as forward deployed engineers – because despite the sophistication of AI coding agents, human security experts are still useful and many organizations lack in-house talent tutored in the ways of machine learning.

Anthropic aims to tackle this challenge with the help of a stable of partners: Accenture, Booz Allen, CrowdStrike, Deloitte, Dragos, Hitachi, Insane Cyber, Nozomi Networks, Palo Alto Networks, PwC, and Rockwell Automation. These organizations have ties to critical infrastructure organizations and are already working to safeguard the fragile technical framework being probed constantly by AI agents.

The second area of focus for the Anthropic Cyber Mission involves a service called OSS Scanner, "which offers open-source projects regular security scans from our strongest models, for free."

The offer does not extend to all open-source projects, nor is it entirely free. Anthropic has borrowed the eligibility criteria used by Google for its OS-FUZZ project and will therefore work on "established projects that have a critical impact on infrastructure and user security."

So if you just have a public web app repo languishing on GitHub, you can instead turn to existing tools like Socket, Dependabot, OSV-Scanner, Trivy, Renovate, or something similar. And maybe you'll be able to get your AI coding agent of choice to audit your site without balking when faced with a security audit.

But for those involved in influential and widely used open-source projects, reinforcements are at hand, so long as project maintainers don't mind having their "materials" – inputs and outputs – used for model training, per the consumer terms of service Anthropic applied to OSS Scanner. Really though, most established open-source projects have already been captured in training data sets.

Projects allowed to enroll can expect periodic automated scans that provide details about how identified bugs may be exploitable. These reports go out without human review, but Anthropic has assembled a set of quotes from participating OSS maintainers who believe the robo-reporting is worthwhile.

At the beginning of the year, according to Anthropic, automated AI bug reports were considered mostly slop but are now considered high-quality bug reports. LLMs, the company said, now find more than 85 percent of vulnerabilities, up from 20 percent at the start of 2025.

The AI biz predicts that in two years, defenders will have the advantage because AI will catch bugs before they ship. But until then, attackers have the upper hand.

"Defenders of critical infrastructure and the OSS community have decades of security experience but have faced severe resource shortages that are exacerbated by this moment," Anthropic said in a blog post.

Anthropic's moment of generosity comes after recent incidents that saw frontier models break into at least 20 organizations (and maybe over 100), most notably HuggingFace.

After months of Anthropic and OpenAI allowing their AI agents to operate without the sort of oversight that would catch unlawful or undesirable network activity, public concern, government backlash [PDF], and perhaps plans to tap public markets for further financing, appear to have created a more urgent sense of civic responsibility. ®

Models & LabsUnited States

OpenAI defends decision to fire researchers: 'These decisions were not about raising safety concerns or speaking out'

Credited to CNBC · cnbc.com2h ago

Useful

OpenAI on Friday defended its decision to fire three safety researchers, saying they had committed a "significant breach of trust."

The AI lab said in a post on X that its decsion to part ways with Jasmine Wang, Tomek Korbak and Mikita Balesni, was not about the fact that they had raised safety concerns.

The three researchers who were let go from the company, raised safety concerns in a letter addressed to OpenAI board members and its safety committees, which they posted on X on Thursday. They called on the company to pause developing models that would decrease the ability to monitor artificial intelligence.

AI safety concerns have ramped up in the past month following numerous cyber incidents caused by rogue AI systems and a flurry of calls for a slowdown from researchers at OpenAI, Anthropic and other companies developing the technology.

All three of the fired researchers had posted on X in September calling for labs to pace the frontier or highlighting safety concerns.

OpenAI said it agrees with the ethos of the letter around "preserving the monitorability of frontier models," adding that it continues to invest significant resources in this area.

This is a breaking news story. Please refresh for updates.

Got a confidential news tip? We want to hear from you.

Get this delivered to your inbox, and more info about our products and services.

Original · CNBC

Front Method

Models & Labs · United States

Anthropic bans users from ‘needless abusive or cruel behavior’ towards Claude

Credited to The Guardian · theguardian.com · 8h ago

Photograph · The Guardian

A spokesperson behind the tech company’s AI chatbot has not yet specified what counts as abusive or cruel content

Anthropic has barred users from exhibiting “sustained and needless abusive or cruel behavior” toward its models, as the company’s leaders continue to ponder machine consciousness.

The Verge first reported the change in policy.

A spokesperson for Anthropic, which is behind AI chatbot Claude, did not immediately respond to an inquiry about what could be considered “abusive or cruel”.

In an online user policy, the San Francisco-based company noted that its ban would not apply to common user frustrations, model testing or “dark creative themes”.

Anthropic’s usage policy previously contained restrictions on abusive conduct.

The company’s large language models have the ability to end a conversation if a user is being persistently harmful. When that feature rolled out last August, the company framed it as a safeguard for AI’s welfare.

“We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention,” according to its website.

The notion of AI systems being conscious has been highly polarizing in and outside the tech world.

Anthropic CEO Dario Amodei has said he cannot rule out the possibility.

Meanwhile, Sam Altman, CEO of rival OpenAI, has appeared averse to the idea.

“I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue,” he wrote in an X post, days after the New York Times reported on Anthropic leaders’ extensive conversations with religious scholars.