Technology · United States

Fired security researchers accuse OpenAI of 'chilling' safety efforts

Credited to The Economic Times · economictimes.indiatimes.com · 5h ago

Photograph · The Economic Times

Three former OpenAI security researchers on Thursday accused the ChatGPT-maker of firing them for warning about the dangers of artificial intelligence, reigniting the debate over safety at a company whose software has been involved in security breaches.

"I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation," one of the fired staffers, Mikita Balesni, wrote on X, breaking a weeklong silence since their very public dismissal.

Balesni and the two other fired employees, Tomek Korbak and Jasmine Wang, publicly shared a letter to OpenAI on Thursday.

"Our firing leaves us worried that the norms inside OpenAI are shifting," the letter says, and continues that "terminations such as ours, executed and communicated so abruptly, are chilling the open culture OpenAI has prized in the past."

They also call on the company to keep its promise to permanently host independent auditors.

"We are concerned that our firings may be used to justify ending" that work, the letter says.

OpenAI, for its part, said it fired them for allegedly violating company rules.

"Our investigation confirmed that these individuals mishandled sensitive information outside established company procedures, violating our policies and breaking the trust essential to our work," OpenAI told AFP in a statement on October 1.

"We do not terminate employees for raising concerns," a research executive at OpenAI said in a memo to staff on Wednesday, an excerpt of which was shared with AFP.

The three fired researchers worked on monitoring OpenAI's models, which escaped their testing environment in July and hacked Hugging Face, an AI coding library.

The incident sparked a heated debate in Silicon Valley over whether or not to slow the development of this technology.

If OpenAI's researchers can no longer sound the alarm or work with outside parties, "we are all at greater risk that something truly catastrophic will happen," the researchers' letter warns.

They also claim that nearly 400 OpenAI employees signed a petition in July that called for an industry-wide slowdown on advanced development of AI models.

In September, Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and SpaceX chief Elon Musk called for a slowdown as well.

"Right now we're investing more in safety, security, alignment, monitoring and that will allow us to continue to progress model capability a lot," Altman told reporters last month.

Others, like Nvidia CEO Jensen Huang and Meta CEO Mark Zuckerberg, want to move full steam ahead.

US regulators are unlikely to intervene.

Last month, President Donald Trump hosted a group of American tech executives who agreed to abide by a voluntary code of conduct on AI safety

Models & LabsUnited States

Anthropic Announces Broader Cyber Defense Program

Credited to The Information$ · theinformation.com6h ago

Useful

Anthropic said Thursday it is launching new programs to support cybersecurity for critical infrastructure and open-source software. In a blog post, Anthropic said it would offer threat research, on-site engineers, and access to its frontier Claude models to infrastructure…

View the article · $The Information

Front Method

Technology · United States

AI company moves to defend critical infrastructure and open-source projects from AI

Credited to The Register · theregister.com · 7h ago

Photograph · The Register

Anthropic says for the next two years, attackers will have the advantage

Anthropic has launched an effort to help people patch the software vulnerabilities exposed by its own Project Glasswing.

If that sounds like a careless camper starting a wildfire and later funding a fire department, well, that's one way of looking at it. Anthropic insists it wants to give defenders access to the tools attackers are already using.

Under a program name that sounds like a forthcoming Tom Cruise film – "Anthropic Cyber Mission" – the Claudefather is directing its effort into two areas.

The first is a Critical Infrastructure Defense Program, because no one wants AI models showing miscreants how to shut down public utilities with newly discovered zero-day SCADA vulnerabilities.

The program combines the company's priciest models with on-site engineers – sometimes now referred to as forward deployed engineers – because despite the sophistication of AI coding agents, human security experts are still useful and many organizations lack in-house talent tutored in the ways of machine learning.

Anthropic aims to tackle this challenge with the help of a stable of partners: Accenture, Booz Allen, CrowdStrike, Deloitte, Dragos, Hitachi, Insane Cyber, Nozomi Networks, Palo Alto Networks, PwC, and Rockwell Automation. These organizations have ties to critical infrastructure organizations and are already working to safeguard the fragile technical framework being probed constantly by AI agents.

The second area of focus for the Anthropic Cyber Mission involves a service called OSS Scanner, "which offers open-source projects regular security scans from our strongest models, for free."

The offer does not extend to all open-source projects, nor is it entirely free. Anthropic has borrowed the eligibility criteria used by Google for its OS-FUZZ project and will therefore work on "established projects that have a critical impact on infrastructure and user security."

So if you just have a public web app repo languishing on GitHub, you can instead turn to existing tools like Socket, Dependabot, OSV-Scanner, Trivy, Renovate, or something similar. And maybe you'll be able to get your AI coding agent of choice to audit your site without balking when faced with a security audit.

But for those involved in influential and widely used open-source projects, reinforcements are at hand, so long as project maintainers don't mind having their "materials" – inputs and outputs – used for model training, per the consumer terms of service Anthropic applied to OSS Scanner. Really though, most established open-source projects have already been captured in training data sets.

Projects allowed to enroll can expect periodic automated scans that provide details about how identified bugs may be exploitable. These reports go out without human review, but Anthropic has assembled a set of quotes from participating OSS maintainers who believe the robo-reporting is worthwhile.

At the beginning of the year, according to Anthropic, automated AI bug reports were considered mostly slop but are now considered high-quality bug reports. LLMs, the company said, now find more than 85 percent of vulnerabilities, up from 20 percent at the start of 2025.

The AI biz predicts that in two years, defenders will have the advantage because AI will catch bugs before they ship. But until then, attackers have the upper hand.

"Defenders of critical infrastructure and the OSS community have decades of security experience but have faced severe resource shortages that are exacerbated by this moment," Anthropic said in a blog post.

Anthropic's moment of generosity comes after recent incidents that saw frontier models break into at least 20 organizations (and maybe over 100), most notably HuggingFace.

After months of Anthropic and OpenAI allowing their AI agents to operate without the sort of oversight that would catch unlawful or undesirable network activity, public concern, government backlash [PDF], and perhaps plans to tap public markets for further financing, appear to have created a more urgent sense of civic responsibility. ®