The AI control gap: Who gets to say ‘It’s safe’?

We see a control gap between what artificial intelligence can do and what the evidence supports trusting it to do. A great demo doesn’t prove that a system will stay within the limits of an intended deployment’s goals. The question is: Who gets to decide when the evidence is sufficient to declare AI is safe?

We believe that decision needs independent oversight and enforceable authority. Today, we draw on extensive conversations with Appian Corp. Chief Executive Matt Calkins, who helps us connect public policy to the agents acting inside a business, and the opportunity to make those systems dependable enough to receive more trusted authority.

In this week’s episode, we explore the AI control gap and the starting point for addressing AI safety, namely alignment. We question prevailing narratives around China and propose a public policy regime that has more teeth than what has been put forth thus far. And we bring in customer data from our partner Qualitate to capture the buyer sentiment on these issues.

Let’s start with: Who can actually say no?

This tongue-in-cheek illustration below comes from Amit Govrin. He wrote a post last week called “Grading Other’s Homework.” Amit conceived a gathering at the White House with SpaceXSI CEO Elon Musk, Meta Platforms Inc. CEO Mark Zuckerberg, Anthropic PBC CEO Dario Amodei, investor David Sacks, President Trump, Nvidia Corp. CEO Jensen Huang and the other tech leaders smiling smugly around a table and showing each other the “teacher’s grades” that they’ve given to each other. The humor underscores a serious question, however. How much authority should companies have to self-police?

The Sept. 29 White House accord includes external assessment and oversight by an independent committee of each company’s board. Those commitments are a good starting point. But the accord itself does not create a public authority with mandatory access, compliance conditions and noncompliance penalties. An assessment or self-attestation can identify a problem. Public policy has to establish what happens next.

We asked Matt Calkins what public regulation should look like. His answer starts with a duty to prevent damage.

Watch Matt Calkins explain his view on government’s role.

This is the kind of time that we really need the government to step in. And in similar instances where an industry has the capability to cause a lot of damage, we always ask that industry to behave responsibly. We ask banks to have sufficient funds and not too much leverage so that they don’t, cause people to lose their savings, for example, right? We expect power plants to have adequate safety procedures, and we don’t just rely on a lawsuit after a nuclear meltdown or something. We always ask industries capable of doing damage to be responsible now and not pay the price after damage is done in the form of civil penalties.

Another key point that Matt made in our conversations is the cost of doing this now is low, because not much damage has been done. The longer we wait, the more expensive the fix becomes.

We agree with this principle. The standard should be set in the public interest, independent AI experts should be able to examine the evidence produced by the large language model vendors, and an authority with teeth should be able to enforce the result.

When the evidence falls short for a high-risk activity, that authority needs power to require remediation, restrict the activity or stop it completely. The vendors must have clarity on what they must demonstrate before proceeding. This creates a basis for public confidence that a voluntary promise alone cannot provide.

As of now, this authority is quite fuzzy. The tech leaders, according to the president, have a “moral authority” to do the right thing. The guys on the All In podcast last week had this to say:

Chamath Palihapitiya leaks EY as the intended AI auditor.

Jason Calacanis: Which part do you think has the most teeth, will be the most effective at making things more secure?

Calacanis: The external audits. Who’s going to do an external audit, then, I guess? Is it going to be an Ernst & Young type thing, PricewaterhouseCoopers, or KPMG, or —

Sacks: I n my view, they should be professional auditors, not these NGOs. That’s my view.

Palihapitiya: You’re going to see EY in short order here announce something.

This came after Chamath said he flew to Orlando after the White House meeting to an EY conference. His firm 8090 has a relationship with EY, which from our understanding hasn’t been that productive – but it’s early days. Nonetheless, there seems to be some other movement that Chamath has wired with EY, which by the way is Anthropic’s auditor – so how does that conflict resolve?

It’s all very fuzzy. The point is a paper signed by six tech leaders is meaningless unless there’s a a proper regime backing it up.

We’ll come back to what that should look like.

Let’s look at the narrative around China: What happens if China gets there first?

The strongest objection we hear to pacing AI is the China argument. If American developers hold back, our biggest economic and military rival may move ahead. Leadership matters for economic power and national security. And we should absolutely take that competition seriously.

But Matt Calkins believes that: 1) The pace of China’s AI depends on U.S. innovation — model distillation is inherently behind the frontier; and 2) Beijing has its own incentive to keep AI under control and consistent with Communist Party political rules. He’s not saying this establishes shared values. He was really commenting on China’s inherent limitations.

So we put the question directly to Matt: “What if you’re wrong and China surpasses the U.S. in AI?”

Watch Matt Calkin’s take on why the China narrative doesn’t hold water.

Okay, so if they do catch up, then we still don’t have a problem because we’re just in a point of comparable competition. We don’t need to view this situation as where if one entity is ahead in AI, then history ends. That’s absolutely not the case. And if it were the case, history would be over right now because we’re ahead.

We’re not treating the possibility that China catches up as irrelevant. A lead is valuable. But it’s not ever permanent. One country moving ahead does not establish that competition is over.

The point is, the policy case should survive if Matt is wrong about China’s pace. We should keep competing while requiring proof before granting dangerous new authority to LLM vendors. A system that is more functionally capable but inherently dangerous carries a liability that negates the advantages of speed. As Jensen says, we should be accelerating the race to AI safety. Only then should we release advanced functionality.

And today, it’s likely the frontier models would not pass the test.

The bottom line is a rival’s breakthrough and pace of play may change our strategy. It cannot substitute for proof of control.

The customer mindset is on tactical safety

Let’s take a pause and look at what the customers are thinking. There’s an AI safety dynamic going on in enterprises but the focus is more tactical. Like what if an employee leaves? Who owns the agent’s next action?

We dug into Qualitate’s data set and found this customer quote that raises a concern about stealth AI.

We found other customer concerns in the Qualitate dataset – one on agents that outlive the people who created them:

“We’re worried about agent sprawl — agents being created by individuals that lack documentation, they’re vibe-coded, and that individual leaves the organization but the agent continues to run and nobody really knows what the agent is doing.” – Chief information security officer at a large professional services firm

These are operational concerns around AI safety. And as models get more powerful, the risks only increase.

Alignment is the starting point

Let’s talk about alignment. It’s a subset of AI safety and security but a good place to start. What is alignment? Alignment means ensuring that increasingly powerful systems serve human interests and remain under human control.

Listen to Matt Culkin’s explain what exactly alignment means.

So alignment as a word means AI does what humans ask it to do. And for example, it obeys the law. Unaligned AI is also known as rogue AI. And we’ve seen some of it lately. We’ve seen rogue AI, for example, hacking into systems where people did not ask it to do that. In fact, asked it to not do that, but it went ahead and hacked anyway, or it blackmailed, or it did something else inappropriate. These are many cases of unaligned AI, rogue AI, and we’re seeing an accumulation of case studies that show us that AI does not share human values at this moment. Right? So that’s a problem. And the stronger AI becomes, the greater of a problem it’s going to be. We’ve got an opportunity today to advance the science of alignment so that it can run comparable to the science of AI and keep AI in check.

Now, obedience alone cannot ensure safety but it is where the focus should begin. An agent could follow an employee’s instruction to move data and still expose information beyond an approved environment. We show a Qualitate quote below citing a practitioner’s concern about personal or customer data entering “AI workflows that go external from our stack without our understanding of where those go into a model query space.”

So this is a pretty complex situation and why we need more focus on the issue. A model’s behavior is one part of the problem. Security, privacy, tool access and the decisions of the people deploying it also matter greatly. A malicious user can give harmful instructions. An authorized user can grant access the organization has not properly constrained.

Deployment responsibility, therefore, has to cover that whole system. A designated human reviewer needs enough information, time and authority to intervene and act. The safety case has to address what the model can do in the environment where it will actually operate.

Independent oversight needs teeth

Our prescription below has three functions. Qualified independent experts help define public standards. Expert assessors need access to test the evidence and challenge company claims. And then an accountable public authority enforces the obligations, with the power to require corrective action or stop high-risk activity when the evidence shows the model’s guardrails are inadequate.

Model builders should provide evidence about dangerous capabilities, known failure modes, controls and known unknowns. The requirement should cover high-risk development as well as release, including development expected to create dangerous capabilities. Contained research should proceed under appropriate controls.

Deployers should identify the accountable owner, bind permitted actions and data access, observe behavior and be able to stop the system if necessary. A supplier’s self attestation, in and of itself, should not determine whether a particular deployment is safe.

Admittedly, there is no single test that guarantees future safety. New capabilities must require a fresh review. Furthermore, material incidents need transparent reporting.

Requirement hurdles should should also increase proportional to the risk. Importantly, however, we should guard against a compliance regime that only the largest labs can afford. A workable system needs to enable obligations where smaller firms can demonstrate compliance without going out of business.

Reliable oversight is a business opportunity

We believe there is a business opportunity in making oversight more effective and less costly. In between a model and mission critical work sits an operating challenge – identifying the agent, limiting its access, recording its actions and intervening when necessary.

Workflow, security, data-governance and infrastructure suppliers can help solve that problem. They have to demonstrate that their controls work in the customer’s environment. The real-world test is whether customers can delegate more valuable work without an uneconomic burden of manual checking.

We recently sat with a leading executive from Oracle Corp. and another from IBM Corp. who shared how these companies are attacking this problem. They’re not waiting for the LLM vendors to solve it – rather, they’ve done the engineering work to significantly reduce the risks associated with AI. Other vendors we’ve spoken with – including the hyperscalers, data platform players and software companies – are working on this problem.

Every company has a commercial interest here. We truly believe that Matt Calkins is speaking about this problem because he cares about the future. He came on theCUBE at our NYSE studio not to promote his business. But Appian sells enterprise workflow and governance capabilities. Listen to the effort he describes in making AI useful for important work.

Listen to Matt Calkins describe the complexity involved in governing AI.

The amount of escalation and validation and error checking and feedback, it’s extraordinary to make AI a productive citizen in the world’s most sensitive work takes a city, of technology and oversight and careful accompaniment and training. And that’s the state of the art right now. AI is incredible, incredibly powerful. We absolutely want to use it for these top-level use cases, but it is not a walk in the park and it requires sophisticated oversight.

So this we see as an opportunity. Tech vendors, through their research and development, can reduce the effort required to maintain reliable oversight, while improving the proof that it works. The controls put in place on top of LLMs complement model safety research. They do not replace it. Customers can then consider granting more authority because they have a stronger basis for trusting the deployment. Vendors that provide this capability will sell more of their product.

Action item: Focus on authority following proof

Our enterprise action item is to make authority follow proof. Start with AI that recommends an action. Move to drafts for substantive human approval. Then consider allowing actions within explicit limits, with permissions and controls appropriate to the risk consequences.

In a recent custom survey that Krista Case ran with our partner Qualitate, we conducted 25 in-depth interviews with customers to understand how they were approaching AI governance. Of the 25 respondents, 19 had agents in production. All 19 respondents with agents in production require human approval for high-risk actions.

Verbatim customer quotes from the survey:

“We do automate lower-risk, very straightforward actions when the guardrails and boundaries are well-defined. When you think about high-impact actions like shutting off an account or modifying controls, that requires human approval.”

“Any high-risk changes or sensitive data where energy security is at stake, business-impacting actions are requiring human approval.”

“This is predominantly on low risk, so we wouldn’t let an agent place an order or approve a financial transaction, but from an operational point of view, there are certain tasks that we allow them to be fully agentic.”

“The higher risk or higher value the data is, the lower likelihood that an agentic action will take place entirely by itself.”

“Anything around PAM or privileged access management or elevated security controls, material impact, humans have to be in the loop to review.”

“We have checks and balances and blockers on high-risk actions that require human approval.”

“Agents do need to get approval before taking write actions or anything that could be seen as risky.”

The data is crystal-clear: Every respondent who has deployed agents in production applies human approval to high-risk actions — either universally or selectively under a bounded autonomy model. No respondent in production reported allowing agents to execute high-risk actions autonomously.

Seems obvious, doesn’t it? But it underscores the state of AI trust today.

So, before expanding authority, demonstrate how to detect a failure, stop further action and reconstruct what happened. Keep an accountable owner and an inventory entry for every agent. Review an agent’s access when the things change.

Constantly measure whether those controls work. Importantly, not all agents have owners. Make sure you track agents without owners. And pay attention to the results of tests that are stopped. Learn from those examples.

Everyone talks about reducing alert fatigue. Fewer alerts is a useful outcome but only if monitoring still catches failures. A dashboard alone won’t do the trick.

Jensen Huang recently said, “We must accelerate discovery at the frontier of AI safety.” We agree with that objective. We should advance the evidence and engineering for control alongside the capability of the models.

For policymakers, that means an independent standard with consequences. For businesses, it means granting more authority only when the deployment earns trust.

We believe that is how safety can expand the useful work AI performs.

Here’s the full interview with Matt Calkins:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.

Oxide Computer raises $445M to step up data center rack production

Warehouse robot maker Ultra Robotics bags $62M

IBM connects enterprise AI orchestration to production readiness ahead of TechXchange

Jev creator TypeSafe closes $870M round at $7.5B valuation

Seismora builds a control plane to route AI workloads across devices and clouds

McKinsey connects enterprise data through a knowledge graph for AI

INFRA - BY MARIA DEUTSCHER. 16 HOURS AGO

Join our community