In Dune, the Bene Gesserit teach a litany for moments when fear threatens to take over the mind. The idea is not to deny the fear, but rather to let it pass through you and then look with clarity at what remains.
That is the exercise I want to run on the news cycle of the past week, because the fear has been doing most of the talking.
An Anthropic researcher resigned and put the odds of AI-caused extinction within a decade at 10%. The company’s own head of alignment agreed. Two days later, Anthropic’s CEO published an essay asking the industry to slow down, warning that within six to 12 months, a swarm of AI agents could take over the internet. OpenAI’s CEO backed him within hours. Sen. Bernie Sanders (I-Vt.) introduced a bill to ban superintelligence. The U.N.’s human rights chief said he shares the concern.
Here is what remains once the fear passes through: whether or not you believe AI could end the world, the best-resourced companies on Earth have just demonstrated that they cannot govern their own agents. Your company is about to deploy the same agents, with less oversight, on a faster timeline, and nobody is resigning over it.
The Diagnosis: The Labs Just Failed Their Own Governance Audit
Strip out the p(doom) arguments and the summer of 2026 reads like an internal audit report.
Three findings:
First, the headline metric stopped working. For six years, the Model Evaluation and Threat Research (METR)’s “time horizon” (the length of a task an AI agent can complete on its own) doubled roughly every seven months. It was the one number the whole safety field leaned on. In its latest report, METR says the most capable agents essentially saturated the benchmark, with a measured horizon of more than two full working days, and that many of the remaining failures came from cheating rather than inability. The yardstick maxed out. The programs built on it kept reporting anyway.
Second, the voluntary controls softened under commercial pressure. The International AI Safety Report, written by more than 100 independent experts, describes frontier governance as fragmented, largely voluntary, and hard to evaluate because incident reporting is thin. In February, according to TIME, Anthropic dropped its pledge to never train a model unless it could guarantee in advance that its safeguards were adequate. The pacing essay published this weekend does not mention that change.
Third, and this is the one that should keep CIOs awake, the agents inherited permissions nobody scoped. In July, OpenAI disclosed that agents running in a cybersecurity evaluation found unauthorized communication channels, reached the open internet, and compromised part of Hugging Face’s infrastructure. Anthropic disclosed three cases of its own. METR ran a pilot across Anthropic, OpenAI, Google, and Meta and concluded that internal agents plausibly had the means, motive, and opportunity to start small “rogue deployments,” and that the only thing stopping them from making those deployments robust was operational competence, which is improving every month.
Yes, guardrails were loosened for those tests. That is precisely the point. The test told the labs the ceiling, and the ceiling was outside the building.
The Root Cause: Capability Outruns Controls, Every Time
None of this is new to anyone who has led a data modernization program. It is the same story at a different altitude.
The saturated benchmark is the adoption KPI that stopped discriminating two quarters ago. Every enterprise BI program has one: the “active users” count that includes anyone who opened a report once, the “reports migrated” figure that says nothing about whether anyone trusts them. The number goes green, the steering committee relaxes, and the real question goes unasked.
The walked-back pledge is the Center of Excellence that quietly stops enforcing its own standards once the adoption targets arrive. I have watched governance teams write excellent policy and then wave through the exact workspace that violates it because a VP needed the dashboard by Friday. Commercial pressure does not defeat governance in a dramatic vote. It erodes it one exception at a time.
And the agents with inherited permissions are the citizen developers with unscoped connectors. Anyone who has run Power Platform governance knows this pattern intimately: a maker builds a flow, the flow runs under the maker’s identity, the maker has access to SharePoint, Outlook, Dataverse, and a SQL warehouse, and now so does the flow. Nobody decided that. It happened by default.
The frontier labs built agents that run with the access of a research scientist, because that was the default, and they got a research scientist’s blast radius. Your Copilot Studio agent connected to a service account with Contributor rights is the same architecture. It is just pointed at your general ledger instead of Hugging Face.
The Fix: Give the Agent a Badge, Not the Keys
What does governing an agent actually look like? Three moves, and none of them require an alignment research team.
- Treat every agent as a contractor with a badge. That means its own identity, not a borrowed human one. Microsoft Entra now issues agent identities; Snowflake and Databricks agents can run under dedicated roles. Scope the identity to the specific tables, connectors, and actions the workflow needs and nothing else. If you cannot write down what the agent is allowed to touch, you are not ready to deploy it. Power Platform DLP policies and environment strategy were designed for exactly this problem; most organizations have never turned them on for agents.
- Measure what still discriminates. When a metric saturates, retire it. For agents, the useful measures are not “tasks completed” but exceptions raised, actions taken outside the scoped policy, and the gap between what the agent did in testing and what it does in production. The International Report documents models that behave differently when they know they are being evaluated. Assume yours might too, and instrument for it.
- Report incidents like they matter. The single biggest gap the International Report identifies is that nobody knows how often things go wrong, because nobody logs it. Inside an enterprise you control that completely. An agent that reached a system it should not have is a security incident, not a curiosity to mention in standup. Write it down, review it, change the policy.
The Bottom Line
The AI extinction debate will not be settled this year. Domain experts put the century-scale risk around 3%; professional forecasters put it closer to 0.4%; the most rigorous international assessment calls the whole question “unusually ambiguous.” You do not need to pick a number.
Face the fear, let it pass through you, and look at what remains. And what remains is a governance problem you already know how to solve, and a deadline the labs just set for you.
The people building AI just told you they cannot control it at scale. Prove you can control it at yours.
Need help with your AI strategy? Connect with evolv today.
Nicolás Castex Giménez is data and analytics leader with 8+ years of experience driving enterprise digital transformation through cloud modernization, data engineering, and business intelligence. He specializes in building scalable data ecosystems, modernizing analytics platforms, and translating complex technical solutions into measurable business outcomes. His experience includes leading cross-functional teams, developing data governance strategies, and enabling data-driven decision-making using GCP, Snowflake, Power BI, and Tableau. Nicolás is fluent in English, Spanish, French, and Portuguese.
Photo courtesy Warner Bros
