Safety testing needs a chain from model capability to institutional control (2026)

Safety claims need a chain from model capability to institutional control

Three recent primary-source documents put safety at three different layers. A new arXiv position paper treats agent security as a property of authorization context and continuous trajectories. Anthropic argues that sufficiently capable models should face mandatory pre-release testing whether their weights are open or closed. The EU's AI Omnibus assigns duties, timelines, sandboxes, and oversight to providers, deployers, high-risk systems, and public institutions. 1 2 3

These proposals are compatible, but they do not answer the same question. A model-level test can tell us that a capability is present. It cannot, by itself, tell us whether a later action was authorized in context. A legal duty can assign responsibility to a provider or deployer. It does not automatically produce the logs, monitors, or pause authority needed to control a long-running agent. The missing engineering problem is the connection between those layers.

Source	Safety object	What it helps establish	What it leaves open
Siu et al., arXiv:2607.22024	An agent's authorization context across its trajectory	Whether commands, goals, actions, and information flows remain authorized	A validated deployment metric and evidence about real-world prevalence; it is a position paper 1
Anthropic's open-weights position	The capability level of a model before release	A proposed common trigger for cyber, biological, and alignment testing across open and closed models	Who runs the tests, what protocol is binding, and what consequence follows a failure 2
EU AI Omnibus	Providers, deployers, high-risk systems, and supervisory institutions	Legal duties, implementation dates, regulatory sandboxes, and expanded AI Office oversight	A universal test for trajectory-level authorization or a portable alignment guarantee 3 4

Agent security starts with context, not the action string

The paper by Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Chenguang Wang, and Dawn Song was submitted to arXiv on 24 July 2026 and is labelled an ICML 2026 Position Paper. Its central example is deliberately ordinary: a command to delete user data could be a routine administrative request or a prompt-injection attack against a production system. The text of the command does not distinguish the two. The authorization context does. 1

The authors propose four properties that must hold together and be checked continuously across the agent's trajectory:

Source Authorization: who issued the command, and were they authorized to do so?
Task Alignment: is the objective the agent is pursuing authorized?
Action Alignment: does each action serve that objective?
Data Isolation: does information stay within the relevant privilege boundaries?

This reframing changes the diagnosis of familiar attacks. Indirect prompt injection becomes a source-authorization failure. Direct prompt injection and some forms of goal drift become task-alignment failures. Tool misuse becomes an action-alignment failure, while memory poisoning and cross-context leakage implicate data isolation. The value of the framework is that it gives each failure a different control target instead of treating every unsafe outcome as a content-classification problem. 1

The paper reports a manual inspection of all 45 injection tasks in AgentDojo and WASP. The authors say each task's action content could also be plausible in a legitimate authenticated workflow. That observation supports their structural criticism of content-only evaluation: a benchmark can label an action malicious because of the scenario in which it appears, even though the same action is normal under a different authorization state. 1

The limitation matters. This is a conceptual position paper, not a deployment study or a validated scoring protocol. It identifies what an evaluation would need to observe, but it does not show that current systems can reliably infer task authorization, detect every cross-boundary data flow, or measure how often these failures occur in production. Its contribution is a sharper object of evaluation, not a finished measurement regime.

Anthropic makes capability the release trigger

Anthropic's 27 July statement on open-weights models rejects a blanket ban on the category. It argues that models without dangerous capabilities are a public good, and proposes three measures instead: restrict access to powerful chips and chipmaking equipment, target industrial-scale distillation, and require all sufficiently capable models, whether open or closed, to undergo mandatory safety testing before release. The proposed tests would cover cyber, biological, and alignment risks. 2

That is a cleaner policy trigger than a rule based on release format. Open weights affect whether a provider can monitor use, apply safeguards, or withdraw a model after release. But the statement argues that these properties do not settle the underlying risk in advance. Testing should determine whether a model is dangerous, and should apply regardless of whether the weights are public. Anthropic also says an effective regime would need to be global, including participation from China. 2

The proposal is useful because it separates two questions that are often bundled together. The first is how much capability a model has. The second is whether the way it is distributed makes misuse easier to monitor or reverse. A capability threshold can decide when testing is required; it cannot decide whether a particular deployment has preserved source authorization, data isolation, or a human ability to intervene.

There is also an evidence boundary. This is Anthropic's policy position, not an independently validated international testing system. The statement does not specify a binding threshold for "sufficiently capable," the test owners, the evaluation protocol, or the legal consequence of a failed result. Those omissions do not invalidate the proposal, but they prevent readers from treating it as an operational standard already in force.

The EU law assigns actors, dates, and oversight

The EU AI Omnibus entered into force on 27 July 2026. The European Commission describes it as a targeted simplification of the AI rulebook that extends some compliance timelines, expands testing opportunities, and preserves safety and fundamental-rights safeguards. The regulation itself is Regulation (EU) 2026/1744, published in the Official Journal on 24 July. 3 4

For high-risk AI, the dates are material. Rules for systems classified under Annex III apply from 2 December 2027, while high-risk AI embedded in physical products under Annex I apply from 2 August 2028. The Omnibus also expands access to regulatory sandboxes, including a possible EU-level sandbox, and extends the AI Office's oversight of some systems built on general-purpose models and systems embedded in very large online platforms or search engines. 3 4

The safety provisions are not limited to model behaviour. The text adds prohibitions covering non-consensual realistic intimate material and child sexual abuse material, permits tightly constrained processing of special-category personal data for bias detection and correction, and clarifies how conformity assessment and cooperation between an initial provider and a later provider should work. These are controls on system use, documentation, and institutional responsibility, not evidence that a model is aligned in every context. 3 4

The delay in some high-risk obligations makes the handoff problem more visible. A regulator may have authority over a system while the technical record needed to reconstruct a long-running agent's decisions is incomplete or inconsistent. Conversely, a lab may have detailed pre-release evaluations while the deployer lacks a reliable way to show that the evaluated authorization context survived a change in tools, memory, users, or backend.

The missing link is a composed safety case

Read together, these documents suggest a four-part reporting discipline for AI safety work:

Name the object. Was the claim about a model's capability, an agent trajectory, an application, or an institution's control over deployment?
Name the protocol. Was the evidence an internal audit, a synthetic benchmark, a pre-release capability test, a sandbox exercise, or an independent assessment?
Name the authority. Which model provider, deployer, platform, regulator, or human operator can pause, restrict, or investigate the system?
Name the consequence. Does a failed test trigger a block, a fallback, a reporting duty, a remediation plan, or a market-access decision?

The order matters. Anthropic's proposal can provide a reason to test a model before release. Siu and colleagues provide a way to ask whether an agent's actions remain authorized after the run begins. The EU framework provides actors and procedures that can make evidence requestable and intervention possible. None of the three, alone, establishes that the entire chain works after a model, tool set, memory state, or deployment authority changes.

For someone entering the field, this is the practical distinction to carry forward: a safety score describes the result of a protocol at a particular layer. It is not a portable property of the whole system. The strongest safety claims will make the handoff explicit, showing how model capability testing feeds deployment controls, how trajectory evidence reaches an accountable authority, and what happens when the evidence fails.

Safety testing needs a chain from model capability to institutional control