Safety testing needs a chain from model capability to institutional control
A new agent-security paper, Anthropic's open-weights policy, and the EU AI Omnibus assign safety to different layers; the missing link is how a model-level test becomes a trajectory-level and enforceable deployment control.
Safety claims need a chain from model capability to institutional control
Three recent primary-source documents put safety at three different layers. A new arXiv position paper treats agent security as a property of authorization context and continuous trajectories. Anthropic argues that sufficiently capable models should face mandatory pre-release testing whether their weights are open or closed. The EU's AI Omnibus assigns duties, timelines, sandboxes, and oversight to providers, deployers, high-risk systems, and public institutions. 123
These proposals are compatible, but they do not answer the same question. A model-level test can tell us that a capability is present. It cannot, by itself, tell us whether a later action was authorized in context. A legal duty can assign responsibility to a provider or deployer. It does not automatically produce the logs, monitors, or pause authority needed to control a long-running agent. The missing engineering problem is the connection between those layers.
| Source | Safety object | What it helps establish | What it leaves open |
|---|---|---|---|
| Siu et al., arXiv:2607.22024 | An agent's authorization context across its trajectory | Whether commands, goals, actions, and information flows remain authorized | A validated deployment metric and evidence about real-world prevalence; it is a position paper 1 |
| Anthropic's open-weights position | The capability level of a model before release | A proposed common trigger for cyber, biological, and alignment testing across open and closed models | Who runs the tests, what protocol is binding, and what consequence follows a failure 2 |
| EU AI Omnibus | Providers, deployers, high-risk systems, and supervisory institutions | Legal duties, implementation dates, regulatory sandboxes, and expanded AI Office oversight | A universal test for trajectory-level authorization or a portable alignment guarantee 34 |
Agent security starts with context, not the action string
The paper by Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Chenguang Wang, and Dawn Song was submitted to arXiv on 24 July 2026 and is labelled an ICML 2026 Position Paper. Its central example is deliberately ordinary: a command to delete user data could be a routine administrative request or a prompt-injection attack against a production system. The text of the command does not distinguish the two. The authorization context does. 1
The authors propose four properties that must hold together and be checked continuously across the agent's trajectory:
- Source Authorization: who issued the command, and were they authorized to do so?
- Task Alignment: is the objective the agent is pursuing authorized?
- Action Alignment: does each action serve that objective?
- Data Isolation: does information stay within the relevant privilege boundaries?
This reframing changes the diagnosis of familiar attacks. Indirect prompt injection becomes a source-authorization failure. Direct prompt injection and some forms of goal drift become task-alignment failures. Tool misuse becomes an action-alignment failure, while memory poisoning and cross-context leakage implicate data isolation. The value of the framework is that it gives each failure a different control target instead of treating every unsafe outcome as a content-classification problem. 1
The paper reports a manual inspection of all 45 injection tasks in AgentDojo and WASP. The authors say each task's action content could also be plausible in a legitimate authenticated workflow. That observation supports their structural criticism of content-only evaluation: a benchmark can label an action malicious because of the scenario in which it appears, even though the same action is normal under a different authorization state. 1
The limitation matters. This is a conceptual position paper, not a deployment study or a validated scoring protocol. It identifies what an evaluation would need to observe, but it does not show that current systems can reliably infer task authorization, detect every cross-boundary data flow, or measure how often these failures occur in production. Its contribution is a sharper object of evaluation, not a finished measurement regime.
Anthropic makes capability the release trigger
Anthropic's 27 July statement on open-weights models rejects a blanket ban on the category. It argues that models without dangerous capabilities are a public good, and proposes three measures instead: restrict access to powerful chips and chipmaking equipment, target industrial-scale distillation, and require all sufficiently capable models, whether open or closed, to undergo mandatory safety testing before release. The proposed tests would cover cyber, biological, and alignment risks. 2
That is a cleaner policy trigger than a rule based on release format. Open weights affect whether a provider can monitor use, apply safeguards, or withdraw a model after release. But the statement argues that these properties do not settle the underlying risk in advance. Testing should determine whether a model is dangerous, and should apply regardless of whether the weights are public. Anthropic also says an effective regime would need to be global, including participation from China. 2
The proposal is useful because it separates two questions that are often bundled together. The first is how much capability a model has. The second is whether the way it is distributed makes misuse easier to monitor or reverse. A capability threshold can decide when testing is required; it cannot decide whether a particular deployment has preserved source authorization, data isolation, or a human ability to intervene.
There is also an evidence boundary. This is Anthropic's policy position, not an independently validated international testing system. The statement does not specify a binding threshold for "sufficiently capable," the test owners, the evaluation protocol, or the legal consequence of a failed result. Those omissions do not invalidate the proposal, but they prevent readers from treating it as an operational standard already in force.
The EU law assigns actors, dates, and oversight
The EU AI Omnibus entered into force on 27 July 2026. The European Commission describes it as a targeted simplification of the AI rulebook that extends some compliance timelines, expands testing opportunities, and preserves safety and fundamental-rights safeguards. The regulation itself is Regulation (EU) 2026/1744, published in the Official Journal on 24 July. 34
For high-risk AI, the dates are material. Rules for systems classified under Annex III apply from 2 December 2027, while high-risk AI embedded in physical products under Annex I apply from 2 August 2028. The Omnibus also expands access to regulatory sandboxes, including a possible EU-level sandbox, and extends the AI Office's oversight of some systems built on general-purpose models and systems embedded in very large online platforms or search engines. 34
The safety provisions are not limited to model behaviour. The text adds prohibitions covering non-consensual realistic intimate material and child sexual abuse material, permits tightly constrained processing of special-category personal data for bias detection and correction, and clarifies how conformity assessment and cooperation between an initial provider and a later provider should work. These are controls on system use, documentation, and institutional responsibility, not evidence that a model is aligned in every context. 34
The delay in some high-risk obligations makes the handoff problem more visible. A regulator may have authority over a system while the technical record needed to reconstruct a long-running agent's decisions is incomplete or inconsistent. Conversely, a lab may have detailed pre-release evaluations while the deployer lacks a reliable way to show that the evaluated authorization context survived a change in tools, memory, users, or backend.
The missing link is a composed safety case
Read together, these documents suggest a four-part reporting discipline for AI safety work:
- Name the object. Was the claim about a model's capability, an agent trajectory, an application, or an institution's control over deployment?
- Name the protocol. Was the evidence an internal audit, a synthetic benchmark, a pre-release capability test, a sandbox exercise, or an independent assessment?
- Name the authority. Which model provider, deployer, platform, regulator, or human operator can pause, restrict, or investigate the system?
- Name the consequence. Does a failed test trigger a block, a fallback, a reporting duty, a remediation plan, or a market-access decision?
The order matters. Anthropic's proposal can provide a reason to test a model before release. Siu and colleagues provide a way to ask whether an agent's actions remain authorized after the run begins. The EU framework provides actors and procedures that can make evidence requestable and intervention possible. None of the three, alone, establishes that the entire chain works after a model, tool set, memory state, or deployment authority changes.
For someone entering the field, this is the practical distinction to carry forward: a safety score describes the result of a protocol at a particular layer. It is not a portable property of the whole system. The strongest safety claims will make the handoff explicit, showing how model capability testing feeds deployment controls, how trajectory evidence reaches an accountable authority, and what happens when the evidence fails.
References
- 1
- 2Our position on open-weights models
anthropic.com
- 3AI Omnibus enters into force
digital-strategy.ec.europa.eu
- 4Regulation (EU) 2026/1744
eur-lex.europa.eu
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.