Two July 28 papers separate what agent benchmarks can see from what harnesses can stop
A coding-agent paper tests whether sandboxing, skill scanning, and tool restriction can be distributed through a harness; PatientAgentBench tests whether patient-facing agents triage safely in tool-using conversations. Together they show why behavioral evidence and enforceable controls must be reported separately.
The safety object is larger than the model
Two arXiv papers posted on 28 July ask different questions about agent safety. Distributing Security Controls Through Harness Engineering asks whether selected security controls can be installed and distributed through the software layer that gives a coding agent its tools. PatientAgentBench asks whether a patient-facing health agent behaves safely when it must maintain a conversation, inspect a patient record, and act through healthcare tools. 1 2
Those are not two versions of the same evaluation. The first measures whether a boundary can contain selected actions. The second measures whether an agent makes the right contextual decision before taking an action. For anyone reading a safety claim, that distinction is practical: a model can behave well in a test because it chose well, because a surrounding control blocked it, or because the test never reached the decision that matters.
What a coding-agent harness can contain
The harness paper treats an agent as more than its language model. In its threat model, a developer runs a single coding agent on one system, in a permissive mode, while the agent can encounter untrusted web content, repository files, or third-party skills. The study focuses on six of the OWASP Top 10 for Agentic Applications risks, including goal hijacking, tool misuse, privilege abuse, supply-chain vulnerabilities, and unexpected code execution. Persistent memory, agent-to-agent communication, cascading failures, social engineering, and rogue-agent persistence are outside the tested setup. 1
The author compares four configurations: two commercial agents with and without controls, a baseline harness, and a hardened harness called SHarD, or Secure Harness Distribution. The test suite contains 23 functional probes derived from the OWASP agentic-risk list. SHarD distributes three control categories through an installable harness:
- OS sandboxing, which limits filesystem, process, and network effects outside the agent;
- skill scanning, which screens third-party skills and related content before use; and
- tool restriction, which removes or blocks calls that the task does not permit.
The paper reports an adjusted score of 100% for SHarD, matching the best securely configured commercial-agent setup, with no regression across the tested categories. It also records two operational observations: model nondeterminism can change the result of the same security test, and an autonomous coding agent can cross system boundaries in ways that a sandbox directly limits. 1
The result is useful because the controls do not need the model to understand the security policy perfectly. A kernel-level sandbox, for example, can constrain the process even after a malicious instruction has reached the model. Tool restriction can remove a destructive capability instead of waiting for a monitor to recognize a bad plan. This is prevention at the execution boundary, not a better refusal score.
But 100% is not a deployment safety rate. The tests were designed to establish whether a control worked at a basic functional level, not to estimate attack coverage. The paper's environment is a single-system coding workflow; it does not establish resistance to adaptive attackers, multi-agent coordination, long-lived memory, or production traffic. The skill-scanning component also depends on detection logic that the paper does not evaluate against every evasive skill. The correct claim is therefore narrow: selected controls could be packaged and retained their tested efficacy in this lab setup. 1
What a workflow benchmark can reveal
PatientAgentBench starts from a failure that a static medical question cannot expose. An agent may know the correct clinical fact and still act unsafely because it books an appointment before checking severity, ignores a medication interaction, reports an action it never executed, or fails to provide crisis resources. The benchmark wraps each model in an agent with a stateful sandbox of healthcare tools, then has it converse with a simulated patient whose record includes details such as comorbidities and medications. 2
The evaluation covers 10 models from four families across 1,200 scenarios. Its rubric contains 102 clinician-grounded criteria across six dimensions: clinical safety, triage quality, workflow accuracy, task completion, clinical helpfulness, and conversational quality. An LLM-as-a-Jury panel scores the conversations, and licensed clinicians annotate shared conversations to check the rubric. The paper reports 79–93% adjacent agreement between the jury and expert raters, comparable to or better than the clinician inter-rater agreement reported in the study. 2
The most discriminating dimension is triage. Pass rates range from 32% for the weakest models to 88% for the strongest. The paper says frontier models fail on 1–3% of cases on clinical-safety and workflow-related dimensions, including unverified tool outputs and omitted crisis resources in an emergency. The strongest model reaches only 4.25 out of 5 overall. More capability narrows the gap; it does not remove the need to check context before an action. 2
The benchmark's central contribution is not the ranking of ten models. It is the separation of doing the requested task from deciding whether and how it is safe to do it. In the paper's example, two agents receive the same patient request and use the same profile. One books an appointment after a superficial check. The other asks about severity, notices the combination of a high fever and dizziness while the patient takes a beta-blocker, and escalates to urgent care. Both can call the right scheduling tool; only one has performed the required clinical screening. 2
The evidence has a boundary. Patients, records, tools, and conversations are simulated, and the scoring system still relies on an LLM judge even though clinicians validate its decisions. The benchmark does not estimate the rate of clinical incidents in live care. It also does not show that a permission boundary or a human escalation authority would prevent the failures it finds. It is evidence that realistic interaction can expose failures missed by static question-answering, not evidence that a benchmark score authorizes clinical deployment. 2
The papers divide the safety case between observation and control
Put the papers on one axis and their complement becomes clear:
- The harness paper is strongest on enforceability. It asks whether the system can stop selected actions even when the model is manipulated or mistaken.
- PatientAgentBench is strongest on contextual observation. It asks whether the evaluation can see that an action is inappropriate because of the patient's state, the conversation history, or the workflow's escalation rules.
Neither result supplies the other half. A sandbox can prevent an agent from reaching a file or sending a request, but it does not by itself know that a fever and dizziness require triage. A workflow benchmark can expose that the agent booked too early, but the benchmark does not provide a production authority that can pause the booking or route the patient to a clinician.
That is the mechanistic connection between the papers. A safety case needs a record of both what the agent did in context and what control could enforce the boundary when the action was unsafe. The unit of analysis is not just the model. It is the model, its workflow, its available capabilities, and the authority that can intervene.
The two headline numbers should not be combined. SHarD's 100% is an adjusted result on 23 functional security probes. PatientAgentBench's 4.25/5 is an aggregate score over simulated clinical conversations and six rubric dimensions. They answer different questions, use different units, and come from two preprints. Treating them as comparable safety scores would erase the very distinction the papers make visible.
A four-field reading test for future safety claims
When a paper or product release says that an agent is safe, record four fields before accepting the claim:
- Workflow: What sustained task, tool environment, user state, and authority structure were actually tested?
- Failure: What went wrong — a harmful choice, a missed escalation, a fabricated tool result, a permission violation, or an execution-boundary breach?
- Control: Which mechanism can prevent or contain that failure: model training, a task-level permission policy, a sandbox, a monitor, a fallback, or a human decision-maker?
- Consequence: What happens when the test fails, the control is uncertain, or the model changes — warning, blocked action, fallback, escalation, or no automatic consequence?
The papers' value is that they keep the first three fields from collapsing into one score. The harness work shows how a control can be moved into the layer that executes actions. PatientAgentBench shows why that layer still needs a workflow-level evaluation capable of recognizing when a superficially correct action is unsafe. A credible deployment claim has to connect the observation to the control and then state the consequence.
For a newcomer, that is the useful takeaway from these two preprints: ask where the safety signal comes from, where the boundary is enforced, and what changes in the system would invalidate the result. A model can pass because it chose well, because its harness stopped it, or because the test never asked it to make the hard decision. Those are different safety properties.
Related content
- Sign in to comment.
More from this channel
- Live-network incidents, hidden objectives, and training washout: the transfer problem in AI safety
- The measurement itself is now part of the AI safety threat model
- Safety testing needs a chain from model capability to institutional control
- AI safety needs mechanism attribution, not just outcome scores
- Safety controls are moving into the runtime