ECP turns an agent's execution trace into a portable evaluation contract (2026)

An agent can return the right answer through the wrong path. It can call an unauthorized tool, waste ten retries, skip a required verification step, or leave no audit evidence. A new proposal called the Evaluation Context Protocol (ECP) turns those execution details into a portable evaluation contract, so a PM can compare agents by what they did as well as what they said.

Quick take

Field	PM read
Trend	Agent evaluation is moving from final-answer checks toward portable, trajectory-level contracts. ECP exposes the user-visible output, tool calls, and evaluator-safe audit context through JSON-RPC, then lets the same evaluation run across agent frameworks and CI systems. 1 2
Fresh signal	Aniket Wattamwar, Manav Anandani, and Mrunal Kakirwar submitted ECP to arXiv on August 18, 2026. The project repository describes an active open-source implementation; a paper announcement on X appeared on August 21 and had 86 views when retrieved. 1 2 3
Problem solved	Final-answer tests hide tool choice, authority violations, recovery behavior, latency, and cost. ECP gives an evaluator a common place to inspect those events without tying the test to LangChain, LlamaIndex, CrewAI, PydanticAI, or one observability vendor. 1 2
Evidence boundary	ECP is an early-stage proposal with a reference implementation. The paper reports no ECP-specific benchmark, cross-framework conformance study, or production reliability result; empirical validation is future work. 1
Action window	Run a two-week pilot on one workflow with a stable acceptance decision. Compare the current agent with the same agent behind a trajectory-level contract, using held-out tasks and repeated runs before the contract becomes a CI gate.

What changed

An ordinary application test often asks whether the final response matches an expected answer. An agent test needs a larger record: which tool the agent chose, what arguments it sent, how many steps it took, what authority it used, and whether the final state is acceptable.

ECP packages that record behind a small, framework-neutral JSON-RPC interface. The runtime exposes three agent methods—agent/initialize, agent/step, and agent/reset—over standard input/output or streamable HTTP. Each step can return three useful objects:

public_output: the answer a user can see.
tool_calls: the actions the agent took, including tool names and arguments.
evaluation_context: evaluator-safe evidence about the execution. The project explicitly positions this field as a way to expose evaluation evidence without depending on raw chain-of-thought. 2

The evaluator can then run deterministic checks against a manifest. The repository documents time limits for each RPC and for the complete run, fresh-agent resets between scenarios, structured audit data, per-step latency, token usage, exit reason, and the difference between planned and executed steps. A failed or timed-out step is recorded as a failed check, which gives a CI system a clear signal to block a regression. 2 4

The product idea is larger than a new SDK. ECP makes the evaluation boundary portable. A team can keep its agent orchestration framework while changing the model, evaluator, or runtime, then ask the same contract questions each time: Did the agent satisfy the user-visible requirement? Did it use the required tool correctly? Did it stay inside the permitted action surface? Did it finish within the cost and latency budget?

Why PMs should care

The final answer is only one product outcome

An agent may produce a plausible answer after a broken sequence of actions. A support agent may quote the right refund policy after looking up the wrong account. A coding agent may pass a visible test after changing a file outside the requested scope. A research agent may give a correct conclusion after using a source it was not authorized to access.

ECP does not solve those product decisions. ECP gives the team a stable place to encode them. The PM still has to define which tool calls are required, which side effects need approval, what evidence counts as sufficient, and what failure should block release.

DAIR.AI made the same measurement shift in an August 12 post about adjacent Microsoft Research work: the action policy carries information about cost, latency, failure mode, and auditability, while a final answer throws that trajectory away. DAIR.AI described a study spanning eight models, six parallel benchmarks, 41 languages, and 2.38 million rollouts. That post is practitioner context for the broader evaluation direction, not evidence that ECP itself works. 5

A common contract makes model and framework swaps cheaper to test

Without a shared contract, every agent stack tends to grow its own trace format, grader hooks, and CI adapter. A team that changes orchestration frameworks then has to rebuild the evaluation layer before it can tell whether the new stack improved the product.

ECP's repository lists integrations for plain Python, LangChain, LlamaIndex, CrewAI, and PydanticAI, alongside a Python SDK, runtime CLI, JSON Schema contracts, examples, and an inspector UI. Those integrations are project capabilities, not independent proof of interoperability; a PM should verify the exact adapters used in the team's own stack. 2

The useful comparison unit becomes model + prompt/context policy + tools + runtime + evaluator. Keep that unit explicit in experiment reports. A model score without the tool surface and evaluator version leaves too much of the product unspecified.

Production teams are already building the surrounding pattern

Salesforce Engineering described an adjacent workflow for AI-generated code on August 20. The workflow starts with a written specification, grounds planning in repository evidence, converts success criteria into executable tests, maps each requirement to implementation and test evidence, and uses independent adversarial review. A judge can make a passing result stricter, while a later stage cannot turn a failed gate into a pass. 6

That workflow supplies a product precedent for ECP's contract layer: define the acceptance decision before the agent runs, retain the evidence that supports the decision, and keep human judgment for choices that repository evidence cannot settle.

LangSmith also announced a production-oriented evaluator on August 18. Its Perceived Error evaluator looks for explicit signals such as user corrections, repeated requests, and rejected actions, plus inferred signals such as contradictions, persistent misunderstandings, and unresolved outcomes. LangChain says the tuned model cut evaluation cost by 82% in its benchmark and by up to 98% in some early-partner workloads. Those figures are LangChain's own product claims; they show the commercial pressure to evaluate more traces, rather than validate ECP. 7

The specific ECP announcement had a small public response. The ECP announcement from Software Engineering Papers had two likes, one repost, and 86 views when retrieved. A recent explainer from Tutorials by Mukesh Kala, published August 16, had 273 views and 1 hour 57 minutes of runtime; it covers production contracts, trajectory evaluation, recovery, idempotency, observability, and rollback. Treat both as orientation signals. Adoption evidence needs usage, repeatable evaluations, and product outcomes. 3 8

Click to open on YouTube

Loading content card…

How to implement now

Use ECP, or a similar internal contract, as a bounded experiment. The goal is to learn whether trajectory evidence changes a product decision before the team standardizes on a protocol.

Choose one workflow with a real acceptance decision. Pick a support resolution, repository change, data-analysis packet, or other task with a named handoff owner. Define the accepted artifact and the unsafe outcomes before collecting traces. Record accepted-task rate, repair work, p50 and p95 latency, cost, and human corrections.
Freeze the deployment unit. Version the model, prompt, context policy, tool schemas, permissions, runtime, evaluator, task seeds, and budget. Keep the current agent as the baseline. A protocol comparison becomes hard to interpret when the model and evaluation boundary change together.
Expose three evidence layers. Capture the user-visible output, every tool call with arguments and result status, and evaluator-safe context such as the goal state checked, the reason for a stop, or the evidence used to choose an action. Keep secrets and raw private reasoning outside the evaluator payload. ECP's public_output, tool_calls, and evaluation_context fields provide one concrete schema to test. 2
Write two kinds of graders. Use programmatic checks for exact state: file diffs, database values, required citations, permission boundaries, and irreversible-action approvals. Use a rubric or human review for user experience, ambiguity handling, and whether the response addressed the actual request. Send disagreements to a human queue instead of averaging them away.
Run repeated, held-out comparisons. Compare the baseline and contract-instrumented agent on the same model, tools, tasks, and budget. Hold out task seeds, users, tool versions, and failure modes. Measure target acceptance, correct tool selection, unauthorized calls, recovery after tool errors, steps, token cost, latency, over-refusal, and reward-hacking or proxy-gaming incidents. Require stable performance across repeated runs; a single successful trajectory is a weak release signal.
Promote only the checks that change a decision. Put deterministic checks in CI first. Route ambiguous traces to review. Add a release gate only when the check predicts a real handoff outcome better than the current test. Keep a rollback path for evaluator changes, because a stricter or cheaper judge can change the apparent product quality without changing the agent.

The PM hypothesis is specific: a portable trajectory contract should increase the team's ability to catch unacceptable agent behavior before release, while adding less evaluation cost and latency than the product risk it exposes. ECP gives that hypothesis a concrete interface. The paper gives it a research direction. The pilot has to supply the product evidence.