Build a Tool Runner That Retries Without Repeating Side Effects (2026)

Imagine a charge_card tool succeeds upstream, but the HTTP response times out. The assistant sees an exception, concludes that the charge failed, and calls the tool again.

A better prompt cannot resolve that ambiguity. The application needs to know whether the operation was accepted, whether another attempt is safe, and when the run must stop. Those are execution decisions, so they belong outside the model.

The current provider and protocol docs make the application-side role explicit. OpenAI's function-calling guide shows the model returning a call with a call_id, then the application executing it and sending back correlated output. That call ID is simply the identifier that pairs one request with its result. Anthropic documents client tools as tool_use requests that application code executes before returning tool_result. The MCP tools specification puts invocation on the client and recommends confirmation for sensitive operations, timeouts, audit logs, and result validation. 1 2 3

The model proposes work. A tool runner decides whether and how that work becomes a side effect.

Put the boundary in code

A thin loop often looks like this:

const call = await model.next(messages);
const args = JSON.parse(call.arguments);
const result = await tools[call.name](args);
messages.push(toolResult(call.id, result));

It works in a demo because every hidden decision takes the happy path. Production adds the questions the snippet avoids:

Decision	Model contribution	Runner responsibility
Which tool?	Proposes a tool name	Checks that the tool is registered and allowed in this run
With what input?	Produces arguments	Parses, validates, normalizes, and freezes them
On whose authority?	May explain intent	Enforces user identity, permissions, scope, and approval policy
In what order?	May request several calls	Serializes conflicting writes and limits concurrency
After an error?	May suggest another action	Classifies the outcome and applies the retry policy
Did it succeed?	Reads the returned result	Records the authoritative result before summarizing it
When should the run end?	May return a final answer	Enforces turn, time, cost, and tool-call budgets

OpenAI distinguishes strict Structured Outputs, which match the declared schema, from JSON mode, which only guarantees parseable JSON. 4 In this runner, schema-valid output is still input to permission, approval, and retry policy. A schema proves shape; it does not grant permission or make a write retry-safe.

Use a state machine, not a pile of catch blocks

One workable internal design uses six states. This is an implementation pattern, not a provider-defined lifecycle. The states matter because recovery depends on where execution stopped.

State	What the runner does	What must be durable before moving on
`proposed`	Accepts a provider call and correlates it with the current run	Provider `call_id`, tool name, raw arguments
`validated`	Parses the schema, normalizes values, checks permissions, assigns risk	Frozen arguments, argument hash, policy decision
`awaiting_approval`	Shows the exact operation to an approver when policy requires it	Approval request and expiry
`executing`	Calls the tool under a timeout, attempt budget, and idempotency policy	Logical operation ID, idempotency key (a stable token reused for the same retryable operation), attempt number
`recorded`	Stores the raw success or typed failure	Authoritative result, timestamps, error class
`reported`	Sends a compact, correlated result back to the model	Provider result payload and delivery status

rejected, failed_permanent, and budget_exhausted are terminal states. outcome_unknown is not. It means the runner needs reconciliation before it can safely choose a terminal state or another attempt.

Persist the transition to executing before the network call. Persist the result before returning it to the model. If the process dies between those writes, recovery starts from a known operation ID instead of inventing a new call.

Temporal documents the fuller version of this pattern: outside-world work runs as Activities, results enter an ordered event history, and workflow replay reuses recorded results instead of repeating the side effect. 5 The transferable rule is simpler: log the intent and result, then recover from that log. You do not need Temporal to apply it.

Give every invocation a typed contract

Keep provider message formats at the adapter edge. The runner should work with its own types:

type ToolRisk = "read" | "reversible_write" | "irreversible_write";

type ToolErrorCode =
  | "invalid_arguments"
  | "forbidden"
  | "approval_rejected"
  | "rate_limited"
  | "upstream_unavailable"
  | "timeout_unknown"
  | "permanent_failure";

type ToolSpec<I, O> = {
  name: string;
  inputSchema: Schema<I>;
  outputSchema: Schema<O>;
  risk: ToolRisk;
  timeoutMs: number;
  maxAttempts: number;
  supportsIdempotency: boolean;
  execute(ctx: ExecutionContext, input: I): Promise<O>;
};

type Invocation<I> = {
  runId: string;
  callId: string;
  logicalOperationId: string;
  toolName: string;
  frozenInput: I;
  inputHash: string;
  idempotencyKey?: string;
  attempt: number;
};

type ToolOutcome<O> =
  | { kind: "succeeded"; value: O }
  | { kind: "rejected"; code: ToolErrorCode }
  | { kind: "failed_retryable"; code: ToolErrorCode; retryAfterMs?: number }
  | { kind: "failed_permanent"; code: ToolErrorCode }
  | { kind: "outcome_unknown"; code: "timeout_unknown" };

Only the runner should produce failed_retryable, and only after the registered tool policy confirms that another attempt is safe. An irreversible write without idempotency support should become outcome_unknown or a terminal failure, never an automatic retry.

Three details matter here.

First, the provider's call ID and your logical operation ID solve different problems. The call ID correlates a result with one model request. The logical operation ID follows the intended side effect across retries, process restarts, and provider turns.

Second, frozenInput must not change between attempts. If the model revises an address, amount, or recipient, that is a new operation that needs a new policy decision.

Third, outcome_unknown prevents a timeout from masquerading as a clean failure. The server may have committed the write before the connection broke.

Retry the operation, not the model's guess

Retries belong in one layer: the runner. If the HTTP client retries, the tool wrapper retries, and the agent loop asks again, a single failure can multiply into several writes.

AWS's retry guidance calls retries "selfish" because they add load while a downstream service is already struggling. It recommends timeouts, capped exponential backoff, jitter, retry limits, and retrying at one point in the stack. It also states the hard rule for writes: side-effecting APIs are unsafe to retry unless they provide idempotency. 6

Use an error matrix instead of catch (err) { retry() }:

Outcome	Example	Runner action
Invalid request	Schema mismatch, impossible enum	Do not call the tool; return a correction-safe error
Forbidden	User lacks scope for the resource	Stop; do not let the model negotiate around policy
Approval rejected	Human declines a destructive action	Stop that operation and report the rejection
Transient failure	`429`, selected `5xx`, connection refused	Retry only when the tool policy allows it; use backoff and stop at the declared budget
Duplicate or replay	The logical operation already has a stored result	Return the stored outcome; do not call the tool again
Unknown outcome	Timeout after sending a write	Reconcile by idempotency key or status lookup; do not create a fresh operation
Retry budget exhausted	Attempts, time, or spend reaches its limit	Enter `budget_exhausted`; block further attempts for that operation
Cancellation requested	User stops an in-flight call	Send transport cancellation when supported; close the model turn and record any late result
Permanent failure	Closed account, deleted resource, invalid state transition	Stop and return a typed reason

For an idempotent API, generate one high-entropy key per logical operation and store it with the frozen arguments. An idempotency key is a client-generated token that lets the server recognize a retry as the same operation. Reuse that same key on every attempt. Stripe's API documents the behavior clearly: later requests with the same key receive the first request's stored status and body, and a parameter mismatch is rejected. 7 That contract is payment-specific, but it shows what a useful idempotent tool interface should promise.

If a tool has no idempotency support, choose one of three policies at registration time:

No automatic retry. Surface outcome_unknown for a person or higher-level workflow to resolve.
Read-after-write reconciliation. Query by a stable business key, such as an order ID, before another write.
Wrap the tool. Put a small service in front of it that owns operation IDs, deduplication, and stored outcomes.

Do not ask the model whether a retry is safe. It lacks the transaction state needed to answer.

Make approval and authorization separate checks

Approval is not authorization. A user can approve an action they are not allowed to perform, and an authorized action may still deserve confirmation because it is expensive or irreversible.

Assign each registered tool a risk class. Read-only calls can often run after ordinary permission checks. Reversible writes may need a policy check plus an audit trail. Irreversible writes should usually pause with the exact target, arguments, and effect visible to the approver. OpenAI's Agents SDK models this as an interruption that preserves resumable run state; the application approves or rejects before execution. 8

Run authorization again at execution time. Permissions may change while an approval waits. Bind the check to the end-user identity and target resource, not to the model's prose explanation.

Parallel calls need the same care. Both OpenAI and Anthropic document controls that restrict parallel tool use. 1 9 Disable parallel execution for writes that touch the same resource, or have the runner acquire a resource-scoped lock. Two individually valid calls can still conflict when they run together.

Return outcomes the model can use

A raw stack trace is a poor tool result. So is a cheerful string that hides uncertainty.

Return a compact machine-readable envelope tied to the provider call ID:

{
  "status": "outcome_unknown",
  "code": "timeout_unknown",
  "logical_operation_id": "op_01J...",
  "message": "The request was sent, but completion is not confirmed.",
  "allowed_next_actions": ["check_status", "request_human_review"]
}

Keep the raw upstream payload in the execution record. Send the model only the fields it needs to choose a valid next step. MCP explicitly distinguishes tool execution errors that a model may correct from protocol errors that it probably cannot, and recommends validating tool results before passing them to the model. 3

Cancellation also needs a recorded outcome. MCP's cancellation pattern warns that late responses can race with cancellation and says clients should enforce request timeouts. If a user stops a run, cancel the in-flight request where the transport supports it, record any late result, and do not silently feed that result into a closed model turn. 10

Evaluate traces, not final prose

A final answer can look correct after the runner did something dangerous. Tool evaluation needs the execution trace.

Record at least:

trace_id, run_id, model turn, and provider call_id
tool name, logical operation ID, input hash, and risk class
state transitions and policy decisions
attempt count, timeout, latency, and backoff
approval actor and decision
outcome class, upstream status, and whether reconciliation ran
a redacted result summary

OpenTelemetry models traces as linked spans with attributes, events, status, and parent-child relationships. Its developing GenAI conventions define an execute_tool span with a tool name, call ID, result, and error type; arguments and results are opt-in because they may be sensitive. 11 12

Turn failure modes into deterministic tests:

Test	Expected trace
Arguments fail schema validation	No network span; terminal validation error
User lacks permission	No execution attempt; policy denial recorded
Approval is rejected	Run resumes with rejection; tool never starts
Tool returns `503` twice, then succeeds	Three attempts under one logical operation ID and one retry budget
Write commits, response times out	`outcome_unknown`, then reconciliation; no new logical operation
Process crashes after the write	Recovery finds the recorded intent and reuses the idempotency key
Model keeps requesting tools	Runner stops at the turn or tool-call budget
Cancelled call returns late	Late result is recorded but not attached to the closed turn

Anthropic recommends explicit stopping conditions such as a maximum number of iterations, plus sandboxed testing for agents because errors can compound across autonomous steps. Treat max turns, max tool calls, wall-clock time, and spend as separate budgets; whichever expires first stops the run. 13

Build checklist

Put every provider tool call through one runner; tool adapters never execute directly from model output.
Validate and freeze arguments, then persist the intent before any side effect.
Register risk, permission, approval, timeout, retry, concurrency, and idempotency policy with each tool.
Use one logical operation ID and one stable idempotency key across attempts.
Keep outcome_unknown distinct from success and failure; add a reconciliation path.
Record the authoritative result before sending a summarized result back to the model.
Enforce tool-call, turn, time, and spend budgets outside the prompt.
Test the trace for duplicate writes, late results, crashes, denials, and exhausted retries.

Start with one write tool and make these cases pass before adding more tools. A ten-tool assistant with a reliable execution boundary is easier to extend than a fifty-tool assistant whose only recovery plan is another model turn.