RL training is becoming the product surface for agent teams

Quick take

What changed

1. Rollout: SGLang engines generate trajectories. Agent sessions can call tools and interact with their own isolated environments.
2. Training: a Megatron-LM or PyTorch FSDP backend consumes completed trajectory groups and applies an RL loss such as GRPO, GSPO, or PPO.
3. Weight update: the new policy returns to the rollout fleet while other work continues. Miles documents peer-to-peer RDMA transfer for connected fleets and disk-delta updates when direct accelerator connectivity is unavailable. 1

• Asynchronous scheduling: rollout workers keep producing while the trainer consumes completed groups. This avoids making every fast trajectory wait for the slowest tool call or long reasoning trace.
• Token fidelity: Miles' token-in-token-out session server keeps the exact token IDs generated during rollout. That prevents a tool-call parser or chat-template round trip from silently changing the training context.
• Environment isolation: each episode can start in a fresh sandbox. The environment owns state, executes actions, and passes the verifier's result into the reward signal. That makes the training loop auditable, but it also makes the environment and test suite part of the learning system. 1

Why PMs should care

1. The bottleneck moves upstream of the user experience

2. The verifier becomes a product requirement

• a task boundary that can be reproduced;
• a verifier whose false positives and false negatives can be measured;
• a safe environment where failed actions cannot reach real users or production data; and
• enough representative tasks to expose shortcut behavior.

3. Vendor claims and engineering evidence answer different questions

How to implement now

1. Choose one verifiable workflow. Start with terminal or coding tasks, structured tool calls, or another workflow with a deterministic acceptance test. Define the task distribution, the held-out set, and the failure modes before generating trajectories.
2. Build the environment contract. Give every episode a fresh sandbox, explicit network and data permissions, a time limit, and a verifier that records both the result and why it failed. Keep the verifier outside the model’s editable workspace.
3. Establish a non-RL baseline. Compare the current production model and a supervised or prompt-only baseline on the same held-out tasks. Record task success, invalid tool calls, time to completion, interaction count, and safety violations.
4. Instrument the loop before optimizing it. Track rollout latency by percentile, tool-wait time, trajectory length, GPU utilization, policy version, sample staleness, reward distribution, and weight-update pause. A faster training loop that learns from stale or corrupted trajectories is not an improvement.
5. Start with the smallest reversible update. Use an adapter or a small model when the task allows it. Keep the last accepted checkpoint, replay a fixed regression set after every promotion, and send only the new policy to an isolated canary environment.
6. Set a promotion gate. Promote only when held-out task success improves without an unacceptable rise in reward hacking, unsafe actions, latency, or cost. Stop when the verifier disagrees with human review often enough to make the reward untrustworthy, or when the system spends more compute recovering from long-tail trajectories than it gains in task success.