How to Size Distributed Rollouts for Tool-Using LLM Agents 

An LLM reinforcement learning run can leave expensive GPUs waiting while a browser, database, or external API finishes one slow action.

Adding workers may raise the request count without shortening the full training cycle because the slowest stage still controls when learning can continue. 

Teams need a capacity plan built around trajectories, not a target GPU count. The method below maps the rollout path, measures long-tail behavior, sets synchronization rules, and ties infrastructure spend to usable learning signals. 

MinT LLM training infrastructure for LLM post-training and reinforcement learning, treats compute scheduling, distributed rollout, and training orchestration as shared infrastructure while leaving the training loop, data, loss functions, and RL environments under the user’s control.

That separation is useful when rollout capacity becomes a systems problem rather than simply a question of adding more GPUs. 

Start With Cycle Time, Not Cluster Size 

Define the unit of progress before estimating capacity. For an agentic RL loop, a useful unit is the time from selecting a task batch to completing policy updates and making the next policy available for rollout. 

Split that cycle into generation, environment actions, reward or judge work, advantage calculation, and optimization. Record elapsed time, queue time, and resource use for each stage.

A stage with modest compute can still dominate the cycle if it waits on rate-limited tools or serial approvals. 

This distinction appears in established RLHF systems. The DeepSpeed-Chat paper published profiling for one of its RLHF configurations; generation accounted for most of the iteration time, even though generation and optimization place very different demands on the hardware.

The broader lesson is not that generation will always dominate, but that training FLOPs alone are a poor proxy for end-to-end cycle time. 

Map One Trajectory From Admission to Update 

Draw the path of a single trajectory before distributing it. Mark every queue, model call, tool call, retry, reward calculation, storage write, and policy-version handoff. Add the owner and concurrency limit for each dependency. 

The map should answer four questions. Which resources are held while the agent waits, and which steps can run concurrently? 

Which failure ends a trajectory, and which artifacts must exist before the optimizer can use it? These answers reveal where work can pause without holding the most limited resource. 

Consider a support agent that reads a policy page, queries an account system, and drafts a response. The model server may finish each generation step quickly, but the account query has a strict rate limit and variable latency.

If a worker holds GPU memory during that wait, model capacity and tool capacity become coupled. 

The first design task is to release or reuse the scarce resource during external waits.

In practice, this often means decoupling model execution from environment execution: the trajectory can remain logically alive while the model-serving slot is released to another runnable trajectory.

The second is to keep enough runnable trajectories in the queue so the model server has work without overwhelming the account system. 

Measure the Distribution Behind the Average 

Average trajectory duration hides the runs that set the wall clock. Record the median, 90th percentile, and 99th percentile for total duration, generated tokens, tool calls, tool-wait time, retries, and reward time.

Break the results down by task family and termination reason. 

Recent agentic-RL systems research makes the same point more explicitly.  The Heddle paper identifies frequent tool calls and long-tailed trajectory generation as causes of queueing delay and interference, and schedules around whole trajectories rather than treating each generation step independently.

For capacity planning, the useful implication is that remaining trajectory length and tool-wait behavior can matter as much as the cost of the next model call. 

Do not solve every long run by raising the timeout. First separate valid hard tasks from loops, dead tools, and malformed actions. Give each category a different response: preserve valid hard cases, repair environment faults, and terminate non-learning loops with a recorded reason. 

Choose a Synchronization Boundary 

Synchronous training keeps rollout data close to the current policy, but one slow trajectory can hold an entire batch.

Fully asynchronous collection keeps resources busy, yet the optimizer may receive data from older policies whose behavior no longer matches the current one. 

A bounded approach often gives platform teams a better starting point. Set a maximum policy-version gap, a batch completeness threshold, and a rule for late trajectories.

Late trajectories can move to the next eligible batch, be retained for an algorithm that explicitly supports off-policy data, or be discarded once their policy gap exceeds the training system’s acceptance rule. 

Track the share of trajectories rejected for staleness. If that share grows, additional rollout capacity may be making the learning loop less efficient rather than more productive.

Possible responses include shortening the collection horizon, adjusting the synchronization boundary, or increasing weight-publication frequency when the synchronization overhead is lower than the cost of stale rollouts. 

Size Each Stage Around Its Own Bottleneck 

Estimate demand separately for model generation, environments, judges, storage, and optimization. Use observed service time and arrival rate to find the stage whose queue grows during a stable pilot. 

For generation, measure tokens per second alongside active sequences, KV-cache use, and tail latency.

The vLLM PagedAttention paper work is a serving result rather than an agent-RL result, but it illustrates an important capacity-planning principle: KV-cache management changes how many sequences can be batched efficiently, so request concurrency alone is not a meaningful measure of model-serving capacity. 

For environments, track actions per second, open sessions, rate-limit responses, and reset time. For judges, track samples per second, prompt length, agreement checks, and manual-review backlog.

For optimization, track accepted tokens per update, step time, checkpoint time, and weight-publication delay. 

Add capacity only where the growing queue sits. If environment wait dominates, more generation replicas can create a larger backlog and higher tool costs. If weight publication is slow, a faster rollout layer will produce more stale data. 

Use Backpressure as a Training Control 

Every queue needs a maximum size and an admission rule. When a dependency reaches its safe limit, stop admitting lower-value work before the system starts timing out indiscriminately. 

Prioritize tasks using an explicit score built from observable signals such as evaluation coverage, failure rarity, policy freshness, task family, and environment cost.

Keep the rule simple enough to audit so that the scheduler does not quietly starve difficult but important task families. 

Set separate budgets for retries and exploration. Retries caused by transient infrastructure errors should not consume the same allowance as deliberate alternative actions.

This split prevents an unreliable tool from appearing to be useful exploration. 

Budget Cost per Accepted Learning Signal 

Tokens per second and GPU utilization describe activity, not learning value. For operations, it is useful to define a simple derived metric: cost per accepted learning signal.

This is not an optimization objective by itself; it is a way to expose how much rollout and evaluation spend is being discarded before an update.

Add a metric that divides total rollout and evaluation cost by trajectories accepted into an update, then segment it by task family. 

Define explicitly what makes a trajectory eligible for the intended update. At minimum, the system should know the policy revision, available action trace, environment state, reward status, and termination reason.

Failed or truncated trajectories should not be discarded automatically; whether they remain useful depends on the learning algorithm and the failure mode. 

Pair the cost metric with coverage. A scheduler that over-optimizes for short trajectories may lower cost while starving long workflows that the agent still needs to learn.

Consider setting an explicit coverage floor for important long-tail task families, and report their cost and learning contribution separately. 

Run a Two-Stage Capacity Pilot 

Start with a small load test that holds the policy fixed. Increase admitted trajectories in steps until one queue grows continuously, then record the bottleneck, tail latency, failure mix, and cost per valid trajectory. 

Next, run the full learning loop at a lower load. For infrastructure such as MinT, this is also the point where rollout scheduling should be evaluated alongside training orchestration rather than as an isolated serving benchmark, because a faster collector only helps when the resulting trajectories reach an eligible policy update efficiently.

Measure cycle time, policy-version lag, accepted-data rate, and time spent publishing new weights.

Raise one capacity limit at a time and keep the change only when accepted signals per hour improve without breaching cost or reliability limits. 

Do not use GPU utilization as the final success criterion. A rollout system can keep every accelerator busy while generating stale, redundant, or invalid trajectories.

The capacity change is useful only when it improves the rate of usable learning data or shortens the policy-update cycle at an acceptable cost. 

The sizing rule is therefore not “add rollout workers until the GPUs stay busy.” Find the stage that limits usable trajectories or delays the next policy update, increase capacity there, and then measure the entire cycle again.

For tool-using agents, a balanced rollout system is one in which model serving, external environments, evaluation, and optimization advance at compatible rates without turning higher throughput into stale or lower-quality training data.