Scaling Agentic RL: Excessive-Throughput Agentic Coaching with Tunix

The main focus of LLM alignment has quickly shifted from static chatbot alignment to dynamic agentic workflows. At the moment’s fashions do not simply discuss—they execute multi-step reasoning, name exterior APIs, and work together with complicated environments.

Coaching reasoning brokers encounters particular challenges and bottlenecks. The latest evolution of agentic RL coaching shifts the method from single-turn alignment to multi-turn decision-making with complicated atmosphere interactions and power utilization. This shift raises new challenges on the infrastructure facet for rollout efficiency and effectivity; when an agent pauses to execute code, question a database, or wait on an internet search, the costly AI accelerator utilization plummets as TPUs sit idle ready for atmosphere steps.

Tunix—Google’s post-training library—natively solves this bottleneck in its newest launch, introducing an environment friendly, composable framework for coaching LLM brokers at scale. Tunix retains accelerators absolutely utilized on two fronts:

Asynchronous Rollouts: A high-concurrency rollout engine fully decouples TPU execution from host-side atmosphere latency (like community I/O or software execution).
Barrier-Free Pipelining: A dynamic producer-consumer structure continuously batches and streams variable-length trajectories to the coach, stopping pipeline stalls.

Past orchestration, agentic RL requires specialised observability. Whereas commonplace profilers like XProf supply deep operator-level traces, their excessive overhead limits them to brief, sporadic captures. Tunix introduces steady, light-weight instrumentation constructed immediately round domain-specific RL metrics. By correlating these high-level loop metrics with TPU timelines, builders get a worldwide view of execution effectivity to rapidly spot and resolve system bottlenecks.

In the end, Tunix is constructed to maximise TPU throughput, maintain environments modular, and make multi-turn coaching effectivity absolutely clear. Right here is the way it works below the hood.

1. Asynchronous & Decoupled Rollouts: Close to-Zero Idle Time, Most Throughput

Attaining peak {hardware} throughput means maintaining TPUs continuously busy. Tunix accomplishes this by combining asynchronous rollouts to get rid of execution bubbles and stragglers with a decoupled pipeline that constantly streams knowledge to the coach.

Asynchronous Rollouts

In Agentic RL, trajectory era (rollout) is probably the most time-consuming part. Nonetheless, the standard synchronous rollout structure creates two main issues, as depicted within the determine beneath.

Execution bubbles: when the rollout synchronously waits for an atmosphere to initialize, or return a state and reward, it’s going to create execution bubbles within the accelerator and degrade effectivity.
Straggler impact: Batched era can also be weak to long-tail issues, the place general latency is dictated by the slowest trajectory within the group.

Tunix solves this with an Asynchronous Trajectory Collector Engine.

Excessive-Concurrency Execution: Leveraging Python’s asyncio inside our RolloutOrchestrator, the framework manages large swimming pools of concurrent agent-environment interactions. Whereas one agent pauses for a host-side software execution, the inference engine instantly pivots to generate tokens for different energetic trajectories.
Async vLLM & SGLang Integration: Tunix natively integrates with performant inference engines like vLLM-TPU and SGLang-Jax. By enabling async request dealing with, the engine ensures non-blocking sampling and most concurrency on the TPU.

This structure fully overlaps mannequin inference, software execution, and reward computation, preserving excessive {hardware} utilization.

Decoupled Rollout & Coaching Pipelining

Whereas async rollouts remedy trajectory era bottlenecks, one other crucial problem for {hardware} effectivity within the end-to-end RL workflow is bridging dynamic, variable-length, and probably long-tail rollouts with a strictly synchronous coaching loop. A naive strategy depends on a synchronization level that forces the accelerator to attend till a complete batch of trajectories is full earlier than initiating the coaching step, ravenous the coach TPU.

Tunix eliminates this bottleneck by decoupling rollout and coaching right into a steady producer-consumer pipeline (illustrated within the diagram beneath):

The Producer: The async rollout orchestrator constantly yields accomplished trajectories right into a high-throughput queue.
The Client: The AgenticRLLearner consumes from this queue. For algorithms like GRPO—which require a number of reasoning paths per immediate to compute group benefits—Tunix dynamically teams these asynchronous trajectories on the fly.

The second a trajectory group is full, it’s post-processed, scored, and streamed immediately into the coach. This pipeline ensures the synchronous coach is continually fed, maximizing end-to-end throughput.

2. Composable Agent and Atmosphere Abstractions – Plug-and-Play OSS Environments

A significant friction level in RL frameworks is the inflexible coupling of the algorithm to the atmosphere loop. Modifying a codebase to assist a brand new open-source software program (OSS) benchmark like SWE-bench, WebArena, or a customized sport engine typically requires an enormous rewrite.

Tunix resolves this with a decoupled, composable structure. By exposing a clear API boundary, Tunix automates step invocation and lifecycle administration so you possibly can focus fully on core interplay logic.

The Agent Layer: Manages immediate formatting, motion era, and dialog histories. It robotically applies the coverage mannequin’s chat parser and preserves particular tokens at multi-turn boundaries—crucial for making certain strict Token-In, Token-Out (TITO) habits. You’ll be able to simply customise era logic by subclassing ConversationAgentBase.
The Atmosphere Layer: Out of the field, Tunix gives prebuilt TaskEnvironment and ToolEnvironment lessons. You may also inherit from BaseTaskEnv to interface with any exterior system. Tunix handles multi-turn episode lifecycles, commentary routing, and reward processing robotically.

Why it issues: You’ll be able to onboard any open-source RL atmosphere in minutes. As a result of the agent and atmosphere logic are fully decoupled from the coaching workflow, swapping a single-turn math verifier for an interactive bash terminal requires zero modifications to your coaching code. To exhibit the facility of this composable design, we subsequent showcase a couple of examples of how simply new brokers, fashions, or environments can be utilized. You’ll find extra detailed examples of custom-made Agent/Env in our recipes.

Instance 1: Prebuilt vs. Customized Brokers

Tunix affords built-in lessons like ModelAgent and ToolAgent that work instantly by way of configuration.

from tunix.rl.agentic.agentic_grpo_learner import GRPOLearner
from tunix.rl.agentic.brokers.model_agent import ModelAgent, ToolAgent


# Non software calling single flip agent
learner = GRPOLearner(
    agent_class=ModelAgent, 
    agent_kwargs={"system_prompt": "my system immediate"},
    ...
)


# Custom-made software name agent
tool_map = {"calculator": CustomizedCalculatorClass, ...}
learner = GRPOLearner(
    agent_class=ToolAgent, 
    agent_kwargs={
        "system_prompt": "my system immediate",
        "tool_parser_name": "gemma",
        "tool_map": tool_map,
    },
    ...
)

Python

Alternatively, you possibly can construct your personal customized Agent and add particular logic on how you can course of the mannequin responses. Tunix will robotically wire this agent ultimately to finish coaching workflow. E.g. SWEAgent, FrozenLakeAgent

from tunix.rl.agentic.brokers.base_agent import ConversationAgentBase
from tunix.rl.agentic.brokers import agent_types


# Deliver your personal agent! 
# Discover how the agent would not must know something concerning the mannequin (whether it is Qwen, Llama, or Gemma)
class MyAgent(ConversationAgentBase):
    def __init__(self, args):
        ...


    def update_from_model(self, response: str, **kwargs) -> agent_types.Motion:
        # Customized logic to course of the uncooked response (e.g., extracting  tags)
        ...


# Tunix robotically wires up the e2e workflow
learner = GRPOLearner(agent_class=MyAgent, agent_kwargs={...}, ...)

Python

Instance 2: Bringing in Customized Environments

Just like Brokers, Tunix affords plenty of pre-built environments together with TaskEnvironment, ToolEnvironment. Alternatively, it’s also possible to deliver your personal customized atmosphere by merely implementing a couple of primary APIs, together with any open supply atmosphere such because the Gymnasium instance beneath.

import gymnasium as gymnasium
from tunix.rl.agentic.agentic_grpo_learner import GRPOLearner
from tunix.rl.agentic.environments.base_environment import BaseTaskEnv, EnvStepResult


# You solely must deal with the core logic of atmosphere interactions, and Tunix will robotically deal with the remainder of the lifecycle administration and performance invocation.
class MyEnv(BaseTaskEnv):
    def _initial_observation(self):
        # deal with env creation and preliminary commentary
        self.env = gymnasium.make("your_chosen_env")
        commentary, data = self.env.reset(seed=42)
        return commentary


    def _step_impl(self, motion):
        # compute commentary, reward, performed, data
        motion = self.env.action_space.pattern() 
        obs, reward, performed, data = self.env.step(motion)
        return EnvStepResult(obs, reward, performed, data)


    def shut(self):
        self.env.shut() # clear up env after trajectory is completed


learner = GRPOLearner(env_class=MyEnv, ...)

Python

3. Eliminating the Black Field: RL-Particular Light-weight Profiling

When working asynchronous agentic coaching at scale, conventional logging falls brief. You want granular but domain-specific visibility to determine effectivity issues: Is the bottleneck within the era part? Is the software name taking an excessive amount of time? Or is the data-loader too sluggish?

Customary profilers like XProf present detailed, op-level traces to grasp micro-level efficiency like kernel and mannequin execution. Nonetheless, capturing long-spanning traces with these instruments is usually cost-prohibitive, and figuring out macro-level bottlenecks throughout the noise of low-level knowledge stays troublesome. For the complicated workflows of agentic RL, builders want a light-weight, macro-level view constructed on domain-specific metrics that map on to RL levels.

Tunix delivers this huge image by fastidiously monitoring a minimal set of crucial RL-specific metrics that characterize each the worldwide pipeline (how rollout, coaching, and weight sync phases work together) and necessary sub-steps (every mannequin name, atmosphere interplay, and so on.). As a result of they’re light-weight, these metrics run constantly all through all the coaching job. Customers can rapidly determine the place the workflow is stalling globally, after which deploy a software like XProf for focused, additional debugging.

The determine above illustrates a Perfetto hint captured from a multi-turn agentic coaching job, detailing the staged execution timelines throughout CPU threads and TPU gadgets. As demonstrated by the hint, TPU gadget utilization is way greater than that of the CPU threads, whose idle time is primarily as a result of atmosphere execution latency.

This macro-level tracing of staged RL pipelines lets you:

Pinpoint TPU Hunger: Visualize the precise time a Python software name or atmosphere execution blocks the asynchronous pipeline. Conversely, it allows you to affirm that parallel rollouts efficiently overlap to maintain accelerators saturated.
Confirm Pipeline Alignment: Observe the exact timing of macro-stages to make sure they align with out introducing hidden latency bubbles. You’ll be able to simply confirm that the coach is not ready on rollout era, or that weight synchronization is not inflicting extreme execution delays.
Optimize Coaching Configuration: Tune efficiency dynamically utilizing metric knowledge. For instance, you possibly can modify max rollout concurrency by correlating thread swimming pools immediately towards TPU idle time, or optimize coaching micro-batch sizes based mostly on knowledge era charges and HBM constraints.

Tunix turns the “black field” of distributed multi-turn RL right into a clear, optimizable timeline for the coaching job.

How Tunix Compares to the Ecosystem

In case you are evaluating frameworks for Agentic RL, right here is how Tunix stands out:

vs. OpenRLHF / veRL: OpenRLHF and veRL have made important strides utilizing Ray + vLLM. Nonetheless, they’re constructed primarily for the PyTorch ecosystem. Tunix brings this functionality natively to the JAX/TPU ecosystem. Sitting seamlessly on high of JAX, Flax, and Optax, Tunix delivers native Pathways multi-host distributed coaching, leveraging XLA’s compiler optimizations.
vs. Hugging Face TRL: TRL is well-suited for normal single-turn SFT (Supervised Wonderful-Tuning) and RLHF (Reinforcement Studying from Human Suggestions). Nonetheless, orchestrating complicated, multi-turn async loops typically requires important customized glue code. Tunix makes multi-turn, tool-use environments a first-class citizen out-of-the-box.
vs. Ray RLlib: RLlib is a complete, general-purpose RL powerhouse. But, mapping trendy LLMs natively to share weights on accelerators with out heavy overhead is complicated. Tunix flips the script: it’s an LLM-first library that brings high-performance RL on to the native LLM serving infrastructure.

Begin Constructing Your Brokers At the moment

Whether or not you’re reproducing SOTA reasoning fashions, fine-tuning the Gemma or Qwen households to “assume,” or deploying complicated multi-agent methods, Tunix gives the high-performance basis wanted for the following era of reasoning brokers. Begin constructing in the present day!

🌟 Star the Repo & Discover the Code: github.com/google/tunix
📖 Recipes: SWE coding agent, math, gaming agent.
📖 Learn the Docs: Deep-dive into our Agentic RL structure at tunix.readthedocs.io
🚀 Attempt the Fast Begin: Bounce into our /examples folder to discover quite a lot of recipes and run your first coaching job in the present day!

Tunix is below energetic open-source improvement by Google and the broader neighborhood. In case you are constructing the following era of reasoning brokers, drop by our GitHub Points and tell us what environments you’re plugging in.