When Inference Becomes Part of Training: A First Look at Miles

I came across Miles while reading about SGLang. At first, I was confused about why an inference engine was sitting inside a reinforcement learning training framework. Inference and training are usually introduced as two separate jobs: training changes a model, while inference uses the finished model to generate an answer.

That separation becomes less clear during RL post-training. The model has to generate responses, receive rewards, learn from those responses, and then generate again with its updated weights. Inference is no longer something that happens after training. It becomes part of the training loop.

Miles is an open-source RL framework from RadixArk. It is built on slime and uses SGLang for rollout, Megatron-LM for training, and Ray to coordinate distributed workers. I have not run a full cluster job with Miles yet, so this post is an architecture read rather than a performance review.

What interested me was not simply the list of frameworks it connects. It was the systems work required to make those frameworks behave like one training loop.

The simple loop hides most of the work

On a whiteboard, RL post-training looks straightforward. The policy generates a group of responses, a reward function scores them, the trainer updates the policy, and the process repeats.

trajectories = rollout(policy, prompts)
scores = reward(trajectories)
policy = train(policy, trajectories, scores)
sync(policy, rollout_workers)

This description makes the policy sound like one object that every part of the program can access. In a distributed system, it is not. One copy of the policy is arranged inside SGLang for fast token generation. Another is sharded inside Megatron-LM for forward passes, backward passes, gradients, and optimizer state. The two sides may use different parallel layouts and may run on different machines.

Data also has to move in both directions. Rollout samples—including token IDs, log probabilities, rewards, and other metadata—travel from inference to training. A much larger set of updated model weights travels back from training to inference. If either path is too slow, the expensive GPUs on the other side wait. If either path changes the data it carries, the algorithm may train on something different from what the policy actually generated.

The usual RL diagram makes the boxes look like the important part. Miles reminded me that, at scale, the arrows between those boxes can be harder.

One policy is doing two different jobs

Rollout and training place very different demands on hardware. During rollout, SGLang serves the policy token by token across requests whose lengths may vary widely. During training, Megatron-LM processes batches, calculates gradients, and updates parameters.

Miles supports two broad ways to place these workloads. In a colocated setup, rollout and training share the same GPUs and take turns. This can make good use of a smaller cluster and keep weight transfers close to the devices, but the two runtimes must share memory and cannot do their main work at the same time.

In a disaggregated setup, rollout and training use separate groups of GPUs. This allows more overlap, but it creates a new boundary. Model weights and rollout data may now have to cross devices or even machines.

Neither arrangement is automatically better. Colocation spends time to save hardware and data movement. Disaggregation spends hardware and network bandwidth to gain concurrency. This is a familiar distributed-systems tradeoff: removing one bottleneck often makes another boundary more important.

Ray coordinates this process by starting and placing workers and helping data move between them. But orchestration alone does not solve the deeper problem. Both representations of the policy still have to agree on which version generated each trajectory and when a newer version becomes available.

Weight synchronization belongs inside the loop

After the trainer performs an optimizer step, the rollout workers hold an old policy. Before they generate with the new policy, the updated parameters have to reach them.

The simplest solution would be to save a checkpoint and ask every rollout worker to load it. This is easy to understand, but repeated serialization and storage access can become expensive when it happens inside the inner loop of large-model training.

Miles therefore treats weight synchronization as a first-class part of the framework. A colocated run can transfer weights through CUDA IPC. In a disaggregated setup, Miles can use NCCL broadcast, while its documented Megatron paths also include peer-to-peer RDMA and disk-delta updates for different hardware and network layouts.

The important lesson is not that one transfer method is always best. It is that synchronization is part of the algorithm’s behavior, not merely a deployment task after training.

Once weight updates happen while the system is running, several questions appear. What happens to a rollout already in progress when new weights arrive? Can the trainer still use work produced by the previous policy? How does it know which version generated a sample? A four-step algorithm can ignore these questions. A working distributed implementation cannot.

Asynchronous execution trades freshness for utilization

A synchronous loop has an obvious waste pattern. The trainer waits for rollout workers to finish a batch. Then the rollout workers wait while the trainer updates the model. Long agent trajectories make the problem worse because a few slow samples can delay the entire group.

Miles also offers a fully asynchronous mode for disaggregated GPUs. Rollout workers keep generation in progress, completed groups enter a bounded buffer, and the trainer consumes them while performing updates. Instead of repeatedly stopping for each other, generation and training overlap.

This improves utilization, but it introduces staleness. A trajectory may wait in the buffer while the trainer advances through newer policy versions. By the time the sample is used, it may no longer represent the current policy.

Miles records the weight version associated with each sample and provides a control for maximum staleness. The user still has to decide what is acceptable. A larger buffer can keep the trainer busy, but it can also allow older data to survive longer. Fresher on-policy data and higher hardware utilization pull in opposite directions.

This is why the Miles documentation recommends starting with synchronous execution while debugging a new recipe. Correctness is easier to reason about before concurrency hides the order of events. I think this is a useful principle beyond RL systems: make the state transitions understandable first, then overlap them.

Text is not the trajectory

The feature that made the inference-training boundary most concrete to me is token-in-token-out, or TITO.

It is tempting to pass plain text between an inference engine and a trainer. The inference side decodes token IDs into a string, and the training side tokenizes the string again. To a person, the result may look identical. To the model, it may not be the same sequence.

reconstructed_ids = encode(decode(token_ids))

# This round trip is not guaranteed to preserve the policy's actions.
assert reconstructed_ids == token_ids  # may fail

Tokenization is not always perfectly reversible. A tokenizer may map the decoded text to a different canonical sequence. Decoder settings may remove special tokens or normalize content. During multi-turn agent rollouts, a system may also render chat history again or parse and serialize tool calls into a new prompt, changing whitespace, reasoning fields, or JSON formatting.

This matters because the original token IDs are the actions the policy actually sampled. If the trainer reconstructs different IDs, it may optimize actions—or token contexts—that did not produce the recorded rollout probabilities. The sentence can look unchanged while the training example has changed underneath it.

TITO keeps the token prefix produced by inference as the authoritative version. On later turns, Miles reuses that prefix and tokenizes only the newly appended suffix, with model-specific handling at the boundary when necessary.

At first glance, this looks like a serialization detail. In reality, it protects the contract between inference and training. It is also a good example of a leaky abstraction. Text is a convenient representation for people, but it is too simple to fully describe the model’s sequence of actions.

Who is Miles for?

Miles includes several RL recipes, LLM and VLM support, low-precision training, agent environments, and model-specific launch scripts. However, it is probably not the shortest path for someone learning PPO or GRPO for the first time.

The documented quick start assumes eight H100, H200, or B-series GPUs, at least 500 GB of free disk space, and Docker with GPU access. Those requirements make its intended scale clear. This is infrastructure for distributed post-training, not a laptop-sized introduction to reinforcement learning.

Even without that hardware, I think the repository is useful reading for developers interested in AI infrastructure. It exposes questions that simpler examples hide: where samples wait, when a policy becomes stale, which process owns each model copy, how parameters move, and whether data keeps the same meaning as it crosses a system boundary.

What I learned

Before reading about Miles, I thought of inference mainly as the stage after training. RL post-training changes that picture. Generation produces the experience that training consumes, and the newly trained policy has to return to generation. The boundary is crossed repeatedly, so every small mismatch or delay becomes part of the inner loop.

Miles also reinforced three broader system-design lessons for me.

First, simple diagrams often hide expensive communication. A clean arrow may represent model-sized transfers, version tracking, buffering, and synchronization across machines.

Second, higher utilization is not free. Asynchronous execution can reduce idle time, but it makes ordering and freshness harder to reason about.

Third, a human-readable representation is not always a faithful system representation. Passing text instead of tokens looks simpler, but that abstraction can discard exactly the details the training algorithm needs.

My next step is to trace one sample through Miles v0.1, starting from the Qwen3 launch script: where the prompt enters, what SGLang returns, how the trainer consumes it, and when the updated weights become visible to rollout workers. If I later have access to the required hardware, I would like to follow that with a small end-to-end run.

For now, Miles gives me a more useful way to think about large-scale RL. The training algorithm may fit in a few lines of pseudocode, but its correctness and performance depend on everything that happens between those lines.

References and further reading

• Miles source code — the framework repository, examples, and release history.
• Miles documentation — architecture, installation, and user guides.
• Fully asynchronous training — Miles's execution model, rollout buffer, and staleness controls.
• Agentic rollout and TITO — how Miles preserves token-level trajectories across multi-turn sessions.
• SGLang — the rollout and inference runtime used by Miles.
• Megatron-LM — the distributed training backend.
• slime — the RL post-training framework on which Miles is built.