RLiD Architecture: Decision-Plane Reinforcement Learning, Single-Forward-Pass Rollouts, and Deleting the Zero-Mean Noise Token-Level RL Carries

TRAINING SYSTEMS

Token-level RL credits every generated token for an action the environment alone executes. RLiD trains on the decision itself: one forward pass reads a compiled state and action menu, and the gradient lands on executable options only.

By Ossam Hashim

KEY FINDINGS

• Token-level RL for language agents assigns an action's advantage to the entire generated response - reasoning chains, formatting, and malformed content included - yet the environment executes only the parsed action. Variation in how the same action is expressed adds zero-mean variance to policy-gradient estimates, and unparseable outputs waste interactions.
• RLiD (Reinforcement Learning in the Decision plane, arXiv Nov 2; UIUC, NUS, Zhejiang, UPenn, Rochester) compiles the observation, an optional compact memory, and a labeled action menu into the input; a large decision model (ldm) picks an action in a single forward pass. No autoregressive generation during rollouts, no parsing, and every selected action is executable by construction.
• The theory (Proposition 1): when reward depends on text only through the parsed action, the whole-string score decomposes into the action score plus a zero-mean realization score. Marginalizing over alternative phrasings preserves the expected update and removes that variance component. An exact credit audit on FrozenLake attributes 71% (answer-only) to 97% (30-word rationale) of token-level gradient variance to realization noise.
• The paper is honest about the trade: reasoning tokens are additional computation, and where a model's zero-shot rationale is strong (Qwen3.5-2B on ARC), token GRPO is 1.8 points ahead at about 4x the cost. On Qwen2.5-1.5B, RLiD gains +2.6 points at about 10x lower cost.
• Headline result: the full train-and-distill pipeline reaches 99.3% success on ALFWorld in 12.7 H100 GPU-hours, versus 86.6% in 17.3 GPU-hours for token-level RL of the same LLM. With compiled memory, a Qwen2.5-1.5B ldm and its distilled generative student reach 100% on all 134 unseen and 140 seen evaluation games.
• Successful ldm trajectories supervise generative LLMs through SFT on action commands - a post-training route that learns decisions with RL first and transfers them to text generation after. This blog's GRPO analysis covers the token-level algorithm RLiD bypasses at the action interface.

The Noise Nobody Executed

In token-plane RL the model writes a response - often a rationale plus an action command - and the action's advantage updates every generated token. But the environment only executes the parsed action. Two responses that specify the same move in different words produce different gradient estimates, and malformed outputs burn interactions on text that never becomes a decision. The gradient is paying for prose the world never reads. RLiD's reframing: if only the parsed action matters to the reward, make the action the unit of optimization.

One Forward Pass Per Decision

A compiler assembles three things into the model input: the current observation, an optional compact memory built from past feedback, and a labeled menu of the actions available right now. The large decision model reads this once - a single forward pass, no token-by-token generation - and returns a categorical distribution over menu labels. Standard RL objectives (the paper uses GiGPO-style estimators for temporal credit) then optimize the selected action's probability directly. Rollouts get cheaper per step because there is no autoregressive decoding, and the invalid-action failure mode disappears because the output space is the menu.

What the Theory Says, and What It Does Not

Proposition 1 gives the decomposition: if the reward depends on the generated text only through the parsed action, the string-level score splits into the induced action score plus a realization score with zero mean. Within a fixed text policy, marginalizing over alternative realizations of the same action keeps the expected update identical while deleting the realization variance. The FrozenLake credit audit quantifies what that buys: realization noise accounts for 71% of token-level gradient variance when the response is just the answer, and 97% when it carries a 30-word rationale. The authors are equally clear about the boundary: the analysis does not guarantee superiority over generative policies, because reasoning tokens buy extra computation, and their own ARC result on Qwen3.5-2B - where token GRPO wins by 1.8 points at about four times the cost - shows the trade is real.

What the Numbers Say

ALFWorld is the headline: 99.3% success for the complete pipeline in 12.7 H100 GPU-hours against 86.6% in 17.3 GPU-hours for token-level RL of the same model - better accuracy on about 73% of the compute. With compiled memory, the strongest 1.5B ldm and its distilled generative student solve all 134 unseen and 140 seen games. Elsewhere the picture is mixed by design: Countdown-after (frozen protocol) improves +12 to +14 points over token RL, mostly from the label regulariser, at 0.46x training time, though plain Countdown stays weak at 1.5B; same-runner Sokoban reaches 44.5% against at most 26%; ARC splits by model as above. The pattern: decision-plane training wins when the action space is enumerable and the model's free-text reasoning adds little.

Assumptions and limitations

The worked example is a Monte Carlo illustration of the variance decomposition, not the paper's estimator; real policies, baselines, and menus are far richer. All benchmark figures (ALFWorld GPU-hours and success rates, ARC, Countdown, Sokoban, the FrozenLake audit) are the authors' reported results; this analysis did not independently re-run training. Results are confined to finite-menu tasks - ARC, Countdown, Sokoban, ALFWorld - and the method does not apply where the action space cannot be enumerated. The benefit shrinks or reverses when a model's zero-shot reasoning is strong, as the paper's own Qwen3.5-2B ARC result shows, and menu/memory design choices affect what information the policy sees. Distillation quality depends on teacher trajectory coverage.

Worked Example: The Variance You Can Delete

This toy reproduces Proposition 1 in miniature. Two actions with true values 2.0 and 1.0 are sampled from a fixed policy (0.6/0.4); the REINFORCE-style gradient estimate for the first action's logit uses the sampled reward minus a baseline. In the token-level run, the reward carries an extra zero-mean Gaussian term - the stand-in for realization noise from how the action was phrased. In the decision-level run that term is marginalized away. Twenty thousand samples each: the means should match and the variances should not.

import random
random.seed(5)
p0 = 0.6
q = [2.0, 1.0]
b = 1.6
def grad(dec):
    a = 0
    if random.random() >= p0:
        a = 1
    n = 0.0
    if not dec:
        n = random.gauss(0, 1)
    r = q[a] + n - b
    if a == 0:
        return r * (1 - p0)
    return -r * p0
def var(dec):
    xs = []
    for _ in range(20000):
        xs.append(grad(dec))
    m = sum(xs) / len(xs)
    s = 0.0
    for x in xs:
        s += (x - m) ** 2
    return s / len(xs), m
vt, mt = var(False)
vd, md = var(True)
print("token mean:", round(mt, 4))
print("dec mean:", round(md, 4))
print("token var:", round(vt, 4))
print("dec var:", round(vd, 4))
sh = (vt - vd) / vt
print("noise share:", round(sh, 3))

Sources & Provenance

• RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs (Chen et al., 2 Nov 2026) - arXiv abstract
• Full paper (HTML), including the FrozenLake credit audit and per-benchmark breakdowns - arXiv full text
• Prior GlobalTechDigest coverage of token-level RL - the GRPO analysis

All performance numbers are attributed to the preprint as published. Sources accessed 4 November 2026. This analysis is independent editorial content for GlobalTechDigest.