Header Ad Placement

LoRA Under SFT and RL: Why Baseline Subtraction Concentrates the Spectrum

POST-TRAINING

Tracking the singular values of LoRA updates across post-training stages shows SFT and RL reshape the adapter in opposite directions - and a single RL ingredient, group-mean baseline subtraction, explains both contrasts.

By Ossam Hashim

KEY FINDINGS

  • Tracking the singular spectrum and subspaces of LoRA updates on GSM8K and MATH across Llama and Qwen models, a new paper finds RL (GRPO) concentrates the update's spectrum at every learning rate, while SFT expands it at moderate learning rates - and RL rotates the singular subspaces much faster than SFT at small learning rates.
  • Interpolating between SFT and RL one ingredient at a time pinpoints group-mean baseline subtraction as the source of both contrasts: subtracting the group's mean reward largely cancels the gradient component shared across a group's responses.
  • A quadratic multi-index model supplies the mechanism: coherent fitting (linear in learning rate, concentrates spectral energy on strongly responding directions) competes with gradient-noise feedback (quadratic in learning rate, redistributes energy evenly). Baseline subtraction preserves the first while weakening the second - and forces the risk-favorable learning rate to be low, which matches RL's practice of using markedly smaller learning rates than SFT.
  • Guided by the dynamics, the authors tailor LoRA updates to each stage: in SFT, flatten the spectrum of the full update (Muon-style norm-preserving flattening) to spread adaptation across singular directions; in RL, flatten only the component orthogonal to the current left subspace to promote rotation, leaving the within-subspace component untouched.
  • On GSM8K, the dynamics-informed configurations top the baselines: SFT on Llama-3.2-3B reaches 45.72 percent mean accuracy versus 42.99 for standard Adam, and RL reaches 47.66 versus 42.36, with the same pattern holding across evaluated models and tasks.

Two stages, one adapter, opposite dynamics

Post-training a language model usually means supervised fine-tuning followed by reinforcement learning, and LoRA is the default adapter for both - often under similar optimizer settings. What the two stages do to the low-rank increment itself has been mostly unexamined. A new paper (Liu, Wang, Dong and Lei) tracks two properties of the LoRA update through training: the spread of its singular values, measured by effective rank, which determines how task knowledge is distributed over the available directions, and the rotation of its singular subspaces, which determines which input and output directions get used at all. The contrast is sharp. RL with verifiable rewards (GRPO) concentrates the spectrum at every learning rate studied; SFT expands it at the moderate learning rates that produce good downstream performance. And at small learning rates, RL rotates the left singular subspace far faster than SFT. Same adapter, same optimizer family, opposite geometry.

One ingredient explains both

The cleanest part of the paper is the ablation. Interpolating between SFT and RL one ingredient at a time - on-policy data, verifiable reward, the loss form - the authors find that adding group-mean baseline subtraction alone is enough to reproduce RL's signature: decreasing effective rank across all learning rates and faster subspace rotation. Rejection-sampling fine-tuning with a baseline mirrors GRPO's dynamics; without the baseline, it behaves like SFT. Why would centering the reward change the geometry of the update? The paper's quadratic multi-index model, where a group of responses is sampled per prompt, separates two forces in the LoRA gradient. Coherent fitting is linear in the learning rate and concentrates spectral energy on directions that respond strongly to the task. Gradient-noise feedback is quadratic in the learning rate and spreads energy more evenly. Subtracting the group mean cancels the gradient component shared across the group's responses: coherent fitting along pretrained directions survives the contrast and successive updates stay anchored to similar directions, while the redistributing noise from task-irrelevant directions loses relative strength. The same model also shows baseline subtraction forces the risk-favorable learning rate downward - a mechanistic explanation for RL's habit of running LoRA at much smaller learning rates than SFT.

Tailoring the update to the stage

If SFT's strength is spectral expansion and RL's is rotation, the update rule should amplify each. For SFT, the authors flatten the singular spectrum of the full update - the norm-preserving spectral flattening familiar from Muon - spreading adaptation across singular directions, and pair it with an asymmetric optimizer choice (SGD direction for the B factor, Adam for A). For RL, they flatten only the component of the update orthogonal to the current left singular subspace, which promotes rotation while leaving the within-subspace component alone. The results on GSM8K (mean accuracy over three seeds, best learning rate per configuration): SFT with the tailored configuration reaches 22.82 percent on Llama-3.2-1B versus 22.24 for standard Adam, and 45.72 versus 42.99 on Llama-3.2-3B. For GRPO, the orthogonal-flattening configuration reaches 10.00 versus 7.61 on the 1B model and 47.66 versus 42.36 on the 3B. The pattern repeats across MATH and the other evaluated models, with the dynamics-informed configuration best in most settings. Notably, plain spectral flattening of the B update with Adam directions does not consistently help - the per-stage assembly is what matters.

Assumptions and limitations

All findings are the authors' own, measured on small models (Llama-3.2-1B/3B, Qwen2.5-3B) on GSM8K and MATH with GRPO as the representative RL method; other RL algorithms, larger models and non-math domains are untested here. Accuracy gains are a few points at these scales and each configuration is reported at its best swept learning rate. The mechanism rests on a quadratic multi-index model, a deliberate simplification. The worked example is an invented toy recursion illustrating the two-force picture, not the paper's analysis.

Worked Example: Two Forces on an Effective Rank

A toy recursion shows the mechanism in miniature. Track eight singular values and an effective rank (the exponential of their entropy). In the coherent-fit regime the top direction grows 50 percent per step while noise adds almost nothing: effective rank falls from 7.92 to 4.26 - concentration, the RL signature. In the noise-dominant regime, starting from a concentrated spectrum, even weak coherent growth plus uniform noise lifts effective rank from 6.94 toward 7.39 - expansion, the SFT signature. Baseline subtraction is what tips a real GRPO update into the first regime.

import math

def erank(s):
    t = sum(s)
    h = 0.0
    for x in s:
        p = x / t
        if p > 0:
            h -= p * math.log(p)
    return math.exp(h)

def run(s0, g, eps, steps=6):
    s = list(s0)
    out = []
    for t in range(steps):
        s[0] = s[0] * (1 + g)
        s = [x + eps for x in s]
        r = erank(s)
        out.append(round(r, 2))
    return out

u = [1.0] * 8
c = [4.0] + [1.0] * 7
print("coherent-fit dominant:")
print(" ", run(u, 0.5, 0.02))
print("gradient-noise dominant:")
print(" ", run(c, 0.05, 0.3))

Sources & Provenance

  • Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning (Liu, Wang, Dong and Lei, 5 Nov 2026) - arXiv abstract
  • Full paper (PDF), including the ingredient interpolation, multi-index analysis and full learning-rate sweeps - arXiv full text
  • Background referenced inside the paper: LoRA, GRPO, Muon, and effective rank.

All performance numbers are attributed to the preprint as published. Sources accessed 7 November 2026. This analysis is independent editorial content for GlobalTechDigest.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages