Header Ad Placement

Callo Turns Invalidated KV into Verified Drafts: 1.05-1.19x All-Request Speedups on a Two-H200 Testbed

DISAGGREGATED INFERENCE AND CACHE RECOVERY

A context edit can invalidate cached attention state without making it useless. Callo uses that state only to propose tokens while exact prefill runs elsewhere, then verifies before committing. Spare decode capacity and the transfer path determine how much this overlap pays.

By Ossam Hashim

KEY FINDINGS

  • On a two-H200 prefill/decode testbed, Callo reports 1.05-1.19x mean all-request latency ratios over unmodified vLLM at moderate load, across three model families and two context-management policies. Miss-request ratios are 1.06-1.24x.
  • Invalidated KV is a draft source, never trusted committed state. Verification waits for the entire exact prompt cache, recomputes draft KV and commits only the accepted prefix plus a correction or bonus token.
  • Part of the improvement comes from transfer and completion handling, not speculation. With the tightest tested decode batch limit, speculation contributes only a 3% latency reduction at capacity relative to Callo-Stream.
  • The 96-GPU results are simulations calibrated against the small testbed, not a physical 96-GPU deployment. Requests with no retained state, no session affinity or insufficient spare decode capacity take the ordinary path.

An Edited Prompt Breaks More Than a Token Match

Suppose a coding agent removes several old tool outputs before its next model call. Most later text is unchanged, but its tokens now have different preceding context and may occupy different positions. Their previously computed keys and values are no longer the exact attention state for the edited prompt.

Ordinary prefix caching can reuse the identical prefix before the edit. The changed suffix must be prefilled again. In a disaggregated system, a prefill instance performs that work while the decode instance serves other requests. The edited request itself still waits for its exact state to arrive.

In Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation, Jianjun Ou and Matthew Lentz separate two uses of a cache: state trusted for output, and state useful for predicting what the exact model might say. The previous request's suffix fails the first test but can pass the second.

This is not permission to serve answers directly from a stale cache. It is a proposal-and-verification design that spends otherwise available decode work during a wait the request already faces.

Reposition First, Then Draft on Provisional State

Callo aligns the previous request's tokens, including its reply, with the edited prompt. It finds ordered runs of identical tokens, copies their retained KV into new positions and re-rotates keys for the changed positions. The unedited prefix remains exact through ordinary prefix caching.

New spans need their own provisional state. The decode instance computes these spans in its ordinary batches, in chunks of at most 112 tokens of one span. Those chunks use scratch blocks separate from the blocks receiving exact prefill results. This is real work and can delay other requests; 'already cached' does not mean staging costs nothing.

Meanwhile, the prefill instance computes the exact suffix. It streams finished layers into the request's destination blocks. The decoder can draft while those updates arrive, so a draft may read a mixture of old and refreshed state. That mixture is allowed only because drafts remain uncommitted.

The paper's mixed-state measurements support this design empirically. They do not establish that arbitrary stale state or every possible context edit will provide useful drafts. The observed opportunity depends on retained information still predicting the exact model's next choices.

The Exact Boundary Is at Verification

Once a completion notification confirms that the entire prompt KV is exact and present at the decode instance, Callo runs verification over the last prompt token and the draft sequence. This pass reads exact prompt KV and recomputes the drafts' own KV.

Verification accepts tokens only up to the first rejection. It discards the remaining draft suffix, then commits a correction token at the mismatch or a bonus token if all drafts pass. Accepted tokens retain the KV recomputed during verification. Unverified speculative state is kept out of the reusable prefix cache.

The default uses fixed-tie greedy verification, with a defined tie-breaking rule. The system also supports accepting any co-maximum under an any-max greedy variant and probabilistic speculative sampling. These are distinct decoding rules, not interchangeable descriptions of one output sequence.

Importantly, 'lossless' here refers to preserving the chosen target decoding rule through exact-state verification. Batch composition can change floating-point numerical results across runs. The authors do not claim byte-identical output to every independent baseline execution under different batching.

Why Agreement Is Not a Blanket Guarantee

In an offline Llama-3.3-70B replay that feeds exact-decoding tokens into the approximate path, invalidated-KV next-token predictions agree at 98.3% of positions. That teacher-forced statistic is not the probability that an entire free-running answer will match.

With a 32-token draft window, 65% of the tested miss requests accept the entire window under fixed-tie greedy verification. Even drafts generated with exact KV reach only 80% in that particular comparison, because one-token decoding and multi-token verification can disagree numerically near ties.

That distinction is useful for system design. A high per-token agreement number can coexist with many partially rejected windows. The relevant quantities include accepted prefix length, verification cost and how much critical-path decoding the accepted tokens actually replace.

Give Exact Requests Priority, Not Merely a Larger Queue

Callo considers speculation only when a request arrives. It needs a free slot not required by a ready ordinary request, at least 256 reusable invalidated tokens, and a staging cost within a budget. The staging rule multiplies staged token count by the active batch size to account for interference across the batch.

During drafting, the policy estimates the probability that every draft so far survives verification. It multiplies model-specific agreement estimates derived from logit gaps and stops when the cumulative estimate drops below the default threshold of 0.3. This is a scheduling heuristic calibrated offline, not a correctness gate. Exact verification remains responsible for correctness.

If ready requests need slots, speculative requests yield, starting with those with the lowest cumulative acceptance estimates. They retain their existing drafts for later verification. The scheduler does not need to discard all earlier useful speculative work merely because it stops making more.

What Was Actually Measured on Hardware

The physical testbed has one prefill instance and one decode instance, each on an NVIDIA H200 NVL with 141 GB, connected by NVLink. It runs vLLM 0.29 with NIXL transfers. The tested models are FP8 Llama-3.3-70B-Instruct, BF16 Gemma-4-31B and BF16 Mistral-Small-3.2-24B.

Workloads come from recorded SWE-agent sessions on 100 SWE-bench Verified issues per model and context policy. Twenty issues are held out for early-stop calibration. Timing replays use the remaining eligible sessions, recorded prompts and tool times, and fixed recorded reply lengths. They measure serving performance, not a newly established improvement in software-issue solve rate.

The L5 policy keeps the last five tool outputs in full; L10 keeps ten. Older outputs become placeholders. On Llama, L10 lowers the edit-induced miss rate from 79% to 58% but raises median prefill per miss from 3.1K to 8.3K tokens. Fewer misses can still create a larger per-miss overlap opportunity.

At moderate load, defined as 52% of each configuration's estimated vLLM capacity, all-request ratios range from 1.05x to 1.19x, and miss-only ratios from 1.06x to 1.24x. These are ratios of mean latencies over matching requests, not percentage reductions or universal throughput multipliers.

Streaming Is Part of the Result

Callo-Stream uses the same transfer path but disables speculation. That control matters because streaming and prompt completion handling can shorten the wait before exact state arrives even without any accepted draft.

For the main Llama L5 workload with a decode batch limit of sixteen, speculation reduces mean all-request latency by 8% at light load and 10% at full load relative to Callo-Stream. With a batch limit of two, its reduction is only 3% at capacity; the transfer path explains most of the overall improvement there.

Most of the transfer-path improvement over baseline vLLM comes from handling completion without waiting for the prefill instance's running step to finish. It would be misleading to attribute the full end-to-end ratio solely to predicting tokens from invalidated KV.

Nor does 'full load' automatically mean the decoder has no spare capacity. A prefill bottleneck can build queues while decode slots remain available. In that case, a longer remote wait can leave more time to draft. Conversely, decode contention reduces the useful room for speculation.

Simulation Shows Where Speculation Fades

The authors simulate a 96-GPU cluster with several prefill-to-decode allocations, using measured engine-step timings and modeled acceptance from recorded speculations. This examines scale beyond the one-pair hardware setup; it is not a reported 96-GPU production deployment.

Against Callo-Stream, simulated miss-request speedups from speculation are 1.025-1.128x at moderate load across the tested allocations. At a 3:1 prefill-to-decode ratio, the ratio returns to about 0.99-1.00 beyond approximately 85% of baseline capacity. There, more speculative activity is not a meaningful latency win.

The paper also marks overloaded experiments where requests remain unissued or unfinished. Ratios over completed requests beyond capacity are not steady-state capacity gains. A deployment comparison must inspect unfinished work and queue growth rather than celebrating only the requests that completed.

Memory and Neighbor Costs Still Count

Callo reserves 0.75 GiB on the decode instance for runtime buffers, shrinking the main configuration's KV pool by 1.5%. Scratch blocks consume an average of 0.24% and a peak of 2.71% of that pool at moderate load. A default cap limits scratch usage to 10%.

Mean overall KV occupancy falls from 27.2% to 24.6% in that setting because some requests finish sooner, but peak occupancy is similar: 75.1% for vLLM and 75.4% for Callo. Lower mean occupancy is not proof of a smaller required peak-memory budget.

Ordinary requests also experience interference. Compared with Callo-Stream, non-speculating requests' mean time from the prefill response to reply completion is approximately 2% longer at low load and 6% longer near capacity on the main configuration. The end-to-end reported gains include that cost; it should still be monitored separately.

What to Test Before Adopting the Design

First establish whether sessions return to the same decode instance and whether their previous KV survives until the next call. Without affinity and retained state, the proposal source disappears. Then measure the distribution of edit locations, reusable suffix length, staging tokens, remote wait and output length.

Benchmark a streaming-only control alongside the full design. Track accepted draft prefixes and time saved after exact KV arrives, not only draft generation count. Separate impacts on speculating requests, ordinary requests and unfinished sessions as load rises.

Finally, audit the correctness boundary: exact-cache completion, verification inputs, rollback state and exclusion of provisional blocks from shared prefix reuse. The authors find no violations in 1,928 instrumented verification events covering 44,792 committed tokens. That is useful evidence within their tests, not a substitute for deployment-specific testing.

Callo's contribution is a practical distinction: cache state can be invalid for commitment while still valuable for prediction. Its payoff comes when that prediction overlaps a real wait, verifies cheaply enough and yields promptly to work that is already ready.

Assumptions and limitations

This is an analysis of the primary paper, without independent reproduction. Physical serving tests use two H200 GPUs and recorded coding-agent workload replays, while cluster results are simulated. Tested gains depend on load, retained cache, session affinity, staging cost and available decode capacity. Fixed reply lengths and recorded prompts do not establish a new agent-task accuracy result. Verification preserves its specified decoding rule, not necessarily identical outputs across numerically different batch schedules. Very long replies can dilute the relative saving, and adapter-switch cache reuse remains untested.

Worked Example: A Speedup Ratio Is Not the Same Percentage Saved

This invented timing example uses a 5.7-second baseline and a 5.0-second alternative. Their ratio is 1.14x, but the latency reduction is approximately 12.3%, not 14%. It illustrates how to read a reported latency ratio; it is not a Callo measurement.

from fractions import Fraction as F

baseline = F(57, 10)
alternative = F(5)
ratio = baseline / alternative
saved = 1 - alternative / baseline
assert ratio == F(57, 50)
print(round(100 * float(saved), 1))

Sources & Provenance

Written by Instinct for GlobalTechDigest. Analysis and the labeled arithmetic example are original. All performance and validation figures are author-reported.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages