Header Ad Placement

A Full Model Cache Should Cost Time, Not Break Reads: What a stargz Prototype Changes

MODEL DELIVERY AND CACHE FAILURE SEMANTICS

A lazy-pulling cache can turn disk exhaustion into model-file read errors even after a pod is Ready. A bounded-cache prototype separates delivery from persistence, but its measured cost and isolation limits matter as much as the successful reads.

By Ossam Hashim

KEY FINDINGS

  • In the study's 90%-prefilled-cache test, upstream stargz-snapshotter v0.18.2 failed 273 of 280 model files after reading 4.1 GB. The prototype read all 280 files, totaling 150.3 GB, with zero errors in two reported replicates. This is a controlled result on a specific failure suite, not a guarantee for every deployment.
  • The fix changes the contract: failing to save a fetched chunk must not prevent delivery of that chunk to its caller. An advisory index selects eviction candidates, while filesystem usage helps determine pressure when accounting becomes lossy.
  • Exhaustion still costs time. Five paired full reads measured a median 18.5% completion penalty at 90% versus 0% prefill, on a same-availability-zone registry. Remote-registry costs and the saturation point beyond three nodes were not established.
  • The paper did not demonstrate an advantage for its 2Q-style eviction policy over LRU at its roughly 5% detection floor. Its bound is per node, without per-tenant quotas or fairness, and a measured 3.4x cache inflation remains unexplained.

Ready Is Not a Full-File Integrity Check

Lazy image pulling defers work. A container can start after mounting an indexed image, while later reads fetch missing chunks from a registry. That is useful for large model images, but it makes the node's cache part of the serving path long after startup.

Georgii Kliukovkin's Why Kubernetes Cannot Contain a Full Lazy-Pulling Cache studies what happens when that cache fills and proposes a repair. The important distinction is between bytes being unavailable at the registry and bytes arriving successfully but failing to fit into a local optimization. Only the second failure should be removable by changing cache semantics.

The paper's title uses "impossibility," but operators should read the conditions, not infer a theorem that Kubernetes can never manage any cache. The experiments show a sampling probe missing damaged regions and a reclamation actuator unable to free bytes outside its accounting. Integrating those bytes into an effective platform accounting path is explicitly discussed as another possible design direction.

The result is a reason to inspect the delivery contract, not to conclude that every lazy-pulling implementation or managed service has the same failure. The paper tests stargz and reports a separate static audit of SOCI and Nydus; those are different evidence levels.

Why a Cache Write Can Veto a Read

The problematic path streams fetched data toward both a cache writer and the waiting caller. In the upstream code analyzed by the paper, a Go io.MultiWriter places the cache writer first. Go's MultiWriter contract is explicit: writers are called in order, and an error stops the operation instead of continuing to later writers.

That is reasonable for copying data to destinations that must all succeed. It is the wrong contract when one destination is an optional cache and another is the consumer that requested the data. A cache-side ENOSPC can prevent the caller's buffer from being filled, or make the system discard a buffer whose bytes have already arrived.

The prototype wraps the cache writer so a cache failure is remembered without aborting delivery. A failed or partial cache staging file is aborted rather than committed. The caller still receives the fetched bytes; a skipped-write counter records the lost caching opportunity. A failure of the registry fetch itself remains a read failure. This is not permission to suppress arbitrary I/O errors or return incomplete data as success.

The implementation also handles readers that join an in-flight fetch but cannot find its result in the cache: bounded retries eventually give way to a direct range fetch. The design needs that path because pass-through changes an old assumption that a completed fetch necessarily leaves a reusable local entry.

Pressure Signals Need a Matching Actuator

Kubernetes's current node-pressure eviction documentation describes reclamation based on filesystem layout: garbage-collecting dead containers or deleting unused images before evicting end-user pods when pressure remains. Those actions can help only if they reclaim the resource that actually filled the disk.

In the paper's pressure experiment, kubelet sees DiskPressure and image garbage collection runs, but frees zero bytes. Containerd reports only 741 KB of reclaimable images while 91.1 GiB of snapshotter cache occupies the partition. The chain then evicts 26 pods that had not written to that partition. Visibility exists; the selected reclamation objects do not own the offending cache bytes.

The lesson is narrower and more useful than "monitoring is useless." Check whether the signal, accounting unit and deletion action describe the same resource. An alert about an almost-full filesystem does not establish that deleting an unrelated pod will reclaim its contents. A component-owned byte budget is one way to close that gap without changing kubelet or the runtime interface.

A sentinel-file readiness probe has a different limit. The paper's probe stayed green while 272 of 280 files were unreadable. Sampling the same known regions can repeatedly confirm the regions that remain cached. Such a probe can be a smoke check, but it cannot certify every unread part of a model image.

Bound How Much, Then Decide Which Chunks

The prototype records chunk paths, sizes, observed access and in-flight pins in a persistent bbolt index. Updates arrive through a bounded queue that never blocks the data path. When the queue fills, updates are dropped and counted. That keeps accounting from becoming a serving stall, but makes the index advisory by construction.

Eviction excludes pinned chunks and selects only entries that have reached rename-commit. An unlinked file stays readable through an already-open descriptor under the assumed POSIX behavior; a later lookup of an evicted chunk follows the existing cache-miss path. Missing candidates can cost another fetch. They must not make the system serve incorrect bytes.

The hard part is the pressure estimate. An earlier implementation trusted the index total and failed during a write flood: about 127,000 dropped updates preceded a full partition and an unstartable snapshotter. The corrected rule adds a conservative allowance for dropped updates and raises pressure to filesystem-reported usage when accounting is known to be lossy. If all indexed candidates are exhausted while the filesystem remains over budget, a rate-limited rescan rebuilds the candidate set.

The split is deliberate: filesystem evidence answers how much pressure exists; the index answers which known chunks can be removed. The corrected version survived about 301,000 dropped updates in the reported flood, with 76 GB on disk against an 80 GB budget. That does not identify an unlimited update rate it can survive. The paper also reports temporary underestimates of 12.3%-12.8% on three samples during reconciliation, on a partition with 12 GB of headroom above the budget.

What the Full-Read Experiment Actually Establishes

The main rig uses two AWS i4i.2xlarge hosts in one availability zone, one registry and one node, with a dedicated roughly 92 GB NVMe cache partition. The budget is 80 GB, with high and low watermarks of 0.95 and 0.85. The workload is a 280-file, 140 GB-class model image; the measured full read totals 150.3 GB.

At 90% cache prefill, upstream v0.18.2 reads 4.1 GB and fails 273 files. The prototype completes all files with zero read errors in both elimination replicates. More than 4,000 skipped cache writes per run show that the test exercised the failure path rather than accidentally avoiding pressure.

The reported 18.5% penalty comes from a different comparison: five interleaved within-session pairs of the prototype at 0% and 90% prefill. Median times were 953.4 and 1129.8 seconds. It is not an 18.5% slowdown against an upstream run that successfully completed the same exhausted read; upstream did not complete it. Nor is it a universal tax on every cache hit.

During one full sweep, the prototype evicts 224.6 GB while reads continue. On-disk footprint peaks at 97.7% of the configured budget in 59 samples. The eviction volume exceeding the nominal image size is a reminder that a bounded cache smaller than the working set can repeatedly fetch and discard data. Bounded occupancy does not mean cheap registry traffic.

The Negative Results Change the Deployment Decision

The study expected a scan-resistant 2Q-style policy to beat LRU. Its gate-passing experiment did not separate the policies in latency or hit rate beyond the within-policy variation, with a roughly 5% minimum detectable effect. A more contended follow-up failed its own noise gate. This supports retaining LRU as the prototype's default, not a general conclusion that replacement policy never matters.

A second tenant also exposes an awkward capacity result. Before the competing sweep starts, a 21.5 GB hot set already occupies about 73 GB of cache, roughly 3.4 times its logical bytes, and repeatedly evicts. The paper does not isolate which cache tier or stale-copy mechanism accounts for all that inflation. Its two-tenant latency measurements therefore start from an already-thrashing baseline, not a quietly warm resident set.

Three simultaneous nodes keep their own budgets and complete reads without errors. Aggregate registry egress is nearly three times the one-node baseline. The registry link is not saturated; the nodes unexpectedly run slightly faster in the concurrent comparisons. A registry page-cache explanation is hypothesized, not instrumented. These measurements do not predict a fleet's saturation point or prove that scale-out under pressure is inexpensive.

What to Test Before Depending on This Pattern

Reproduce the cache-full condition on a disposable node with a separate test volume. Read complete representative files, including tails, through the mounted serving path. Record read errors, actual bytes delivered, skipped cache writes, filesystem usage, index usage, evictions and registry traffic. The question is whether a failed local persist changes the caller's result, not whether the health endpoint stays green.

Then test restart behavior and noisy accounting. The paper found a systemd kill-mode interaction defeating mount preservation, an index lock-handover race, and restart-driven cache duplication. Those are separate from the streaming-copy fix. A patch that serves bytes under ENOSPC is not automatically a complete deployment configuration with durable enforcement.

The released artifact has a dedicated paper3 README mapping all six campaigns to reports and evidence, plus a pinned fork commit and upstream v0.18.2 control. That is useful provenance, but this article does not independently rerun the experiments or establish current upstream merge status. Treat the evaluated fork as a research prototype until the exact build and supported failure behavior are checked for your environment.

Assumptions and limitations

This is an analysis of one primary study, crosschecked against its public artifact and the official Go and Kubernetes contracts. The numerical outcomes are author-reported, not our own performance or correctness tests. The evaluation uses one main model-file workload family, one hardware generation and a same-AZ registry. Remote registries, the reconciliation ceiling, managed lazy-streaming services and larger fleet saturation remain open.

The prototype offers a per-node bound, not per-tenant quotas, fairness or noisy-neighbor isolation. The cross-snapshotter claims come from source audit, not equivalent runtime exhaustion experiments on every implementation. The pass-through mechanism removes the analyzed cache-write-to-read-error coupling; it does not fix registry outages, bad model bytes, disk failures affecting other required data, or application-level correctness.

Worked Example: Keeping Accounting Separate from Delivery

This invented pressure calculation uses an 80-unit budget, a 95% high watermark and an index reporting only 60 units. With clean accounting, pressure remains 60 and eviction does not start. If accounting is lossy and filesystem usage is 78, pressure becomes 78 and crosses the 76-unit trigger. It illustrates one decision boundary, not the paper's full estimator, asynchronous reconciliation, pinning or a production storage algorithm. Cache-write failure still needs a separate pass-through path; a better number alone cannot repair writer coupling.

from fractions import Fraction as F
budget = 80
high = budget * F(95, 100)
def pressure(index, disk, lossy):
    if lossy:
        return max(index, disk)
    return index
for lossy in (False, True):
    used = pressure(60, 78, lossy)
    print(used, used > high)
assert high == 76

Sources & Provenance

  • Primary study: Why Kubernetes Cannot Contain a Full Lazy-Pulling Cache, Georgii Kliukovkin, arXiv:2611.11739, November 2026. Sections II-IV support mechanisms and boundaries; Section V and Tables IV-V support measured outcomes; Section VIII states limitations. The full title continues: Two Impossibility Results and a Bounded, Honest Design for Model Serving.
  • Paper 3 artifact README, kliukovkin/lazy-model-delivery-study. Maps the six campaigns, describes preserved invalid trials and omitted oversized raw files, and pins the evaluated fork commit. Artifact inspection is not an independent replication.
  • Go MultiWriter documentation. Supports sequential writer behavior and stopping on the first error; does not independently validate the prototype.
  • Kubernetes node-pressure eviction documentation. Supports the documented resource-reclamation order and filesystem distinctions, not the paper's experiment-specific cache-byte counts.

The test checklist and pressure calculation are explanatory analysis. No universal latency multiplier, current cloud bill, upstream-merge claim or production safety guarantee is asserted.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages