Header Ad Placement

FIRE Contains Replay Failures, but Its Monitors Do Not Prevent the First Bad Request

REPLAY TESTING AND PRODUCTION RELIABILITY

Meta's production-connected testing study separates isolation, detection and worker exposure. Its 200,000 contained exits are not 200,000 distinct bugs, and its cancellation times do not start at fault onset.

By Ossam Hashim

KEY FINDINGS

  • FIRE reports more than 200,000 abnormal worker exits in isolated ranking tiers over 90 days, without directly affecting production-serving workers through those isolated paths. Repeated crash loops contribute to that count; it is not a count of unique defects or avoided outages.
  • Production-connected replay can protect the service under test while still reaching shared feature and prediction services. Isolation removes a shared edge, monitors react to visible violations, and traffic pinning restricts reachable workers. None substitutes for all the others.
  • In a separate 16-day sample, median termination intervals were 62 seconds for dependency-health detectors and 121 seconds for experiment-attributed detectors. The clock begins at the first observed violation, not when the fault began. Input validation is still needed before a malformed request reaches an accelerator worker.
  • The study reports no derived resource burden during an 81-day follow-up on covered paths, but faults continued. That observational result is not a randomized causal estimate, a company-wide safety guarantee or evidence that pinning prevents shared-state side effects.

The Boundary Is Egress, Not Just Ingress

FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale, by Shripad Nadgowda and colleagues, describes an uncomfortable testing arrangement. A candidate service receives recorded requests, not live user requests, yet selected outgoing calls still reach dependencies that serve users. That can expose realistic interactions before a change lands, but it also makes the experiment part of a larger production request graph.

This differs from an online user A/B test and from a fully mocked staging environment. The service under test, or SUT, can crash without taking away a product endpoint. Its downstream calls can nevertheless consume shared capacity, emit malformed inputs or change resource-lifecycle accounting. Isolating the candidate process does not isolate every consequence of executing it.

The paper's recommendation-serving path moves through query and aggregation, ranking, feature retrieval, an inference gateway and prediction workers. One ranking request can fan out into many downstream calls. A small replay stream is therefore not automatically a small blast radius: one crash-triggering request per worker can reach a large worker set without representing much total traffic.

Two Failure Stories Explain the Design

In the first example, replay repeatedly exercised a latent timeout defect. A ranking request was canceled while asynchronous feature extraction still held references to request-scoped objects. Once that state was torn down, the surviving task could access freed state and crash a worker. Ordinary requests had rarely exercised the necessary timing when the change was submitted.

Replaying familiar request contents did not preserve their original timing. Repetition and controlled utilization changed queuing and cancellation interleavings. The experiment exposed a real defect, but its calls reached a shared dependency. Worker-restart signals revealed the failing service before they identified the upstream experiment that should be stopped.

The second example began with a candidate ranking change that collapsed a tensor's batch dimension. A model expecting batch-by-feature data received a one-row shape. The malformed request crossed into a prediction service, triggered an accelerator assertion and left affected workers unable to serve until restarted. Stopping replay reduced further exposure; it did not undo the first worker failure or repair the model-contract violation.

These are different causes with the same operational lesson. A test can be useful precisely because it exercises a dangerous downstream behavior. The platform needs to preserve that diagnostic signal while limiting where the effect can travel.

Isolation Removes an Edge, If the Replica Is Faithful

FIRE routes replay toward isolated ranking and inference-gateway tiers where representative copies are practical. Maintaining such a tier requires more than launching another binary. State, configuration, scaling and health must remain close enough that control-versus-treatment differences still mean something.

That is harder for large, continuously changing feature stores and accelerator-backed prediction fleets with many model variants. A stale copy can turn a candidate comparison into a state-divergence measurement. Reserving enough accelerator capacity for every model is also expensive. The design therefore chooses isolation by dependency class rather than claiming to duplicate the entire production graph.

The reported ranking isolation uses less than 1% of the corresponding shared ranking tiers' provisioned worker capacity while handling billions of replay requests per day. Volume is estimated from sampled routing records; capacity is compared using worker counts. Those denominators matter. This is not a measurement that the whole testing platform costs less than 1% of all production compute, nor a universal scaling law for another service.

Monitoring Detects Trouble Before It Proves Blame

A dependency-health detector observes downstream counters, such as unexpected prediction-worker restarts, against a starting baseline and absolute limits. It can stop an experiment because the environment is unhealthy without proving that experiment caused the disturbance. Several experiments can terminate on the same shared event.

An experiment-attributed detector is narrower. It matches downstream malformed-request records against the control and treatment jobs for the experiment. That requires usable caller identity to survive the request path. If errors are logged for a batch containing multiple jobs, the evidence can implicate several contributors rather than one original request.

Each detector has a warm-up, polling interval, consecutive-violation threshold and monitoring duration. A clear check resets the violation counter. This rejects a single transient excursion, but also postpones cancellation and permits some intermittent faults to escape the consecutive threshold. Retiring a detector after its window is another coverage limit, not continuous protection forever.

The reported 62- and 121-second medians measure time from the first observed violation to termination. They exclude the preceding wait for visibility, warm-up and the first poll. The experiment-attributed sample has only 14 terminations, versus 534 for dependency health, with a maximum attributed interval of 364 seconds. These are measured detector outcomes in a particular window, not universal recovery deadlines.

Pinning Restricts Reach While Concentrating Load

Where replicating prediction is impractical, FIRE directs experiment traffic to a designated shard rather than distributing it across every shard for the model. For a task-local crash, this narrows the worker set exposed before monitoring reacts. It does not make a malformed request acceptable.

Concentration creates another risk. A pinned shard can become a straggler under replay load. Pinning also does not confine effects that propagate through shared state, delayed cleanup or common control paths. A routing bound is therefore not a general bound on customer impact or resource damage.

The study groups repeated crashes on pinned tasks rather than counting each as an independent success. Over the examined telemetry, bursts account for 39% of crash-like exits but 11% of failure episodes after grouping consecutive crashes on a task within ten minutes. The tasks were selected from a one-day routing snapshot and examined over the preceding 60 days. This characterizes recurrence on those targets; it does not measure how many other workers would have failed without pinning.

Read the Outcome Numbers With Their Own Windows

The isolated-tier result counts more than 200,000 abnormal exits over 90 days, excluding planned updates and infrastructure preemptions and deduplicating reports by task instance. Fatal signals account for 96%, OOM kills 3% and other exits 1%. Repeated occurrences of the same crash loop remain worker failures, not new defects.

The monitor-cancellation result comes from a separate three-month window: approximately 5% of experiments were canceled by dependency monitors, accounting for about 87% of all cancellations for the measured service. Of monitor-initiated cancellations, roughly 81% came from dependency health and 19% from malformed-request attribution. The 87% figure is a share of cancellations, not a percentage of all production bugs caught.

The aggregate rollout analysis uses a derived resource-burden measure for targeted malformed-input and latent-defect observations. Pre-rollout and follow-up each contain 46% of measured replay experiments; the partial-rollout interval contains 8%. During the 81-day follow-up, faults continued but no burden was recorded by that measure. Absolute burden values and per-class shares are not published, and records without reported values are excluded.

This is evidence of observed containment on covered paths. The incremental rollout is not a randomized trial with a complete counterfactual. It cannot establish zero incidents, zero residual cost, or a causal reduction in the underlying fault rate. Keeping those distinctions intact is more useful than presenting a large exit count as a safety score.

Fidelity and Test Selection Have Their Own Blind Spots

Both sides receive the same sampled requests, with randomized start order, and run on matched environments. That reduces request-mix noise. It does not make them independent when both sides share a dependency: treatment can perturb the environment that control observes.

Recording samples range from about one in 100,000 to one in 14,800 eligible requests. A typical one-to-four-hour lookback reduces stale-model lifecycle effects, but preserves neither complete serving concurrency nor causal user sessions. Rare patterns can be absent. More replay is not automatically better if it refreshes references that should expire or contaminates downstream telemetry.

Build-graph reachability reduces unnecessary source-change experiments. The reported 25% reduction counts diff-time workload executions, with remaining artifacts and trial settings unchanged. It is not a measured 25% reduction in all testing cost or downstream incidents. Declared dependency graphs can miss dynamic loading and undeclared inputs, so conservative fallbacks still matter.

Configuration selection uses observed runtime consumption instead. An offline setting-level refinement would suppress 72.3% of settings selected at file granularity, but 1.7% of suppressed settings were later read in a held-out week. That refinement was not enabled in the reported measurements. Its denominator is changed settings, not experiments, compute or regressions missed.

What Existing Tools Do, and What They Do Not Establish

ServiceLab's OSDI 2024 paper supplies useful context: paired replay may call production dependencies, and service owners must prevent undesirable side effects. It describes dropped writes, redirected writes and default restrictions on generic RPCs in specific setups. Its FrontFaaS discussion treats test load as negligible relative to production. FIRE's crash-class examples show why traffic share alone does not settle worker exposure in a different measured setting; this is not a controlled comparison between the papers.

Response comparison solves a different problem. The archived Diffy project multicasts requests to a candidate and two known-good instances, using disagreement between known-good responses to identify nondeterministic noise. That can reveal behavior changes, but equal responses do not prove outgoing calls were harmless. GoReplay documents traffic capture and shadowing; getting requests into a test instance still leaves the instance's egress policy to design.

Argo Rollouts documents metric-driven progressive delivery, including aborting a rollout when analysis fails. Its failureLimit is an overall failed-measurement threshold, not FIRE's consecutive-violation counter that resets after a clear check. Copying the same number into those mechanisms would not give equivalent behavior. These tools provide context, not evidence that they implement FIRE's complete containment design.

A Practical Review Starts at Each Shared Dependency

For each outgoing edge, record whether requests are isolated, pinned or fully shared; who owns the dependency; which mutations or lifecycle effects can occur; and how a replay request is recognized in downstream records. Check that caller identity survives batching and intermediate gateways, and test whether the kill path stops new calls and drains or cancels outstanding work.

OpenTelemetry baggage is one possible context carrier, not an authorization mechanism. Its documentation warns that automatic propagation can expose values to unintended third parties and that baggage lacks built-in integrity checks. Use a bounded experiment identifier with an explicit propagation boundary, rather than secrets or a trusted-looking caller assertion. Adding baggage also does not automatically attach it to telemetry attributes; instrumentation must read and record it deliberately.

Then separate safety checks from comparison checks. Validate tensor shapes and other dangerous contracts before admitting shared work. Test restart limits, replay-load ceilings, missing telemetry and stale baselines separately from latency or quality regressions. Those are evaluation suggestions from this analysis, not a verified ready-to-deploy FIRE configuration or a claim that its monitor prevents the first crash.

Assumptions and limitations

This is an analysis of the November 11 FIRE study, with published systems work and official project documentation used for context. Results are author-reported and primarily concern one recommendation-service family. No production experiment, independent incident audit, tool deployment or benchmark replication was performed for this article. No downloadable FIRE implementation is asserted.

Isolation, attribution and pinning cover configured request paths and failure classes. They assume suitable scheduling, service introspection and downstream identity. Shared state, concurrent experiments, unobserved requests and dynamic dependencies limit transfer to another system. In particular, the study's observed containment outcome must not be read as permission to connect an unreviewed candidate to live services.

Worked Example: A Clear Poll Resets the Cancellation Counter

This invented monitor has 30-second polls, no warm-up and a two-consecutive-violation threshold. Violations at 30 and 60 seconds cancel at 60. A different trace with a violation at 0, a clear result at 30, then violations at 60 and 90 cancels at 90, because the clear result resets the count. These are scheduled observation times, not fault-onset or recovery times. Real telemetry delay, warm-up, monitor retirement and cancellation work are omitted. This models FIRE's reset rule, not an Argo failureLimit configuration.

from itertools import count
def stop_at(checks):
    streak = 0
    for t, bad in zip(count(0, 30),
                      checks):
        if bad:
            streak += 1
        else:
            streak = 0
        if streak == 2:
            return t
print(stop_at([0, 1, 1]))
print(stop_at([1, 0, 1, 1]))
assert stop_at([0, 1, 1]) == 60
assert stop_at([1, 0, 1, 1]) == 90

Sources & Provenance

  • Primary study: FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale, Shripad Nadgowda and colleagues, arXiv:2611.14962, November 2026. Full main Sections 1-10, figures, tables and references checked; counts retain their separate observation windows and denominators.
  • ServiceLab paper, Mike Chow and colleagues, OSDI 2024. Read for production-connected replay, side-effect responsibilities and specific workload policies, not as an independent validation of FIRE.
  • OpenTelemetry baggage documentation. Context propagation, explicit telemetry recording and security limits; baggage is not authenticated experiment identity.
  • Argo Rollouts analysis documentation. Metric-based progressive delivery and overall failure-limit semantics; not evidence of FIRE's consecutive-failure policy.
  • Archived Diffy repository. Candidate and dual-known-good response comparison and noise handling; cited as a historical comparison pattern, not a maintained deployment recommendation.
  • GoReplay repository. Traffic recording and shadowing context; capture capability alone does not establish downstream isolation.

The monitor arithmetic and dependency-review checklist are explanatory analysis. Worker isolation, detector cancellation and routing confinement answer different safety questions.

Comments

Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage

KUBERNETES FAILURE RECOVERY A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage. By Hossam The important question is what failure should trigger A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action. Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed. This is a documentation-based de...

Editorial Governance & Pages