G-0004 · INC: G-0003 p95_latency_ms breach

## Summary Goal G-0003 (#4): "Reduce failed payments caused by gateway timeouts by 20%". Lengo Core killed the flag `goal_0003_timeout_retry` after the `p95_latency_ms` guardrail broke. The flag is already off (rollout_pct 0) and users are safe. This incident does not turn it back on, and nothing here is merged. ## What happened (kill record) - Reason, quoted: `guardrail p95_latency_ms: more than 5% of treatment requests exceed 1756 ms ((1+0.2) x control p95); lower bound 0.054` - Source: `ingest_fast_path` (the fast path killed it, not the periodic tick). - Time to kill: 620.333 ms, plus up to 5 s for clients to refresh flag state (per the kill note on #4). - Policy: hands_off, automatic kill, actions disable_flag, open_incident, reopen_goal (max 2 kills per goal). - `http_5xx_rate` was **not** part of the kill reason. Warning signs before the kill, from the analyst's decision 17 (verdict expand, 1% to 10%): - control p95 1,463 ms, treatment p95 3,242 ms. - 8.7% of treatment requests were above the 1,756 ms threshold. The lower bound of that share was 4.3%, just under the 5% trigger, so the guardrail still read "ok". - The analyst flagged p95 latency as the guardrail to watch. The kill came when the lower bound crossed 5% (0.054). - failure_rate was only -3.5% relative (interval -5.73 to +5.02 points, includes zero), far from the 20% target. - 5xx was 0.27% in control and 0.40% in treatment (+0.13 points, interval includes zero), well under the 2% hard limit. - The sample ratio check showed no mismatch (chi-square p = 0.94). ## Impact - Who: users in the flag-on arm of `goal_0003_timeout_retry`. At the 1% step the analysis had 501 flag-on users against 48,504 flag-off users. The flag was moved toward 10% in decision 17 just before the kill. I could not see the exact number of users exposed at 10%. - How long: the release notes on #4 were posted at about 16:45 UTC (flag at 1%). The kill note was posted at about 17:01 UTC, on 2026-10-24. This is roughly 16 minutes of exposure at most. I worked this out from the issue note timestamps, so treat it as approximate. - What users saw (hypothesis, see below): slower payment attempts, not more errors. The 5xx rate did not clearly move. - Users with the flag off were never affected. ## Runbook steps being followed - **RB-001#1**: confirm the timeout pattern by comparing the `gateway_timeout` share of failed payments for the last hour against the previous day. A rise above 5 points means a timeout incident. I have no payment-level data in the context note, so this comparison is still **to do** for the on-call owner. It is not done yet. - **RB-001#0** (symptoms): latency p95 sits close to the 1,500 ms client timeout because timed-out attempts are counted at the timeout value. This explains the control p95 of 1,463 ms and is relevant to the cause below. - **RB-002#0** (5xx): checked and ruled out as the trigger. 5xx was about 0.4% in treatment, against a 2% hard limit and a baseline of about 0.3%. No RB-002 steps apply to this breach. ## Suspected cause (from reading the change) The change is MR !3. `apps/pesalite/app/release.py` sets `FLAG_KEY = "goal_0003_timeout_retry"` and `TREATMENT_RETRY_POLICY = "backoff"`. In `apps/pesalite/app/retry.py` the `backoff` policy has `max_attempts=2`, `base_backoff_ms=400`, `jitter=True` and an idempotency key. The loop in `charge_with_retry` adds the elapsed time of each attempt plus the back-off delay to the reported latency. Reasoning: 1. A timed-out attempt is counted at `timeout_ms` = 1,500 ms (`gateway_sim.py`). 2. The retry then waits 200 to 400 ms (400 ms base, jittered to 50% to 100%). 3. The second attempt can time out again at 1,500 ms. The simulator treats 66% of timeouts as persistent by default (`transient_timeout_share` 0.34). 4. A payment that times out twice therefore reports about 1,500 + 200 to 400 + 1,500 = about 3,200 to 3,400 ms. This matches the observed treatment p95 of 3,242 ms. 5. The guardrail threshold is 1,756 ms