Memory

# RB-001: Payment gateway timeouts > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms Failed payments rise and most failures carry the reason `gateway_timeout`. Users see "payment could not be completed, please retry". Support tickets mention payments "hanging", "timing out" or "stuck on processing". Latency p95 sits close to the client timeout (1,500 ms), because timed-out attempts are counted at the timeout value.

Step 1. Confirm the pattern: compare the `gateway_timeout` share of failed payments for the last hour with the previous day. A rise above 5 points is a timeout incident, not ordinary declines.

Step 2. Split by cohort. If a feature flag is rolling out, compare the flag-on and flag-off arms. If only the flag-on arm is worse, the change is the suspect: follow RB-003 to switch it off.

Step 3. Check the retry policy of anything that calls the gateway. A retry must wait with exponential back-off and jitter (at least 200 ms before the first retry) and must carry an idempotency key. Immediate retries without a key collide with the attempt the gateway is still processing and fail with `duplicate_in_flight` (a 5xx).

Step 4. Look for `duplicate_in_flight` and `gateway_error` in the 5xx breakdown. A rise in either alongside a retry change is the retry-storm signature recorded in outcome M-0002.

Step 5. Limit retries to one attempt for payment calls. More attempts multiply load on a gateway that is already slow, and they rarely recover the payment.

Step 6. If the gateway itself is degraded (timeouts rise equally in both arms), do not ship a retry change during the incident. Hold the rollout and open an incident with the gateway status attached.

RB-002 (high 5xx), RB-003 (kill a flag), ADR-001 (flags over redeploys), outcome M-0002.

# RB-002: High rate of 5xx responses > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms The `http_5xx_rate` guardrail turns amber or red on the dashboard. PesaLite returns 502 for upstream faults (`gateway_error`, `duplicate_in_flight`). The usual baseline is about 0.3%. Anything above 2% in the flag-on arm breaches the hard limit in `lengo.yml`.

Step 1. Check whether the rise is in one arm or both. Lengo only kills when the flag-on arm is confidently worse than control, so a rise in both arms points to the platform or the gateway, not the change.

Step 2. Read the kill record on the goal issue: it states the guardrail, the observed rates in each arm, the confidence bound that crossed the limit and the time to kill.

Step 3. Break 5xx down by reason. `duplicate_in_flight` means a retry collided with an attempt still in progress (see RB-001 step 3). `gateway_error` without a matching change means the gateway is degraded.

Step 4. Confirm the flag is off (RB-003 step 2) and watch the flag-on arm's 5xx rate fall back to baseline within one refresh interval (5 seconds) plus one ingest batch.

Step 5. Open an incident with the guardrail data, the reason breakdown and the commit that introduced the change. Draft a fix as a merge request, but never merge it automatically.

Step 6. Before re-planning, write the lesson down: the Scribe stores it in memory so the next plan sees it.

RB-001 (payment timeouts), RB-003 (kill a flag), outcome M-0002.

# RB-003: Kill a feature flag > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms A guardrail has breached, or a human has decided a rollout must stop now. Lengo Core kills automatically when `kill.mode` is `automatic`. With `alert_only`, it posts the evidence and waits for `/lengo approve`.

Step 1. Lengo Core sets the flag's `active` field to false through the GitLab Feature Flags API. Kills are allowed during freeze windows and are never blocked by budget or hop limits. No agent does this; agents only explain it.

Step 2. Verify the flag is inactive in GitLab (Deployments → Feature flags) and that the kill record on the goal issue shows the API call succeeded.

Step 3. Wait one client refresh interval (PesaLite refreshes every 5 seconds). New payments in the formerly flag-on cohort should now take the flag-off path.

Step 4. Record the time to kill: from the moment the breach was detected to the moment clients stopped seeing the flag. Lengo stores this in `flag_changes.time_to_kill_ms`.

Step 5. Leave the flag off. Do not re-enable it to "check again": a new rollout starts from the first step with a fixed change.

Step 6. The goal moves to `killed`, the Commander opens an incident and the Scribe reopens the goal with the lesson.

RB-002 (high 5xx), ADR-001 (flags over redeploys), ADR-002 (decisions in code).

# RB-004: Sample ratio mismatch (SRM) > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms The rollout holds with the reason "sample ratio mismatch": the share of users in the flag-on arm differs from what the rollout step should produce (for example 52% at the 50% step). While SRM persists, no result can be trusted, and Lengo never calls the goal proven.

Step 1. Read the SRM numbers in the decision record: units in each arm, expected share, observed share and the sequential p-value. A tiny p-value with a small share difference still matters at high traffic.

Step 2. Check that every client evaluates the flag with the user ID as context. Without a user ID, "percent of users" strategies cannot assign users consistently.

Step 3. Check client refresh. A client that stopped refreshing keeps serving an old percentage, so its users land in the wrong arm. PesaLite refreshes every 5 seconds; outcome M-0005 records a stale client that caused SRM.

Step 4. Check that exposure events are not lost for one arm only (for example, a crash on the new code path before the event is emitted). Compare event counts with request logs.

Step 5. Check for bots or load tests hitting only one arm. Synthetic traffic must be spread over stable user IDs.

Step 6. Fix the cause, then restart the step: Lengo resets the comparison at each step, and the hold clears once the split matches again.

ADR-002 (decisions in code), outcome M-0005.