G-0003 · Reduce failed payments caused by gateway timeouts by 20%

closed cycle 2 · flag goal_0003_c2_timeout_retry_budget · issue #4 · MR !5

## Goal Reduce failed payments caused by gateway timeouts by 20%. ## Why it matters Customers whose payment times out at the gateway see "payment could not be completed, please retry" and often give up. Support-ticket themes in Lengo's memory point to timeouts as the main cause of failed payments. (All data in this project is synthetic.) ## How we will measure it | | | |---|---| | Goal metric | failure_rate | | Direction | decrease | | Target | 20% | | Guardrails that must not get worse | defaults from lengo.yml: http_5xx_rate, p95_latency_ms | ## Autonomy Preset from lengo.yml (Hands-off). ## Constraints No changes to the ledger, no new dependencies. --- _LengoAI picks this up because of the `lengo` label. Live smoke test, Run 1 profile (third take)._

Goal metric: failure rate

proven

always-valid interval estimate goal

With the flag on: -2.62 pts versus off (always-valid interval -2.95 pts to -2.29 pts), a relative change of -25.9%. Control rate 10.12%; 177569 events with the flag on, 229454 off. Method: msprt.

Guardrails

must not get worse

always-valid interval estimate kill line (tolerance)

http 5xx rate: ok within limits

p95 latency ms: ok treatment p95 within tolerance

Latest decision

#2752 at 100% · 2026-10-25 18:38 UTC · tick

goal proven and guardrails clean at 100%

What happened

55 events

flow2026-10-24 15:23 UTC · core

lengo-plan-build started (plan, attempt 1)

session completed
handoff2026-10-24 15:24 UTC · ai-lengo-plan-build-gitlab-ai-hackathon

strategist → builder: done accepted, citations 100% valid

Plan a flag-guarded single gateway-timeout retry with back-off, jitter and an idempotency key to cut failure_rate by 20% while avoiding the 5xx retry storm seen in M-0002.
state2026-10-24 15:24 UTC · core

proposed → planned

Strategist plan validated
handoff2026-10-24 15:24 UTC · ai-lengo-plan-build-gitlab-ai-hackathon

builder → gatekeeper: done accepted, citations 100% valid

Flag-guarded single gateway-timeout retry with exponential back-off, jitter and idempotency key, wired in release.py with tests for flag on and off.
state2026-10-24 15:24 UTC · core

planned → built

Builder opened !3
flow2026-10-24 15:33 UTC · core

lengo-gate started (gate, attempt 1)

session completed
handoff2026-10-24 15:34 UTC · ai-lengo-gate-gitlab-ai-hackathon

gatekeeper → launcher: pass accepted, citations 100% valid

MR !3 passes: pipeline green, policy-check passed, flag-guarded single back-off retry with idempotency key, tests cover both flag states, only allowed paths changed, within size limits, no secrets or real data. No critical findings. SAST (9, none critical) and the unfixable high-severity base-image CVEs are not introduced by this MR. Image digest not available to me (registry 403, empty package log); SBOM is on the container_scanning_pesalite job.
state2026-10-24 15:34 UTC · core

built → gate_passed

Gatekeeper verdict: pass
state2026-10-24 15:42 UTC · core

gate_passed → staged

deployed to staging (deployment 1640160828)
flag2026-10-24 16:44 UTC · core

flag on at 1% (applied)

allow: all checks passed
state2026-10-24 16:44 UTC · core

staged → rollout_1

flag goal_0003_timeout_retry on for 1% of users
flow2026-10-24 16:44 UTC · core

lengo-launch started (launch, attempt 1)

session completed
handoff2026-10-24 16:45 UTC · ai-lengo-launch-gitlab-ai-hackathon

launcher → analyst: done accepted, citations 100% valid

Release notes for G-0003 (MR !3, flag goal_0003_timeout_retry, currently at 1%): a single back-off retry with an idempotency key for gateway timeouts, rollout steps 1/10/50/100 each gated by a statistics decision and minimum dwell, guardrails on http_5xx_rate and p95_latency_ms, and automatic switch-off by Core.
flag2026-10-24 17:00 UTC · core

flag at 10% (applied)

allow: all checks passed
state2026-10-24 17:00 UTC · core

rollout_1 → rollout_10

flag expanded to 10%
flow2026-10-24 17:00 UTC · core

lengo-analyze started (analyze_17, attempt 1)

session completed
flag2026-10-24 17:01 UTC · core

flag off (applied) time to kill 620 ms

guardrail p95_latency_ms: more than 5% of treatment requests exceed 1756 ms ((1+0.2) x control p95); lower bound 0.054
state2026-10-24 17:01 UTC · core

rollout_10 → killed

guardrail p95_latency_ms: more than 5% of treatment requests exceed 1756 ms ((1+0.2) x control p95); lower bound 0.054
flow2026-10-24 17:01 UTC · core

lengo-incident started (incident, attempt 1)

session completed
handoff2026-10-24 17:01 UTC · ai-lengo-analyze-gitlab-ai-hackathon

analyst → core: done accepted, citations 100% valid

Core expanded the flag from 1% to 10%: guardrails clean, failure_rate down 3.5% relative but the interval (-5.73 to +5.02 points) includes zero so nothing is proven yet, p95 latency is the guardrail to watch, and the sample ratio check shows no mismatch.
handoff2026-10-24 17:02 UTC · ai-lengo-incident-gitlab-ai-hackathon

commander → scribe: done accepted, citations 100% valid

Core's fast path killed goal_0003_timeout_retry in 620 ms on a p95_latency_ms breach (more than 5% of treatment requests above 1756 ms, lower bound 0.054); 5xx was not the trigger. Hypothesis: the single back-off retry plus a second 1,500 ms timeout puts retried payments around 3.2 s, matching treatment p95 of 3,242 ms. This is not a repeat of M-0002 (5xx retry storm). No fix MR was opened because no small, safe fix is clear; the flag stays off and a new goal cycle is needed.
flow2026-10-24 17:02 UTC · core

lengo-scribe started (scribe, attempt 1)

session completed
handoff2026-10-24 17:03 UTC · ai-lengo-scribe-gitlab-ai-hackathon

scribe → core: done accepted, citations 100% valid

Goal killed on the p95_latency_ms guardrail, not 5xx; failure_rate was -3.5% relative at 1% and unproven. Recommend reopen with a plan that bounds total retry latency.
state2026-10-24 17:03 UTC · core

killed → reported

Scribe posted the outcome report
state2026-10-24 18:02 UTC · core

reported → reopened

allow: all checks passed
flow2026-10-24 18:02 UTC · core

lengo-plan-build started (plan, attempt 1)

session completed
handoff2026-10-24 18:03 UTC · ai-lengo-plan-build-gitlab-ai-hackathon

strategist → builder: done accepted, citations 100% valid

Plan a flag-guarded single idempotent back-off retry with a total per-payment time budget, to cut failure_rate by 20% without repeating the M-0002 5xx storm or the M-0006 p95 latency breach.
state2026-10-24 18:03 UTC · core

reopened → planned

Strategist plan validated
handoff2026-10-24 18:07 UTC · ai-lengo-plan-build-gitlab-ai-hackathon

builder → gatekeeper: done accepted, citations 100% valid

Flag-guarded single idempotent back-off retry with a 1,600 ms per-payment budget (first attempt cut at 600 ms), wired in release.py as the capped_budget policy, with tests for flag on and off and for the budget cap.
state2026-10-24 18:07 UTC · core

planned → built

Builder opened !5
flow2026-10-24 18:15 UTC · core

lengo-gate started (gate, attempt 1)

session completed
handoff2026-10-24 18:16 UTC · ai-lengo-gate-gitlab-ai-hackathon

gatekeeper → launcher: pass accepted, citations 100% valid

MR !5 passes: pipeline green, policy-check passed, flag-guarded single back-off retry with idempotency key and a 1,600 ms budget, tests cover both flag states, only allowed paths changed, within size limits, no secrets or real data. No critical findings. The 9 SAST findings are all in files this MR does not touch. Image digest not available to me (registry 403, empty package log); SBOM is on the container_scanning_pesalite job.
state2026-10-24 18:16 UTC · core

built → gate_passed

Gatekeeper verdict: pass
state2026-10-24 18:24 UTC · core

gate_passed → staged

deployed to staging (deployment 1640232423)
flag2026-10-24 18:27 UTC · core

flag on at 1% (applied)

allow: all checks passed
state2026-10-24 18:27 UTC · core

staged → rollout_1

flag goal_0003_c2_timeout_retry_budget on for 1% of users
flow2026-10-24 18:27 UTC · core

lengo-launch started (launch, attempt 1)

session completed
handoff2026-10-24 18:28 UTC · ai-lengo-launch-gitlab-ai-hackathon

launcher → analyst: done accepted, citations 100% valid

Release notes for G-0003: MR !5 (gate pass) adds a budgeted single idempotent back-off retry behind goal_0003_c2_timeout_retry_budget, live for 1% of users, growing 1/10/50/100% with a statistics decision and minimum dwell per step, protected by http_5xx_rate and p95_latency_ms guardrails with automatic switch-off by Core.
flag2026-10-24 20:40 UTC · core

flag at 10% (applied)

allow: all checks passed
state2026-10-24 20:40 UTC · core

rollout_1 → rollout_10

flag expanded to 10%
flow2026-10-24 20:40 UTC · core

lengo-analyze started (analyze_345, attempt 1)

session completed
handoff2026-10-24 20:41 UTC · ai-lengo-analyze-gitlab-ai-hackathon

analyst → core: done accepted, citations 100% valid

Core expanded from 1% to 10%. failure_rate is 12.0% lower relative to control, but the interval (-0.0594 to +0.0350) includes zero, so it is not proven. http_5xx_rate and p95_latency_ms are within limits and the sample ratio check shows no mismatch.
flag2026-10-24 20:46 UTC · core

flag at 50% (applied)

allow: all checks passed
state2026-10-24 20:46 UTC · core

rollout_10 → rollout_50

flag expanded to 50%
flow2026-10-24 20:46 UTC · core

lengo-analyze started (analyze_357, attempt 1)

session completed
handoff2026-10-24 20:46 UTC · ai-lengo-analyze-gitlab-ai-hackathon

analyst → core: done accepted, citations 100% valid

Core expanded from 10% to 50%. failure_rate is 19.4% lower relative to control (estimate -0.0199, interval -0.0397 to -0.0001, p-value 0.048), so the interval excludes zero, just short of the 20% target. http_5xx_rate and p95_latency_ms are within limits and the sample ratio check shows no mismatch.
flag2026-10-25 18:32 UTC · core

flag at 100% (applied)

allow: all checks passed
state2026-10-25 18:32 UTC · core

rollout_50 → rollout_100

flag expanded to 100%
flow2026-10-25 18:32 UTC · core

lengo-analyze started (analyze_2740, attempt 1)

session completed
handoff2026-10-25 18:33 UTC · ai-lengo-analyze-gitlab-ai-hackathon

analyst → core: done accepted, citations 100% valid

Core expanded from 50% to 100%. failure_rate is 25.9% lower relative to control (estimate -0.0262, interval -0.0295 to -0.0229, p-value 1.3e-151), so the interval excludes zero and the 20% target is exceeded. http_5xx_rate and p95_latency_ms are within limits and the sample ratio check shows no mismatch.
state2026-10-25 18:38 UTC · core

rollout_100 → proven

goal proven and guardrails clean at 100%
flow2026-10-25 18:38 UTC · core

lengo-scribe started (scribe, attempt 1)

session completed
handoff2026-10-25 18:39 UTC · ai-lengo-scribe-gitlab-ai-hackathon

scribe → core: done accepted, citations 100% valid

Goal proven at 100%: failure_rate down 25.9% relative, both guardrails clean. Recommend close.
state2026-10-25 18:39 UTC · core

proven → reported

Scribe posted the outcome report
state2026-10-25 18:39 UTC · core

reported → closed

allow: all checks passed

Memory the agents cited

71 / 71 supplied

ID	Cited by	Source
M-0002	strategist	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	strategist	M-0002 > failure_rate
RB-001#0	strategist	RB-001: Payment gateway timeouts > Overview
AGENTS#4	strategist	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
M-0002	builder	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	builder	M-0002 > failure_rate
RB-001#0	builder	RB-001: Payment gateway timeouts > Overview
AGENTS#4	builder	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
AGENTS#4	gatekeeper	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
AGENTS#3	gatekeeper	AGENTS.md: conventions for every agent working in this repository > 2. Repository map
CONV-feature-flags#3	gatekeeper	Feature flag conventions > Guarding a change
RB-001#3	gatekeeper	RB-001: Payment gateway timeouts > Step 3
M-0002	launcher	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	launcher	M-0002 > failure_rate
M-0005	launcher	Outcome M-0005: Lower gateway timeout from 3 s to 1.5 s to fail fast
M-0001	launcher	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%
AGENTS#4	launcher	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
CONV-feature-flags#3	launcher	Feature flag conventions > Guarding a change
M-0002	analyst	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	analyst	M-0002 > failure_rate
M-0001	analyst	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%
M-0004	analyst	Outcome M-0004: Reduce support tickets about fees by showing the fee before confirming
M-0002	commander	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	commander	M-0002 > failure_rate
RB-001#0	commander	RB-001: Payment gateway timeouts > Overview
RB-001#1	commander	RB-001: Payment gateway timeouts > Step 1
RB-002#0	commander	RB-002: High rate of 5xx responses > Overview
M-0002	scribe	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0002#failure_rate	scribe	M-0002 > failure_rate
M-0005#p95_latency_ms	scribe	M-0005 > p95_latency_ms
M-0006	strategist	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#failure_rate	strategist	M-0006 > failure_rate
M-0006#p95_latency_ms	strategist	M-0006 > p95_latency_ms
M-0006#http_5xx_rate	strategist	M-0006 > http_5xx_rate
M-0002	strategist	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
RB-001#0	strategist	RB-001: Payment gateway timeouts > Overview
M-0005	strategist	Outcome M-0005: Lower gateway timeout from 3 s to 1.5 s to fail fast
AGENTS#4	strategist	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
M-0006	builder	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#p95_latency_ms	builder	M-0006 > p95_latency_ms
M-0002	builder	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
RB-001#0	builder	RB-001: Payment gateway timeouts > Overview
AGENTS#4	builder	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
AGENTS#4	gatekeeper	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
AGENTS#3	gatekeeper	AGENTS.md: conventions for every agent working in this repository > 2. Repository map
CONV-feature-flags#3	gatekeeper	Feature flag conventions > Guarding a change
RB-001#3	gatekeeper	RB-001: Payment gateway timeouts > Step 3
RB-001#0	gatekeeper	RB-001: Payment gateway timeouts > Overview
M-0006	launcher	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0002	launcher	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0006#failure_rate	launcher	M-0006 > failure_rate
M-0006#p95_latency_ms	launcher	M-0006 > p95_latency_ms
M-0006#http_5xx_rate	launcher	M-0006 > http_5xx_rate
M-0005	launcher	Outcome M-0005: Lower gateway timeout from 3 s to 1.5 s to fail fast
AGENTS#4	launcher	AGENTS.md: conventions for every agent working in this repository > 3. Rules for every code change
M-0006	analyst	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#p95_latency_ms	analyst	M-0006 > p95_latency_ms
M-0002	analyst	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0001	analyst	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%
M-0006	analyst	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#p95_latency_ms	analyst	M-0006 > p95_latency_ms
M-0002	analyst	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0001	analyst	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%
M-0006	analyst	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#p95_latency_ms	analyst	M-0006 > p95_latency_ms
M-0002	analyst	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0001	analyst	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%
M-0006	scribe	Outcome M-0006: Reduce failed payments caused by gateway timeouts by 20%
M-0006#p95_latency_ms	scribe	M-0006 > p95_latency_ms
M-0002	scribe	Outcome M-0002: Reduce failed payments caused by gateway timeouts by 20%
M-0001	scribe	Outcome M-0001: Reduce checkout abandonment on the payment form by 10%

Agents may cite only IDs Core supplied in the latest context note; anything else counts against citation validity.

Flag changes

When	Change	Status	Time to kill
2026-10-24 16:44 UTC	goal_0003_timeout_retry create → 1%	applied	—
2026-10-24 17:00 UTC	goal_0003_timeout_retry set percentage → 10%	applied	—
2026-10-24 17:01 UTC	goal_0003_timeout_retry deactivate	applied	620 ms
2026-10-24 18:27 UTC	goal_0003_c2_timeout_retry_budget create → 1%	applied	—
2026-10-24 20:40 UTC	goal_0003_c2_timeout_retry_budget set percentage → 10%	applied	—
2026-10-24 20:46 UTC	goal_0003_c2_timeout_retry_budget set percentage → 50%	applied	—
2026-10-25 18:32 UTC	goal_0003_c2_timeout_retry_budget set percentage → 100%	applied	—

Audit trail

hash chain verified

#	When	Actor	Action	Subject
105	2026-10-25 18:39 UTC	core	goal.transition	reported->closed
104	2026-10-25 18:39 UTC	ai-lengo-scribe-gitlab-ai-hackathon	handoff.accepted	scribe
103	2026-10-25 18:38 UTC	core	flow.start	lengo-scribe
102	2026-10-25 18:38 UTC	core	goal.transition	rollout_100->proven
101	2026-10-25 18:33 UTC	ai-lengo-analyze-gitlab-ai-hackathon	handoff.accepted	analyst
100	2026-10-25 18:32 UTC	core	flow.start	lengo-analyze
99	2026-10-25 18:32 UTC	core	goal.transition	rollout_50->rollout_100
98	2026-10-25 18:32 UTC	core	flag.set_percentage	goal_0003_c2_timeout_retry_budget
91	2026-10-24 20:46 UTC	ai-lengo-analyze-gitlab-ai-hackathon	handoff.accepted	analyst
90	2026-10-24 20:46 UTC	core	flow.start	lengo-analyze
89	2026-10-24 20:46 UTC	core	goal.transition	rollout_10->rollout_50
88	2026-10-24 20:46 UTC	core	flag.set_percentage	goal_0003_c2_timeout_retry_budget
87	2026-10-24 20:41 UTC	ai-lengo-analyze-gitlab-ai-hackathon	handoff.accepted	analyst
86	2026-10-24 20:40 UTC	core	flow.start	lengo-analyze
85	2026-10-24 20:40 UTC	core	goal.transition	rollout_1->rollout_10
84	2026-10-24 20:40 UTC	core	flag.set_percentage	goal_0003_c2_timeout_retry_budget
83	2026-10-24 18:28 UTC	ai-lengo-launch-gitlab-ai-hackathon	handoff.accepted	launcher
82	2026-10-24 18:27 UTC	core	flow.start	lengo-launch
81	2026-10-24 18:27 UTC	core	goal.transition	staged->rollout_1
80	2026-10-24 18:27 UTC	core	flag.create	goal_0003_c2_timeout_retry_budget
79	2026-10-24 18:24 UTC	core	goal.transition	gate_passed->staged
77	2026-10-24 18:16 UTC	core	mr.merged	!5
76	2026-10-24 18:16 UTC	ai-lengo-gate-gitlab-ai-hackathon	handoff.accepted	gatekeeper
75	2026-10-24 18:15 UTC	core	flow.start	lengo-gate
74	2026-10-24 18:07 UTC	core	goal.transition	planned->built
73	2026-10-24 18:07 UTC	ai-lengo-plan-build-gitlab-ai-hackathon	handoff.accepted	builder
72	2026-10-24 18:03 UTC	ai-lengo-plan-build-gitlab-ai-hackathon	handoff.accepted	strategist
71	2026-10-24 18:02 UTC	core	flow.start	lengo-plan-build
70	2026-10-24 18:02 UTC	core	goal.transition	reported->reopened
57	2026-10-24 17:03 UTC	ai-lengo-scribe-gitlab-ai-hackathon	handoff.accepted	scribe
55	2026-10-24 17:02 UTC	core	flow.start	lengo-scribe
54	2026-10-24 17:02 UTC	ai-lengo-incident-gitlab-ai-hackathon	handoff.accepted	commander
51	2026-10-24 17:01 UTC	ai-lengo-analyze-gitlab-ai-hackathon	handoff.accepted	analyst
50	2026-10-24 17:01 UTC	core	flow.start	lengo-incident
49	2026-10-24 17:01 UTC	core	goal.transition	rollout_10->killed
48	2026-10-24 17:01 UTC	core	flag.kill	goal_0003_timeout_retry
47	2026-10-24 17:01 UTC	core	decision.kill	10%
46	2026-10-24 17:00 UTC	core	flow.start	lengo-analyze
45	2026-10-24 17:00 UTC	core	goal.transition	rollout_1->rollout_10
44	2026-10-24 17:00 UTC	core	flag.set_percentage	goal_0003_timeout_retry
43	2026-10-24 16:45 UTC	ai-lengo-launch-gitlab-ai-hackathon	handoff.accepted	launcher
42	2026-10-24 16:44 UTC	core	flow.start	lengo-launch
41	2026-10-24 16:44 UTC	core	goal.transition	staged->rollout_1
40	2026-10-24 16:44 UTC	core	flag.create	goal_0003_timeout_retry
38	2026-10-24 16:15 UTC	core	flag.create.failed	goal_0003_timeout_retry
36	2026-10-24 16:00 UTC	core	policy.deny	create
35	2026-10-24 15:45 UTC	core	policy.deny	create
34	2026-10-24 15:42 UTC	core	goal.transition	gate_passed->staged
32	2026-10-24 15:35 UTC	core	mr.merged	!3
31	2026-10-24 15:34 UTC	ai-lengo-gate-gitlab-ai-hackathon	handoff.accepted	gatekeeper

Full audit log for G-0003 →