Memory

Outcome M-0007 for goal G-0003: Reduce failed payments caused by gateway timeouts by 20%. Change: Behind the flag, retry a gateway timeout at most once, after an exponential back-off with jitter of at least 200 ms, and send the same idempotency key on both calls. Also enforce a total time budget per payment: compute the remaining budget before the retry, shorten the per-attempt timeout so both attempts fit, and skip the retry if it cannot finish inside the budget. Keep the budget below the 1756 ms p95 threshold that killed M-0006. Do not use immediate retries or more than one retry (M-0002), (

In outcome M-0007 (Behind the flag, retry a gateway timeout at most once, after an exponential back-off with jitter of at least 200 ms, and send the same idempotency key on both calls. Also enforce a total time budget per payment: compute the remaining budget before the retry, shorten the per-attempt timeout so both attempts fit, and skip the retry if it cannot finish inside the budget. Keep the budget below the 1756 ms p95 threshold that killed M-0006. Do not use immediate retries or more than one retry (M-0002),), the goal metric failure_rate changed by -25.9% relative (control 0.10119675403

In outcome M-0007 (Behind the flag, retry a gateway timeout at most once, after an exponential back-off with jitter of at least 200 ms, and send the same idempotency key on both calls. Also enforce a total time budget per payment: compute the remaining budget before the retry, shorten the per-attempt timeout so both attempts fit, and skip the retry if it cannot finish inside the budget. Keep the budget below the 1756 ms p95 threshold that killed M-0006. Do not use immediate retries or more than one retry (M-0002),), guardrail http_5xx_rate was ok: control None, treatment None.

In outcome M-0007 (Behind the flag, retry a gateway timeout at most once, after an exponential back-off with jitter of at least 200 ms, and send the same idempotency key on both calls. Also enforce a total time budget per payment: compute the remaining budget before the retry, shorten the per-attempt timeout so both attempts fit, and skip the retry if it cannot finish inside the budget. Keep the budget below the 1756 ms p95 threshold that killed M-0006. Do not use immediate retries or more than one retry (M-0002),), guardrail p95_latency_ms was ok: control None, treatment None.

Outcome M-0006 for goal G-0003: Reduce failed payments caused by gateway timeouts by 20%. Change: Behind the flag, retry a payment call that timed out at the gateway at most once, after an exponential back-off with jitter, and send the same idempotency key on the original call and the retry so an in-flight attempt cannot be duplicated. Do not use immediate retries or more than one retry, because M-0002 showed that three immediate retries without an idempotency key raised 5xx tenfold, and recovered payments came from the first retry only. Keep the old path unchanged when the flag is off, add (

In outcome M-0006 (Behind the flag, retry a payment call that timed out at the gateway at most once, after an exponential back-off with jitter, and send the same idempotency key on the original call and the retry so an in-flight attempt cannot be duplicated. Do not use immediate retries or more than one retry, because M-0002 showed that three immediate retries without an idempotency key raised 5xx tenfold, and recovered payments came from the first retry only. Keep the old path unchanged when the flag is off, add ), the goal metric failure_rate changed by n/a relative (control None, treatment

In outcome M-0006 (Behind the flag, retry a payment call that timed out at the gateway at most once, after an exponential back-off with jitter, and send the same idempotency key on the original call and the retry so an in-flight attempt cannot be duplicated. Do not use immediate retries or more than one retry, because M-0002 showed that three immediate retries without an idempotency key raised 5xx tenfold, and recovered payments came from the first retry only. Keep the old path unchanged when the flag is off, add ), guardrail http_5xx_rate was ok: control None, treatment None.

In outcome M-0006 (Behind the flag, retry a payment call that timed out at the gateway at most once, after an exponential back-off with jitter, and send the same idempotency key on the original call and the retry so an in-flight attempt cannot be duplicated. Do not use immediate retries or more than one retry, because M-0002 showed that three immediate retries without an idempotency key raised 5xx tenfold, and recovered payments came from the first retry only. Keep the old path unchanged when the flag is off, add ), guardrail p95_latency_ms was breach: control None, treatment None.

Read the flag once per request with the user ID as context: `flags.is_enabled(FLAG_KEY, user_id)`. Keep the old behaviour as the flag-off path, unchanged. The new behaviour lives only in the flag-on path. Never read a flag inside a loop or a retry, because the answer could change mid-request. In PesaLite, a goal's change is wired in `apps/pesalite/app/release.py`: `FLAG_KEY` (the planned key, exactly), `TREATMENT_RETRY_POLICY` (what the flag turns on, one of the policies in `app/retry.py`), and `in_treatment`, which returns `flags.is_enabled("<flag key>", user_id)` with the key written out. P

> Synthetic architecture decision record for LengoAI.

Accepted.

A release that hurts users must be reversed in seconds. Rolling back a deployment takes minutes: build, push, deploy and warm-up. It also reverses every other change in that deployment. We also need to compare users with and without a change at the same moment, which a deployment can't do.

Every user-facing change ships dark, behind a feature flag that is off by default. Deployment and release are separate steps. Lengo Core releases by changing the flag's percentage through the GitLab Feature Flags API, and kills by deactivating the flag. Clients refresh flag state every 5 seconds.

A kill is one API call plus one client refresh, so the change is undone for users within seconds, without a redeploy. Flag-on and flag-off users form concurrent cohorts, which is what the statistics engine compares. The cost is code that carries both paths until the Scribe's cleanup merge request removes the old one, and a per-project limit on the number of flags.

> Synthetic architecture decision record for LengoAI.

Accepted.

LengoAI's agents are language models. They plan, write code and explain well, but they can be wrong in confident ways, they vary between runs, and they can't be held to an error rate. Expanding a rollout, killing a flag, merging, closing a goal or writing memory changes what users experience or what the system believes. Those decisions need to be reproducible and auditable.

LLMs propose, code disposes. Agents produce plans, code and explanations, and they hand off through a JSON block that Core validates. Lengo Core makes every decision that has consequences, using deterministic, tested code: the policy engine (`check_policy`), the statistics engine (`decide`), the flag controller and the state machine. The Analyst explains a decision but can't change it.

The same data always gives the same decision, and every decision is written to the audit log with the numbers behind it. Statistical guarantees (false-alarm rates) are measured by simulation, not hoped for. Agents can still be wrong, but only in places a human or a check can catch: a merge request that fails the pipeline, a plan that fails validation, or a citation that wasn't supplied.

GitLab Duo reads this file automatically, and Lengo indexes it into memory as a convention (`AGENTS.md#<section>`). Humans follow the same rules.

**LLMs propose, code disposes.** Agents plan, write code and explain. Lengo Core makes every decision that has consequences: accepting a handoff, merging, changing or killing a feature flag, calling a goal `proven`, closing or reopening a goal, and writing memory. Never try to do any of these yourself, and never claim they happened unless Core's note says so.

| Path | What lives there | May agents edit it? | |---|---|---| | `apps/pesalite/app/` | The demo payments app that Lengo releases | **Yes**: this is where goal changes go | | `apps/pesalite/tests/` | PesaLite tests | **Yes**: every change adds or updates tests here | | `apps/pesalite/app/db.py` | Balance ledger storage | **No** (protected) | | `core/` | Lengo Core itself | **No** (protected: Core enforces the rules, so a change to it goes through an owner) | | `memory/` | Seed knowledge for RAG | Scribe only, through an MR | | `.gitlab/duo/` | Agent flows and prompts | No | | `infra/**`, `.gi

1. **Behind a feature flag.** New behaviour is off unless the flag is on. Read flags with `app.flags.is_enabled("<flag_key>", user_id)`. The flag key is `goal_<goal number>_<short_name>`, for example `goal_0007_gateway_retry`. The old code path must keep working when the flag is off. See `memory/conventions/feature-flags.md`. 2. **Tests required.** Every MR adds or updates tests that cover both flag states. 3. **Small changes.** At most `code.max_files_changed` files and `code.max_lines_changed` changed lines (see `lengo.yml`). Split anything larger. 4. **No secrets, no real data.** Never comm

- Branch: `lengo/<goal-id>-<short-slug>` (e.g. `lengo/G-0007-gateway-retry`). - Commits: imperative mood, a reference to the goal (`Refs G-0007`), and DCO sign-off (`Signed-off-by:`). - MR: use the `lengo` MR template. Link the goal issue, name the flag key, list files and tests, state the risk class. - Never merge your own MR. Core merges when policy allows.

Every Lengo flow ends by posting **one** note on the goal issue that contains a fenced block tagged `lengo-handoff` with JSON matching `core/lengo_core/schemas/handoff.v1.json`. After posting it, you are done: stop, and don't post anything else. Core validates the block and starts the next step. A malformed block is retried with the error message, up to two times.

Goals `G-0007` · memory records `M-0031` · runbook steps `RB-004#2` · incidents `INC-0003` · decision records `ADR-002` · ticket clusters `TC-0004`.

> Synthetic conventions for the PesaLite demo, written for LengoAI's agents and its human reviewers.

Python 3.12 with type hints everywhere. Code must pass `ruff check`, `ruff format --check` and `mypy --strict`. Use small functions with one job each. Docstrings explain why a function exists and what it guarantees, not only what it does.

Never swallow exceptions with a bare `except`. Catch the specific errors you expect and let everything else surface. Every call to another service has a timeout. Retries use exponential back-off with jitter and are limited in number. A retry of a payment always reuses the same idempotency key, so the gateway can recognise the repeat.

Every change adds or updates tests in `apps/pesalite/tests/`. A flag-guarded change is tested in both flag states: the old path with the flag off and the new path with the flag on. Tests use the seeded gateway simulator (`GatewaySimulator(seed=...)`), so results are deterministic.

Amounts are integer cents, never floats. The ledger (`apps/pesalite/app/db.py`) is a protected path: agents must not edit it. All demo data is synthetic. Never log or commit real names, phone numbers, emails or card numbers.

Keep merge requests small: at most 8 files and 400 changed lines (`code` in `lengo.yml`). The `policy-check` pipeline job enforces this. Split anything larger into separate goals.

> Synthetic conventions for the PesaLite demo.

Flag keys are `goal_<number>_<short_name>` in lower case, for example `goal_0001_gateway_retry`. Keys start with a letter, use only letters, digits and underscores, and never end with an underscore (GitLab's naming rule).

Lengo Core creates the flag with a "percent of users" strategy (`gradualRolloutUserId`). The group ID is fixed to the flag key, so a user who has the change at 1% keeps it at 10%, 50% and 100%. Steps only increase (1 → 10 → 50 → 100). Each step waits at least `rollout.min_dwell_minutes`, and the statistics engine must say `expand`. The step to 100% also needs the goal to be proven.

Only Lengo Core changes or kills flags; agents never call the Feature Flags API. After a goal closes, the Scribe opens a cleanup merge request that removes the flag check and the dead code path, because stale flags make the code harder to reason about. GitLab.com allows 50 to 200 flags per project, depending on the tier.

# RB-001: Payment gateway timeouts > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms Failed payments rise and most failures carry the reason `gateway_timeout`. Users see "payment could not be completed, please retry". Support tickets mention payments "hanging", "timing out" or "stuck on processing". Latency p95 sits close to the client timeout (1,500 ms), because timed-out attempts are counted at the timeout value.

Step 1. Confirm the pattern: compare the `gateway_timeout` share of failed payments for the last hour with the previous day. A rise above 5 points is a timeout incident, not ordinary declines.

Step 2. Split by cohort. If a feature flag is rolling out, compare the flag-on and flag-off arms. If only the flag-on arm is worse, the change is the suspect: follow RB-003 to switch it off.

Step 3. Check the retry policy of anything that calls the gateway. A retry must wait with exponential back-off and jitter (at least 200 ms before the first retry) and must carry an idempotency key. Immediate retries without a key collide with the attempt the gateway is still processing and fail with `duplicate_in_flight` (a 5xx).

Step 4. Look for `duplicate_in_flight` and `gateway_error` in the 5xx breakdown. A rise in either alongside a retry change is the retry-storm signature recorded in outcome M-0002.

Step 5. Limit retries to one attempt for payment calls. More attempts multiply load on a gateway that is already slow, and they rarely recover the payment.

Step 6. If the gateway itself is degraded (timeouts rise equally in both arms), do not ship a retry change during the incident. Hold the rollout and open an incident with the gateway status attached.

RB-002 (high 5xx), RB-003 (kill a flag), ADR-001 (flags over redeploys), outcome M-0002.

# RB-002: High rate of 5xx responses > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms The `http_5xx_rate` guardrail turns amber or red on the dashboard. PesaLite returns 502 for upstream faults (`gateway_error`, `duplicate_in_flight`). The usual baseline is about 0.3%. Anything above 2% in the flag-on arm breaches the hard limit in `lengo.yml`.

Step 1. Check whether the rise is in one arm or both. Lengo only kills when the flag-on arm is confidently worse than control, so a rise in both arms points to the platform or the gateway, not the change.

Step 2. Read the kill record on the goal issue: it states the guardrail, the observed rates in each arm, the confidence bound that crossed the limit and the time to kill.

Step 3. Break 5xx down by reason. `duplicate_in_flight` means a retry collided with an attempt still in progress (see RB-001 step 3). `gateway_error` without a matching change means the gateway is degraded.

Step 4. Confirm the flag is off (RB-003 step 2) and watch the flag-on arm's 5xx rate fall back to baseline within one refresh interval (5 seconds) plus one ingest batch.

Step 5. Open an incident with the guardrail data, the reason breakdown and the commit that introduced the change. Draft a fix as a merge request, but never merge it automatically.

Step 6. Before re-planning, write the lesson down: the Scribe stores it in memory so the next plan sees it.

RB-001 (payment timeouts), RB-003 (kill a flag), outcome M-0002.

# RB-003: Kill a feature flag > Synthetic runbook for the PesaLite demo. Not operational guidance for any real system. ## Symptoms A guardrail has breached, or a human has decided a rollout must stop now. Lengo Core kills automatically when `kill.mode` is `automatic`. With `alert_only`, it posts the evidence and waits for `/lengo approve`.

Step 1. Lengo Core sets the flag's `active` field to false through the GitLab Feature Flags API. Kills are allowed during freeze windows and are never blocked by budget or hop limits. No agent does this; agents only explain it.

Step 2. Verify the flag is inactive in GitLab (Deployments → Feature flags) and that the kill record on the goal issue shows the API call succeeded.

Step 3. Wait one client refresh interval (PesaLite refreshes every 5 seconds). New payments in the formerly flag-on cohort should now take the flag-off path.

Step 4. Record the time to kill: from the moment the breach was detected to the moment clients stopped seeing the flag. Lengo stores this in `flag_changes.time_to_kill_ms`.

Step 5. Leave the flag off. Do not re-enable it to "check again": a new rollout starts from the first step with a fixed change.

Step 6. The goal moves to `killed`, the Commander opens an incident and the Scribe reopens the goal with the lesson.

RB-002 (high 5xx), ADR-001 (flags over redeploys), ADR-002 (decisions in code).