Changelog
Public changelog rendered from the project’s CHANGELOG.md. Sprint sign-off documents carry the detailed change story; this page is the one-line index.
EPIC — Changelog
All notable changes to EPIC. Follows Semantic
Versioning. One-line entries per merged PR,
each pointing at the sprint sign-off that closed the work — the
detailed change story lives in the sprint document, not here
(sprints/CLAUDE.md §4).
Format reference: Keep a Changelog, with section ordering Added / Changed / Deprecated / Removed / Fixed / Security.
[Unreleased]
Phase 10 — Federated sign-in: 🔧 IN PROGRESS — planned 2026-09-14, opened 2026-09-19 at Sprint 10.1. ADR-068 (Proposed) and the six sprint documents under sprints/phase-10/. Documentation only; no code or deployment change.
❄️ SUBSTRATE-CORRECTION FREEZE — FerraVault — 2026-06-01. Sprint 1.7 retracts every prior CHANGELOG framing that describes EPIC's
epic-vaultadapter as "an HTTP client" / "hyper-based" / "talks to the/v1/secret/{key}endpoint" / "uses theX-Vault-Tokenheader" / "long-polls/v1/revisionfor hot-reload events". FerraVault does not have any HTTP/REST surface. FerraVault is gRPC-over-TLS-1.3-only via the officialferravault-clienttonic crate, withVault.KvGet/KvPut/KvDelete/Watch(server-streaming) RPCs and bearer-token or mTLS-bind authentication. Sprint 0.5 shipped the wrong substrate (hand-rolled hyper + JSON envelopes against URL paths that don't exist on FerraVault's wire); Sprint 1.3 + Sprint 1.4 extended it; Phase 1 sign-off (phase-1-final, 2026-05-31) and the Ferrum substrate-retrofit tag (phase-1-final-ferrum-retrofit, 2026-05-31) both built on it. The only reason CI shipped green is thatepic-vaultwas exercised exclusively againstwiremockstand-ins and the in-memoryMemKeyVaultfake — no integration test ever booted a realferravaultdaemon. Caught 2026-05-31 during the post-Sprint-1.6 cross-project audit; rework lands in Sprint 1.7. ADR-025 ratifies the corrected substrate. The standing project rule is recorded inproject_ferravault_is_grpc_only. Until Sprint 1.7 closes, no CHANGELOG entry below that describesepic-vault,VaultClient, the watch path, or any FerraVault-shaped wire as "production-grade" or "complete" stands — the wire never matched the real FerraVault. Sprint 1.7 also bumps the dev-stack PostgreSQL pin frompostgres:16-alpinetopostgres:18-bookwormper the operator's "Postgres should at least be 18" directive and the ChatXP reference architecture.
❄️ SUBSTRATE-CORRECTION FREEZE — Ferrum — 2026-05-31. ✅ LIFTED 2026-06-01. During Sprint 2.2 close-out the operator caught a structural defect that invalidated a portion of the post-Sprint-0.5 work-product: every
Ferrum*-named EPIC adapter (epic-store::ferrum::FerrumClient,FerrumCache,FerrumMeter(Sprint 0.5 + Sprint 2.2 copies),FerrumRateLimiter,FerrumStream, the Sprint 2.2Aggregator, and everyLayeredStore/LayeredSessionLookupthat wraps them) was implemented against the Rustrediscrate speaking RESP3 to aredis://...URL — but Ferrum does not speak RESP3 at all. Ferrum is its own native data store with its own binary protocol (FBP — Ferrum Binary Protocol — over QUIC or Unix socket) and its own official Rust client (ferrum-client/ferrum-client-tokio). Sprint 0.5 shipped the wrong substrate; Phase 1 sign-off built on it; Sprint 2.2 inherited and amplified it. Resolution: the rework split into Sprint 1.6 — Ferrum substrate retrofit (Phase 1 retrofit, closed 2026-05-31, tagphase-1-final-ferrum-retrofit) + Sprint 2.0 — Ferrum substrate + meter rework (Phase 2 prerequisite, closed 2026-06-01). Both sprints have now closed; everyFerrum*adapter,epic-store::FerrumStream, theepic-meter::FerrumMeterproducer, theepic-meter::Aggregatorconsumer + tick, and the meter integration suite all run against the real Ferrum FBP wire surface (verified bycargo test -p epic-meter --test integration_aggregator— 7/7 against a liveferrum-serverchild process). The Sprint 2.2 doc remains in the tree as a historical record; the load-bearing sign-off for theepic-router+epic-meterpipeline is Sprint 2.0 §6. See Sprint 2.2 §8 Freeze block for the per-item disposition table and theproject_ferrum_is_not_redismemory for the standing rule. Phase 2's Sprint 2.1 (epic-ferron-client) was unaffected — it talks to Ferron (the LLM inference server), not to Ferrum.
Slot purchasing & access levels ✅ COMPLETE 2026-09-19. Tag: slot-purchasing-final (unsigned annotated — commit signing remains indefinitely deferred per root CLAUDE.md §9.1), at b18e5c5, the commit SP.4 and the sign-off landed on main. Named rather than numbered per ADR-064: the next free number is claimed by the federated sign-in plan, so a number here would collide with a phase that has not opened. SP.1 closed 2026-09-14, SP.4 2026-09-16, SP.2 2026-09-17, SP.3 (gating sign-off) 2026-09-19. 34 §2 items across the three production sprints, all ✅ Delivered; zero four-field deferrals. Three audits ✅ PASS: quality (workspace line 88.33 % — above the 80 % floor, above Phase 9's 86.51 %, and the highest recorded here; 2 889 tests / 137 binaries / 0 failed), security, architecture 77/77. Definitive ci.sh --full exit 0 (3 441 passed / 0 failed across 245 suites; 68m 17s; fuzz 6/6; conformance openai + anthropic over both plaintext and TLS; Stage-8 bench gate 0 regressed, 13 stable vs local-3826c6c). Two ADRs Accepted: ADR-067 (superseding four ADR-007 decisions) and ADR-069 (superseding ADR-067 §1 and §3 in part). The thesis: EPIC's commercial model became enforced rather than described — billing was a subscription writing period dates onto the organisation, payment webhooks could never find an organisation, and every signup was FoundingMember with seat_count = 0, so every organisation passed every gate. A slot is now a row with a kind, four rights, a status and an end; each right is decided by its own count; and the edge refuses a key whose account holds no slot carrying API access. What the gate found is the part worth remembering: five real defects, all fixed in-sprint, and three of them share one shape — a check that had been returning the right answer for the wrong reason. The ADR-026 pipeline-invariant command named a file that had stopped containing the import it polices; a dual-stack listener test asserted "both sockets are bound" about a socket that had never bound; and an epic-meter coverage figure had been carried by two prior phase audits as a live-substrate measurement gap when the substrate suites were running all along. The tracker's L1 (live merchant) is deliberately not closed by this phase — no merchant is live.
Phase 2 — Edge MVP, EPIC native ✅ COMPLETE 2026-06-02. Tag: phase-2-final (unsigned annotated — commit signing remains indefinitely deferred per DESIGN.md §20; a signed phase-2-final-signed tag will be created against the same history when signing eventually schedules). Sprint 2.1 closed 2026-05-31. Sprint 2.2 ❄️ FROZEN 2026-05-31 → Superseded by Sprint 2.0 2026-06-01. Sprint 2.0 closed 2026-06-01. Sprint 2.3 closed 2026-06-01. Sprint 2.4 closed 2026-06-02. Sprint 2.5 (gating sign-off) closed 2026-06-02. 67 distinct work-contract items across 5 sprints, all ✅ Delivered. Zero deferrals after the sign-off remediation pass — the original aarch64 link-stage four-field block was retired (root cause was a missing linker = "aarch64-linux-gnu-gcc" in /.cargo/config.toml; both binaries now link clean for aarch64-unknown-linux-gnu and execute under qemu-aarch64-static), and the original "tokenizer byte-parity property test" carry-forward was narrowed to cross-tool byte parity at the shim layer because the remediation pass added an edge-side tokenizer determinism gate covering everything testable inside Phase 2. One cosmetic carry-forward remains (epic-ferron-client/phase-2-final/ baseline rename → next cache-warm window). Three audits ✅ PASS: quality, security, architecture 27/27. Full cargo geiger workspace sweep at sign-off remediation confirms zero unsafe in EPIC-owned src/ except the documented Secret::<T>::into_inner ManuallyDrop discipline in epic-crypto. Five ADRs Accepted/Revised: ADR-009, ADR-010, ADR-011, ADR-021 (revised), ADR-026. Phase 1 — Control plane MVP ✅ COMPLETE 2026-05-31 (originally on substrate-incorrect Sprint 0.5 Ferrum stand-in; corrected via Sprint 1.6 + Sprint 1.7 → tags phase-1-final-ferrum-retrofit + phase-1-final-substrate-corrections).
Phase 8 — Coding-harness enablement (native tool-calling) ✅ COMPLETE 2026-07-03. Tag: phase-8-final (unsigned annotated — commit signing remains indefinitely deferred per DESIGN.md §20). Sprints 8.1 + 8.2 + 8.3 (gating sign-off) all closed 2026-07-03. 26 §2 task IDs across the two production sprints, all ✅ Delivered; zero four-field deferrals. Three audits ✅ PASS: quality (workspace line 85.16 %; epic-protocol::native::chat 100 %), security (tool surface held as an untrusted-input boundary; no server-side tool execution; the schema-bomb depth bound source-verified and made mechanical in ci.sh Stage 1), architecture 61/61. Definitive ci.sh --full exit 0 (2,620 tests / 209 suites / 0 failed; fuzz 6/6; Stage-8 bench gate 0 regressed vs phase-4-final; conformance openai==2.44.0 15/15 + anthropic==0.116.0 11/11 as Stage-4 gates). All three exit drills run for real against the live workstation Ferron tool build: the three-surface tool round-trip (14/14; every reassembled argument string valid JSON — the XGrammar-2 constrained-decoding guarantee) and a real Cline CLI agentic loop (grammar-constrained read_files → local execution → role:"tool" → answer). Two gate-caught defects fixed in-sprint, not deferred: the dev-fixture ADR-052 detok gap, and the tool-stream EOS special-token leak — fixed both sides (EPIC's ADR-052-revised scrub_special_markers defense-in-depth + Ferron's root-cause id-level strip shipped as v1.2.1 in response to EPIC's FR, validated live post-fix). Phase 8 benches frozen under phase-8-final/. Two ADRs Accepted: ADR-054, ADR-055; ADR-052 revised (special-token wire contract extended to the tool-content channel). Next: Phase 9 — Launch (see sprints/LAUNCH-READINESS-TRACKER.md).
Phase 3 — Shims (OpenAI + Anthropic SDK compatibility) ✅ COMPLETE 2026-06-07. Tag: phase-3-final (unsigned annotated — commit signing remains indefinitely deferred per DESIGN.md §20; a signed phase-3-final-signed tag will be created against the same history when signing eventually schedules). Sprint 3.1 + 3.2 closed 2026-06-02; Sprint 3.3 + 3.4 closed 2026-06-07; Sprint 3.5 (gating sign-off) closed 2026-06-07. 63 §2 task IDs across the four production sprints, all ✅ Delivered; zero four-field deferrals. Three audits ✅ PASS: quality, security, architecture 34/34. Conformance gates green (openai==2.41.0 11/11; anthropic==0.107.0 7/7). The plaintext JWT signing key is gone from epic-control process memory (ADR-030). The project's longest-running carry-forward (T-1.4.16 visual-regression, deferred Phase 1 → 2 → 3) retired in full. Five ADRs Accepted: ADR-027, ADR-028, ADR-029, ADR-030, ADR-031. Next: Phase 4 — Abuse + diagnostics + voting.
Added
-
The public pages get a light mode and the dashboard's two switches (2026-10-22, #395; Melvyn's request, stacked on #373). The public bar carries an animations button (a red pause while things move, a green play once they are still) and the dashboard's dark/light dot, the same buttons as the signed-in header's, beside Login. Pause stops every animation on the page, the robots' walks, the star glow and its scroll drift, the shooting stars and the badge tilt; Play resumes them, and outranks the system's reduced-motion setting. This is #355 finding 3's pause control (WCAG 2.2.2); a page with no script, and so no switch, is now still rather than running the robots' idle loops forever. The light mode puts the page on a daylight sky with the same stars as faint specks in the logo palette (
tools/make-starfield.py's day copy), the logo colours in darker shades where text needs AA (tools/palette-contrast.pynow checks the light palette and the switches), and the bar's and the comparison table'sEPICmark in the dashboard's dark grey (tools/make-header-mark.pywrites both inks). A first visit is dark, whatever the device asks, and its motion followsprefers-reduced-motion; both choices are remembered in the browser's localStorage, never sent to the server, applied before the first paint by one small hashed inline script (ADR-083, Accepted 2026-10-24). The signed-in dashboard goes the other way: a member who never chose a base sees the light one (migration 0095 makes it the column default too), and the Profile tab's colour box gains "Follow my device's light/dark setting", off by default, applied before the first paint by a hashed script at the top of the body; the header's dark/light button turns it off (ADR-073 §7). The signed-in header's animations switch takes the same play/pause disc, both its switches become toggle buttons witharia-pressed, and their forms lose a stray 1 rem margin, so the dashboard's pair is spaced like the public bar's; nothing else on the dashboard changes. Against baselines recorded before the change, every pixel that moved on the 29 baselined pages is inside the header bar. -
Robots on the landing page (owners' landing round, 2026-10-20). The dashboard's robot cast plays four short scenes on
/, each under the words it acts out: under the hardware footnote Pip lifts GPU cards out of a crate in its beam into a server under test while Strider checks them in; beside the electricity rate Max keeps feeding coins into the meter of a humming local rig, the meter turning at the visitor's own rate; under the principles robots drop their votes in a ballot box and Clerk writes each one down; under the join form a queue waits at a velvet rope Max lets them through one at a time, and once per visit Glitch tries to push in and is sent away. Same drawings and the sameActordirector as the dashboard (assets/scenes/actors.js), served as the same-origin/assets/js/landing-scenes.js; decorative (aria-hidden, nothing focusable), in fixed-height bands of their own (no layout shift, nothing covered, clipped stages), still underprefers-reduced-motionor with no script, and asleep off screen. The robots' stylesheet is split so the landing carries only what it draws:cast.css(the drawings and moves, shared),scenes.css(the dashboard's scenes, moved verbatim) andlanding-scenes.css. The visual-regression set gainslanding_full, the whole landing page top to bottom under reduced motion, stitched from viewport screenshots so either engine can take it (FULL_PAGES); its baseline is recorded after merge. -
epic-secret-sync exports metrics and is alerted on (2026-10-16, issue #224). The reconciler had no metrics surface: a renewed client certificate FerraVault refused, a run of failed reconciles or a Watch that never reconnected showed only in its log, and 30 days after a missed renewal every synced Secret went stale with no alert. It now serves
/metrics+/healthzon a required, dual-stackmetrics_bind(as epic-vault-audit does), exporting:- Watch health, reconcile outcomes and the last clean full resync;
- identity renewals and refusals, and the certificate's expiry.
The only label is the allow-listed
outcome. A PodMonitor scrapes it, and the network policy admits only the scrape. The shared identity alerts (EpicVaultIdentityRenewalRefused,…NotRenewed,…ExpiryImminent) now match its series, and two new tickets cover it:EpicSecretSyncNotConvergingandEpicSecretSyncWatchDown. All are promtool-tested. The FerraVault gate now checks its certificate's renewal window against the expiry rules. Nothing evaluates alerts yet (#157).
-
The coin desks race, talk prices and move the bar (owners, 2026-10-14). Under "Slot deposits received" and "Slots sold" the line is now a race: robots come in at speeds of their own, first come first served; now and then a faster one overtakes in front of or behind a slower one, and now and then Strider flies over the line on his rocket pack (neither often; the desk stands further right to leave room). At the desk Clerk asks "How many deposits?" or "How many slots?", the robot answers, and Clerk names the total: the deposit per slot, or the price per slot by the size of the purchase (single for 1, duo for 2, multi for more). A real rise in the box's figure is what the next robot says; Strider then flies up to the bar above the scene and pulls its end to the new figure (or pushes it back for a fall), the box's count changing with it. Coins carry a dollar sign; the slots-sold desk's three coins are gold; coins and papers are drawn in the robots' hands and move with them (Tread reaches out further with a paper, tilted). The desk wears the EPIC wordmark on its panels and a large one on a banner on two poles behind Clerk. The deposit per slot is set with the slot prices at
/app/admin/slot-prices(AUD 50 to start, never more than the multi price; migration 0084, audited with the prices). -
The project stages crew, all the way to the servers and on to the houses (owners, 2026-10-14). The crew starts at the first stage not yet done, whatever it is called, and works spot by spot along the strip of stages to the server rack: Strider breaks the ground with his jackhammer and steps backwards out of the way; Max, beside him, steps in and digs the spot out with a new spade (
has-shovel, front on,is-shoveling: the blade into the ground as his knees bend) in three spadefuls onto a heap that grows with each; Tread scoops the heap up in one go (has-load,is-scooping,is-carrying) while Pip lays the cable, the solid cable running out at Pip's own pace; Tread tips the earth back over it (is-dumping) and drives back and forth to pack it down. Clerk watches from the side, writing on his notepad or swapping it for the megaphone in his top hat (the hat hops off his head to meet his hand,is-swapping), shouting at the crew and pointing. At the rack the cable runs up into its bottom-left corner, the crew stands clear, the lights come on for a moment, and they start again. The dashed route now turns up into the rack's corner, and a second line leaves its other corner for three houses (a cottage, a tall house, a bungalow) under "Full launch", turning up into each; once "Servers installed" is done the crew works its way to the houses instead, lighting each one's windows. The rack stays under "Servers installed" and the houses under "Full launch" wherever those stages move; when the crew's target is the last stage, a stretch of plain ground follows it for the crew to stand on. The strip scrolls along with the crew unless the member has scrolled it in the last 15 s. The props and moves are copied from the robot designs, where they were added the same day. -
The launch scenes, round 2 (owners, 2026-10-14). The dig: Strider replaces Max as the repairer, flying over Tread on his rocket pack with his legs dangling; Tread faces left where the give-away bar's green meets its red, digs into the green and throws onto a pile behind him on the red; Strider flies to the pile, picks up a clod, flies back and drops it into the hole, and when the green has no room left for him he keeps to the pile and flies up from there. Tread's scoop now reaches down into the ground in every scene he digs in. The desks: the line pays one at a time, each robot first saying how many it is after (the box's real rise when its figure has moved, otherwise a number like the interest visitors'); with its paper it walks on past the desk and off the far end, and comes back in from the start with new coins after a while of its own, so the order keeps changing — there is always someone besides Clerk. The carnival: Max carries a two-handed sledgehammer on his shoulder (a new
has-sledgeprop for him, with anis-sledgingswing), walks between the buttons half as fast again, and hits a button from the side he walked in from — from the left going right, from the right going left — standing far enough off that the value he lights up is never behind him. Each button stands on its column's foot, its base 3px below it; the buttons, values and likeliness numbers are centred on their columns (they had sat 1% to the right); the values are 1.4rem, pop in and grow to 2rem before they fade; and the chart is taller, reaching about as far down as the entry form beside it. The question: Clerk now keeps the question mark and writes the numbers on a notepad, and Strider is one of the visitors; a visitor arriving while another is with Clerk waits in line behind it, and the line moves up. Clerk gains a notepad and pencil (has-pad,has-pencil, written on withis-writing) and a megaphone (has-megaphone, shouted through withis-shouting). OneGET /app/launch/figures(the interest total and entries, slots sold, slots secured by a deposit;no-store, signed-in members only) now feeds every scene, replacing the interest-only endpoint; each browser remembers the last figures it saw. -
The interest scenes, reworked (owners, 2026-10-14). Strider now stands by the question mark facing the viewer; visitors come from either end, he walks to their side of the question mark, points at it, turns back, the visitor says a number (1 to 5 nineteen times in twenty, otherwise up to 100) and he writes it on a clipboard; the visitor goes back or walks on at random, others wait their turn, never more than four robots on the bar. The numbers are real when the tracker moves: the scene compares the total with the last one this browser saw and asks
GET /app/launch/figuresevery 20 s while the page is open; a rise is what the next visitor says, a fall brings Glitch (now in the site's cast, only ever for this) with the drop in red. On the chart, the buttons sit at the columns' feet with the numbers underneath, Max walks the floor in front of them, and a column shows its value at its top once its light gets there. -
The robots wear the EPIC wordmark (owners, 2026-10-14). The header's wordmark, traced into one vector symbol (
ms-epicin the robots' shared defs; the server-rack "I" redrawn clean), printed on Clerk (a new top hat with the wordmark on its crown, a small badge on his vest), Strider (his shoulder, up his rocket pack, and the gold clasp of a new belt) and Max (across the top of his shield, the disc moved down below it, and on his shoulder). Every wordmark on a side view turns back the right way round when the robot faces left; on Max it goes round his front edge, cut at his outline. Strider's legs lean in a little in the front view and his hips sit inside his belt. The designs are the owners' approved standard of 2026-10-14. -
The question and the carnival in the interest tracker (owners' site-text round 1). Over the interest bar, Strider keeps a 3D question mark just before the middle; at random intervals Max, Tread, Pip or Clerk wander in from the left — never more than two at once — Strider jumps and points at the question mark, the wanderer stops by him, they look at each other, a bubble shows a random number from 1 to 100, and it walks on, fading in and out at the bar's ends. Under the likeliness chart, a button under each column and Max with his hammer: he walks to a column with slots in it and hits its button, and the column, dimmed while the game runs, lights from the bottom up to its height and fades. The dev server's new
--sample-interestflag seeds eight answers so the chart (held back until five members have answered) can be seen. -
The cable crew under Project stages (owners' site-text round 1). "A few of the robot[s] in hard hats and with equipment like diggers and jackhammers … putting a cable in the ground that will go to the Servers installed stage where we place an image of a server": the stage slide gains a strip of ground under the dates. The crew works across the first stage not yet done and the next — Strider with a jackhammer, Tread digging the trench, Pip lowering the cable from a reel in its beam — moving up step by step while the trench and the laid cable grow behind them. The cable runs laid through the done stages and as a dashed route to a server rack at "Servers installed" (or the last stage, if renamed), whose lights come on once every stage up to it is done. It scrolls with the stages.
-
The dig scene over the free give-away bar (owners' site-text round 1). "A smaller robot and a bigger robot 'fight' in trying to move the tracker": Tread stands on the green just past the red and digs into the bar, lobbing clods over his back onto a pile on the red; Max hops over him, picks a clod up, hops back and pats it into Tread's hole, and every third trip trips on the way. The scene overlaps the bar so they stand on it;
scenes.jsfinds where the green begins from the meter itself, so no figure is passed to the script. -
The coin desk scene under Slot deposits and Slots sold (owners' site-text round 1). Clerk keeps a desk with a stack of papers; Strider, Pip (its coin held in a new beam of light, the
has-beamprop), Max and Tread queue with a coin each — three for Slots sold — pay at the desk, take a piece of paper (a bigger one for Slots sold), walk back behind the line to its end and find their coins again, in an unending loop. Directed by a new inline script,assets/scenes.js(hashed into the CSP with the others): it only sets classes and the--x/--tcustom properties, moves are compositortransformtransitions, it never starts with animations switched off or underprefers-reduced-motion, and a scene waits while it is off screen or the tab is hidden. Without it the scene is a still picture of its opening moment. -
An animations switch in the signed-in header (owners, 2026-10-14). "An option to turn off the animations in the top bar in case some people find them annoying": a button beside the dark/light switch, on both the member dashboard and the operator console, that stops every animation in the signed-in shell (the robots, the launch summary's walker, the pulsing help marks) exactly as
prefers-reduced-motiondoes, and starts them again. Saved per account with the colour scheme (epic_core::Animations, migration 0083user_appearance_preferences.animations, CHECK-constrained toon/off, under the table's existing user-scoped FORCE-RLS), written into the body'sdata-animationsattribute; no JavaScript, and paused on the pages that show a code or key only once, like the dark/light switch. -
The dashboard's robot cast, and Strider walks the launch summary (owners' site-text round 1, 2026-10-14). Five robots the owners approved on 2026-10-14 — Pip, Tread, Clerk, Strider and Max — join the dashboard as inline SVG (
epic-web::mascots), each drawn as a three-quarter side view and a front view, shaded by gradients that follow the member's colour scheme and rendered once per page. Every move and prop is a class on the robot (is-walking,is-front,has-hat, …), so a new scene is markup and CSS rather than drawing. Strider replaces the stick figure in the Founding launch summary, with the road's markings and the hills set to the pace of his stride. Nothing moves underprefers-reduced-motion, the scene keeps its pause control, and no inline style or new script is needed. The scenes for the other launch boxes follow in their own PRs. -
The owners' accounts come back in every preview slot (2026-10-13, issue #194; ADR-049 addendum).
epic-migrate ensure-accounts --manifest deploy/dev-accounts/preview.tomlmakes a preview slot's database agree with a checked-in list — each account by email (case-insensitive, its email marked verified), an organisation if it has none, its owner membership, its operator flag (the list decides both ways) and one complimentary slot on a named server (or the bootstrap seat for a new organisation) — and changes nothing that already agrees, so it can run after every migration. Passwords are Argon2id v19 PHC strings at no less than the platform's cost, read from FerraVault atepic-preview/dev-accounts/<key>/password-hash, held as secrets and written unchanged; a new one revokes the account's sessions and refresh tokens, as the web password change does. Every hash is checked before a row is written, and the changes and their audit records (dev-accounts-bootstrap, a newPgAuditSink::emit_ininside the caller's transaction) commit together or not at all. Only a preview slot:EPIC_DOMAIN_ENVIRONMENTfrom the environment alone, the list, and the database's name read on the server;devis refused until a dev database carries a marker. An account that is only a member of someone else's organisation, or whose organisation is not active, is refused by name. The complimentary-slot grant moves to a moduleseed-organdensure-accountsshare. Tests against a migrated Postgres as a NOSUPERUSER NOBYPASSRLS role, including an unwritable audit log leaving nothing behind (checked to fail with the audit after the commit). Runbook:docs/operations/dev-accounts.md; running it in every slot waits on an image re-pin (#266). -
Members write their own promo profile for the podium, may leave the affiliation programme, and a code is never given out twice (2026-10-13, Sprint AP.7, the owners' answers on #257 D1/D7 and B9; ADR-077 §8). A Your promo profile card on the Account tab takes a few words about the member, their channel's main page and one page on each of eleven social networks; every link is checked to be
httpson that network's own hosts (in the browser by each field'spattern, again on the server, and by migration 0079's CHECKs), and nothing is saved while one field fails. A podium place by name opens ("About …") on the text and the links, drawn as each network's logo (Simple Icons, CC0), never on an anonymous line; the console can remove a profile but never writes one, and its audit records the slots, not the words. Leave the affiliation programme (closed, with the warning and a required "I understand I will lose all my points") keeps every credit to the member spent and naming no one, retires their codes and removes their profile; agreeing to the terms again starts over. A code that stops naming its account — left, removed by an operator, or the account erased — is kept as a SHA-256 inretired_affiliate_codesand never given out again. The marker role is renamedepic_portal_calls, and the three new tables are revoked from the edge role. Postgres tests for leaving, the promo profile and the console; a browser step opens the podium's About. -
The source gates read a page past its test-module declaration (2026-10-13, Sprint AP.7, found by
ci.sh --full).inline_payloads_are_declared(the CSP gate) andci/routes/verify.pyboth stopped reading a file at its first#[cfg(test)], assuming test code is always a file's tail. An out-of-line#[cfg(test)] mod tests;breaks that: inpages/app/me/affiliate/mod.rsit sat at line 40, hiding the rest of the page from both gates, and thetests.rsit names was read as if it rendered. Both now cut only at an inline#[cfg(test)] mod … {(clippy::items_after_test_modulekeeps that the last item) and skip a file declared out of line under#[cfg(test)]; one new test each, each failing under the old rule. -
Smaller fixes after the stack review (2026-10-12, Sprint AP.7, issue #257 items 34, 36, 37, 39, 40, 41). Closing the Dashboard welcome from the keyboard hands the focus to the current tab (browser test, which fails without the fix); the tab-bar fades are measured again once fonts and images have loaded; the stylesheet no longer carries a global
.transformand.ease-outread from words in the scripts (@source not inline, nothing else changed); every browser-test step that sizes the window puts it back, which showed one step had depended on another's size, now set by itself; a Postgres test drives moves and drags past a paused box through the store's locked transaction; the in-memory paused list breaks ties as Postgres does;page_layoutand the in-memory affiliate store are split under the 500-line guideline. -
Slot prices after the stack review (2026-10-12, Sprint AP.7, issue #257 items 17, 18, 33; ADR-077 §5). The prices include GST (owners), said on the console page, in the migration and in the code (
SlotPriceAud::cents_forgives a purchase's amount in cents). "Already these prices" is decided under the row's lock, so two operators saving the same prices at once record one change; a stored price below zero is reported as damaged rather than as zero; a doc no longer promises a "who" it does not carry. The work to make checkout charge these prices is filed under launch tracker L1, and L2 now says what remains of the price decision. -
The Q&A after the stack review, and "Email this member" (2026-10-12, Sprint AP.7, issue #257 items 12–16, 29–32, 47; ADR-077 §4). Removing a spam question no longer gives the spammer their allowance back: removed and withdrawn questions keep their row for the quota, their words erased (F-251-1). An answer saved but not published shows to its asker as still waiting (F-251-2). Members can withdraw a question or stop showing their name on it, and a closing account is named nowhere (F-251-3). The console opens on waiting questions, fifty to a page, with filters (F-251-4); publish and unpublish are audited under their own names; "answered" became "published" on the box. A test sends two questions at once and expects one through (F-251-5); it fails without the trigger's lock. The published list is no longer callable by the edge role. New: Email this member on each console question — the operator's message goes from the site's sender to the asker's sign-in address with their question quoted, audited by the question's id only.
-
Four broken interactions fixed after the stack review (2026-10-12, Sprint AP.7, issue #257 items 7–10 and 38). The page content is its own stacking context, so a box's controls no longer paint over the account menu and the legal notice and take their clicks (F-246-1). On a phone, where a row's boxes stand one above the other, dragging one moves it instead of flickering in place: the drag reads from the layout whether to compare across or up and down (F-247-1), and finishes a running glide before measuring (no jump on two quick swaps). A drag keeps a paused box where it was, so it comes back in place and the member's order stays default (F-249-1). The header search no longer offers a paused box (F-249-2). Each fix has a browser or unit test, and the two browser steps were checked to fail without their fix.
-
The affiliation programme after the stack review: one code per account, the terms first, members in transit (2026-10-12, Sprint AP.7, issue #257 items 1–6, 19–28, 35, 46; ADR-077 §8). There is one kind of account, with one set of rules: migration 0074 drops AP.6's influencer register and carries nothing over (owners, 2026-10-13: only the operators had accounts, and an influencer is a member like any other), and the console's Influencers page becomes Affiliates (the top 50 by points, find by code or email, remove an abusive personal code or promo profile). A code is made only after the member reads the owners' affiliation terms, ticks "I agree to these terms" and presses "Generate my affiliation code and link"; the version and date are kept and exported. A member may generate one personal code of up to 30 characters after a "you can only generate a personal affiliation code once" question; either code credits the same account, and Billing and the data export show the 8-character code, never a name. Leaving the named board puts a member in transit for 21 days before their anonymous line appears (option B), and so does waiting to come back; a rounded line under the leaderboard counts points, users, members in transit and members under 5 points. Points drop in proportion to a partial refund; equal points go to whoever reached them first. Profile names are unique and "Anonymous" is taken ("That name is already in use"). The portal's SECURITY DEFINER functions are no longer executable by the edge role, read the database's own clock, and the leaderboard is read in one transaction; an account already credited cannot probe which codes exist.
-
The leaderboard switch saves itself; anonymous members stand on the leaderboard (2026-10-11, Sprint AP.7 PR 11, ADR-077 §7). The Account tab's switch is green when public and red when anonymous and saves as it is flipped (no Save button, but for one in a
<noscript>). Turning it off asks "Are you sure you want to become anonymous?" in a modal dialog when going public again would take three weeks; Cancel or Escape leaves it on. The first time a member goes public is immediate. Members not shown by name stand on the Dashboard leaderboard from 5 points as their own "Anonymous" line, never summed, throughaffiliate_anonymous_standings, which returns nothing that names them; the bonus note says it goes to the top three public members. The owners' new rules text. Migration 0078. -
Dashboard polish from the owners' notes of 11 October (2026-10-11, Sprint AP.7 PR 10). Models, "Get started!" and Q&A get the boxes' title format and their own icons (a chip with its pins, a rocket, a question mark). The launch summary loses "The stages below show what comes next." and gains a little walker on a mountain road, as wide as the row of figures: the layers slide by compositor-only
transformanimations, only his 48×58 drawing repaints, it stops under reduced motion and has a pause control (WCAG 2.2.2), and its edges fade so a phone shows the middle. The marketing pool's explanation folds after its first paragraph behind "Read more" / "Read less" (READ_MORE_JS; without script the whole text shows), and its free slot tracker counts down — "400 of 400 FREE give-away slots left", the bar emptying as slots are given, labelled "400 in total" — as does the summary's figure. The interest form reads "Who would your order be for?" and the owners' new privacy line. The share pictures carry the landing page's full logo, and the Profile tab's custom colour shows a painter's palette in the swatches' colours instead of a pipette. The dev server sells nothing and gives two operators 20 complimentary slots each, as production will have them. -
The leaderboard ranks by points, from 5 points (2026-10-11, Sprint AP.7 PR 9, ADR-077 §6). The Dashboard's leaderboard ranks influencers and public members by points — one per slot bought through their code — then sign-ups, and shows both ("12 points · 4 sign-ups"). No one is listed under 5 points, so a figure beside a name never stands for one or two small purchases; a public member below it is told how many points they have. Migration 0077 (
affiliate_public_standingsreturns points and applies the floor); the influencers' standings apply it in their query. Replaces AP.6's "members see sign-ups only". -
Slot prices the operators set: single, duo and multi (2026-10-10, Sprint AP.7 PR 8, ADR-077 §5). A new console section, Slot prices (
/app/admin/slot-prices), holds the price per slot for a purchase of 1, 2, or 3 or more slots — AUD 4,950 / 4,850 / 4,800 as the owners set them — in one form, audited with the prices before and after. Whole dollars, a bigger purchase never dearer per slot (also a CHECK). The Dashboard's marketing-pool example ("23 slots, worth AUD 110,400") now follows the multi price, and leaves the amount out if the prices cannot be read. Migration 0076 (slot_prices, one row, operator-global). Not yet what checkout charges: no payment provider is connected. -
A Q&A box: members ask, the team answers and publishes (2026-10-09, Sprint AP.7 PR 7, ADR-077 §4). The Dashboard's new last box shows the team's published answers and a form to ask — anonymously or under one's profile name — with a confirmation, the member's own questions and where they stand, and the owners' quota message past one question in 24 hours or three in 7 days (counted by a database trigger). Operators answer, publish, unpublish and remove at
/app/admin/questions(audited by id, never the words). Migration 0075 (FORCE-RLS, hidden from the edge); the GDPR export carries a member's questions. -
Every member has an affiliation code, with an opt-in leaderboard (2026-10-09, Sprint AP.7 PR 6, ADR-077 §3). A new Affiliation code box on the Account tab: the member's own code (made the first time it is shown) and link with copy buttons, the device's share sheet and plain share links (X, Facebook, LinkedIn, WhatsApp, Telegram, Reddit, email), three 1080×1080 pictures with the code on them drawn in the browser and downloadable as PNG, their own figures (accounts linked, points, free slots earned), and a switch to be shown on the leaderboard — anonymous by default, public at once or three weeks after the first account links. Member codes credit through the same path as influencers' (set once, never one's own, one namespace), and close once every slot is sold. The Dashboard's pool box carries the owners' explanation, a free-slot bar, a three-bar tracker (linked → deposit → bought) and a leaderboard of influencers and members who opted in. Migration 0074; the code table is FORCE-RLS and hidden from the edge; GDPR export carries the code.
-
Operators can pause any box for every member; Servers starts paused (2026-10-09, Sprint AP.7 PR 5, ADR-077 §2). A new console section, Paused boxes (
/app/admin/paused-boxes), lists every box of every member tab with Pause… (through the confirm page) or Restore; an operator also sees a pause control on each box. A paused box is hidden for everyone, members keep their own place for it, and moving a box steps over a paused one so what the member sees and what is saved agree. Migration 0073 (paused_boxes, operator-global), seeded with the Dashboard's Servers box. Audited. The in-box text editor the owners floated is not built (it would fork the copy from the code); a draft issue explains why. -
The Dashboard in the owners' words, and a team may put up to 500 slots on the interest tracker (2026-10-09, Sprint AP.7 PR 4, ADR-077 §1). The summary box ("Founding launch summary — a quick overview of the numbers") and every tracker carry the owners' copy and titles in the four formats; the summary's five figures — interest, deposits, sold, give-away slots, stages — are links that jump to their tracker. The interest form offers "A team" beside "Myself", up to 500 slots for a team (migration 0072 widens the CHECK and keeps an individual and a carried-over entry at 100, in the database too;
InterestWrite::newchecks the same). "Slot deposits received" and "Slots sold" carry the owners' notes, the latter's goal read from the servers. The Usage box gets a speedometer, the owners' copy, accent stat titles, and a fair-share standing with a "?" that opens the full explanation on hover or click (browser-tested: hover, leave, pin, close, Enter, Escape). DESIGN.md's last "4,000 per server" corrected; the dev server and the pixel-gate fixture seed 9,000 slots per server. -
A dragged box follows the pointer, and drags down as well as up (2026-10-09, Sprint AP.7 PR 3). The box lifts (shadow, above the others) and stays under the pointer at the spot it was picked up by; the other boxes glide out of its way and it settles into its slot on release. A box now swaps once its own leading edge passes the middle of its neighbour — measured from the pointer, as before, dragging a box down meant dragging past most of the box below it, so it seemed only to move up. Layout positions come from offsets, so a block still gliding cannot undo its own swap. Browser-tested (drag down, follow mid-drag, saved, settled); the test fails against the old script.
-
A quieter portal header: the tab bar rises level with the logo and icons, and the Dashboard welcome folds away (2026-10-09, Sprint AP.7 PR 2). On a tablet or wider the member tab bar is pulled up until its raised tab lines up with the wordmark and the header icons; tabs scrolled under the icons fade behind them and cannot be clicked there by accident, and the gaps between tabs now scroll the bar instead of the page. The EPIC logo goes home (the member's opening tab), or to the console for an operator. The welcome reads "Welcome, " (or just "Welcome"), centred, folds away after 20 seconds once per browser tab, and has a close button; the "signed in as a member of …" line and the repeated page titles under the tab bar are gone (still announced to screen readers). Long words wrap rather than widening a phone's page; the interest form's likeliness scale wraps to two rows on a phone (the Dashboard was 37px too wide at 390px).
-
The portal's house style: one box design and four text formats (2026-10-09, Sprint AP.7 PR 1). Every box on every member tab is drawn by its wrapper with the launch tracker's shaded surface and an accent edge along the top (a little thicker on the launch box), so every title and control sits inside its box — the API-key box's contents no longer hang below it — and boxes side by side share one height. Four text formats (
.fmt-eyebrow,.fmt-heading,.fmt-lede,.fmt-box-title) for the owners' copy. Portal only; the public site is untouched. -
The visual-regression gate: the
maindrift explained, the engine pinned, and provenance recorded with every baseline set (2026-10-09, issue #175). Renderingphase-9-finalandf28a11cunder one engine reproduces #175's table to three decimals, so the drift was intended product change. The public pages are 0.00000 fromf28a11cto today. The gate had never run its pinned Firefox: geckodriver launched the auto-refreshing snap instead. The binary is now passed explicitly and the tarball's updater is disabled.PROVENANCE.json(browser, binary, fonts, viewport, commit) makes the next drift say whether the environment changed. Non-200 pages are refused and retired pages' baselines removed. Re-recorded atd059c85; the workstation's/dataset needs oneci/visual/run.sh --record. -
The AP.5/AP.6 stack review, answered (2026-10-08, Sprint AP.6 §7, PR 7/7). Security: an email change is confirmed only from the signed-in account that asked, and a taken address is not revealed; the edge database role can no longer read members' password hashes, TOTP secrets or contact details; the interest list's clear is
epic_admin-only and audited before it runs; hidden formatting characters and private-use country codes are refused. Correctness: menu, layout and colour-scheme saves are applied atomically and sent one at a time (browser-tested); account details save in one compare-and-set write; the cache is cleared after writes too; referral standings count only purchases made after the referral. Owners' decisions: members see influencer sign-ups only, and the interest breakdown from five entries. Migrations 0069–0071. Issues #239 (TOTP replay, pre-existing) and #240 filed. -
The Profile tab: the avatar always in sight, and a mini tab bar for menu preferences (2026-10-08, Sprint AP.6). The Profile card's view mode is the account menu's header enlarged — avatar, profile name, email — and "Edit profile" opens the same table as before. Menu preferences gains a small copy of the top bar: drag a tab, or select one and use the arrow keys or the buttons, and "Open on this tab" moves a house icon to the tab signing in opens on; every change saves at once, and the plain form remains for a browser without a script.
-
Every box on every member tab collapses and moves, saved per account (2026-10-08, Sprint AP.6, ADR-076 §5). Dashboard, Group, Account and Profile: each box has a collapse arrow (a real form, so it works without a script; instant with one) and a grip — drag it, or focus it and use the arrow keys, or press it for arrow buttons. Side-by-side boxes swap left and right and move up and down as one row. Stored per page (migration 0068, FORCE-RLS); the server applies each change to the stored arrangement. Links to a collapsed box open it. Browser gates green: keyboard, contrast on every scheme, overflow.
-
Influencers, referral credit, and the promotional launch tracker (2026-10-08, Sprint AP.6, ADR-076 §4). Operators register influencers at
/app/admin/influencers(name, referral code, optional https links, required consent; edit, pause, remove — audited). Members are credited once, never to themselves, by the influencer's link (/r/{code}→ sign-up, a hidden field, no cookie) or by entering the code on Billing (migration 0067; FORCE-RLS on who referred whom; standings are counts of verified sign-ups and paid slots). The Dashboard's launch tracker now has the owners' five trackers in their order — interest, Admin/marketing (complimentary slots given against the budget; top three with avatars and links, next five, the rest collapsed), deposits (not open yet, at zero), slots sold, stages — under a hero banner with live figures, an icon per tracker and glowing meters. The GDPR export carries the referral. -
A Paid slots interest tracker replaces the interest list (2026-10-08, Sprint AP.6, ADR-076 §2–§3). One honest entry per account on the Dashboard (migration 0066, FORCE-RLS on
app.user_id): who it is for, 1–100 slots, how likely the member is to order (1–10); the form shows the account's email (counted once, used for nothing else, deleted when the server is full). A meter runs to one server's 9 000 slots and, once passed, slides the 9 000 marker left with a second marker for the total; a red-to-green column chart shows slots per likeliness, with a screen-reader table. The figures cross accounts only as counts. The operator console clears the list (confirm + reason, audited). Old interest-list entries carry over marked not stated; the old table stays, unread. Billing and the sold-out checkout link to the tracker; the GDPR export carries the entry. -
The launch tracker's project stages are the owners' to edit, and each server sells 9 000 paid and 400 admin/marketing slots (2026-10-08, Sprint AP.6, ADR-076 §1). The stages move from code to
launch_stages(migration 0065), seeded with the owners' eleven (the β platform done, the rest coming up, no dates) and edited at/app/admin/launch— title, status, expected date, order, add and remove, every write audited. The Dashboard shows them as one long slide, dulled red / orange / green with the status also in words, above a date track. Concurrent edits keep positions gapless under an advisory lock (the test fails without it). New servers default to 9 000 paid and 400 complimentary slots. -
The member dashboard is five tabs — Dashboard, Playground, Group, Account, Profile — as the owners laid it out (2026-10-08, Sprint AP.5, ADR-075). Wiki Site Layout & Tabs, table 2. Dashboard: a launch tracker (slots sold, live, as a count and a bar; project stages that follow the servers' status; a labelled placeholder for the influencer leaderboard, which has no data yet), usage, servers, models, and a way to an API key. Group: group details, members and the fleet on one tab. Account: "Account details" (the old Profile card renamed — full name, email, and new optional phone, postal address, company and ABN, the ABN checksum-validated; the user public ID is no longer shown), Security with a button to the audit log, API keys, Billing, Privacy. Profile (new,
/app/profile): profile name and avatar, colour schemes, menu preferences (the member's own tab order, by move buttons, and the page signing in opens on — migration 0062, FORCE-RLS), email notifications. A header search for pages and settings (an icon that opens a box without any script; suggestions as you type; a server-rendered results page), every "X" button reads "Close", and on phones the account menu lists the sections. Former pages (/app/usage,/app/models,/app/fleet,/app/keys,/app/members,/app/billing,/app/security) 303-redirect to their sections,/app/securityclosing F24's error path. Saving one of the two cards that used to be one can no longer clear the other's fields, and a tab whose one section cannot load shows that section as unavailable rather than failing (usage included — a failed read no longer shows as zero activity). The Account tab's billing notices come from a fixed list of keys rather than as text in the address, so a crafted link cannot put words on the page. Browser gates (overflow, contrast on every scheme, keyboard) green; pixel baselines move by design and are re-recorded with #175. -
FerraVault drift is checked continuously, under an identity that can change nothing (#193, ADR-072 §4 Amendment, 2026-10-06). FerraVault v1.15.0 shipped ferravault#8: policies and mTLS bindings are readable per object with
read(no longersudoon**), carry a BLAKE3 content digest, and binding changes reachVault.Watch. New binaryepic-vault-audit(epic-system,deploy/k8s/epic-vault-audit/) holdsread+watchonsys/policy/**andsys/mtls/**only, and compares FerraVault's own digests of every stored policy and binding with the repository's policy files and identity registry. It re-checks within 2 s of every policy or binding change, whenever its Watch reopens, and every 5 minutes. It exportsepic_vault_audit_{policy_drift{policy},binding_drift{identity},unexpected_policies,unregistered_bindings,last_success_timestamp_seconds,checks_total{outcome},watch_connected}, with four alerts: drift pages after 30 m, an unregistered binding pages at once, and a check that cannot run never reads as clean. The expected state ships as generated ConfigMaps (ci/ferravault/render_expected.py); Stage 1 fails when they are stale.verify.pygains consumerauditor(its watch list read from the binary'sWATCH_PATTERNS) and refuses any other identity whosereadrules reachsys/policy/sys/mtls, which turns v1.15.0's upgrade hazard into a permanent gate.ferravault-verify-live.shnow runs the same binary'scheckin the auditor's pod;ci/ferravault/live.pyis deleted. FerraVault crates 1.14.0 → 1.15.0 (all sixteen in lockstep; no EPIC source change needed; client HTTP/2 + TCP keepalive on by default — ferravault#5, epic#183). Tests: 29 unit;deployed_policy(the deployed policy against the binary's requests, with FerraVault's evaluator);integration_live_audit5/5 against an in-process FerraVault 1.15.0 over mTLS — every drift kind, refused writes, and Watch-driven re-check (mutation-checked: disabling the event re-check fails it); 4 new gate mutation tests;promtool test ruleson all five alerts. Review fixes (#195): the auditor now picks up its renewed client certificate. It polls the mounted pair by content (ADR-071's reasoning), re-dials, and reopens its Watch under the new certificate. A check or Watch reopen that fails at FerraVault re-dials, so a process that dialled once can no longer go silently blind when its 90-day certificate expires. Newepic_vault_audit_{identity_renewals,redials}_total. A live test starts the auditor on a 6-second certificate, swaps in the renewal and expires the first, with zero failed checks across the expiry (mutation-checked twice). Probes now race shutdown;PatternSetmoved towatch/pattern_set.rs. Corrected: theActivestate and its metric help (granted at open, not proof of delivery), theWatcherHandledrop semantics, and ADR-072's readmission claim. Found while fixing this and filed: #216 (the edge, control and secret-sync never pick up a renewed FerraVault certificate —rotate_identityhas no production caller) and #217 (readmission does not replay;OUT_OF_HISTORYretries forever). Review round 2 (#195): (1)WatcherHandle::shutdown()could hang. Every shutdown arm usedwatch::Receiver::changed(), which marks the change seen, so when a re-probe's arm consumed it,consume's own arm never fired again whileshutdown()held the sender awaiting the join. The loop kept streaming and re-probing, and the edge's exit waited for SIGKILL. All six arms (five in the session, one in the synthetic loop) now use oneshutdown_requestedhelper overwait_for, which tests the flag. A new live test stalls a TCP relay in front of an in-process FerraVault until a re-probe is provably in flight, then requiresshutdown()to return; withchanged()restored it hangs 3/3. (2)ferravault-unseal.md§9 was wrong about rollback. A pre-1.15 daemon on a freshaudit_orphan.dbre-runs the one-time baseline and grandfathers every record past the chain head with no tamper signal (v1.14.0reconcile_at_boot). The runbook now recordsferra audit verify --output jsonon v1.15 first, stops on any unaccounted record, says what the baseline log line means, and drops theferra audit acknowledgestep v1.14 cannot run. (3) The auditor swaps in a renewal only once FerraVault accepts it: under TLS 1.3 a rejected certificate still dials, so the new client makes one authenticatedPolicyListfirst (Healthbypasses authentication and would pass an unbound certificate). It re-dials only on transport failures, not on vault decisions such as a missing grant. Newepic_vault_audit_{watch_opens,identity_renewals_refused}_totaland anEpicVaultAuditRenewalRefusedticket (promtool-tested). New live tests: an untrusted-issuer renewal and an unbound renewal are refused while checks continue, then a good renewal is picked up; a missing grant raises errors but no re-dials; the renewal test now requires the Watch to reopen before the old certificate expires. Each was mutation-checked, including the deletion ofclose_watchthat Melvyn showed the previous test missed. -
The avatar upload and Change Password, hardened through two review rounds (2026-10-06, PR #164 round 2 B1–B5 and PR #197 items 1–7, ADR-074). Avatar: decode limits checked from the header and against the decoder's own size before allocation (8192 px per axis, 80 MiB — a 24 MP phone photo fits, a decompression bomb does not); decoding on the blocking pool behind a two-permit semaphore whose owned permit stays held when a request is cancelled mid-decode, so resetting uploads cannot run unbounded pipelines; a 15 MiB body limit with an HTML 413 page; EXIF orientation applied, every other EXIF field (GPS included) dropped; 8-bit re-encode and a 400, not a 500, for anything still over the 256 KiB the column allows;
Cache-Control: private, no-cachewith anETag; its own §9.7 bands, per IP and per account; at most four uploads received at once per replica and one per account, a slot taken before the body is read and released only when the decode ends, so the bodies in memory are bounded as well as the decodes; a "busy" answer is not charged against the member's allowance, and removing an avatar is its own plain route (rounds 3–4). Change Password: a per-IP and a per-account rate limit ahead of Argon2; an MFA step-up (current code or a recovery code, checked after the password so a wrong password never burns a recovery code); every failed step audited asuser.password_change_failedwith its reason; a failure after the rotation says the password changed rather than that it did not; a success banner. Personal data: avatar and name changes audited (that they changed, never the values);/api/me/exportand/app/me/exportcarry the structured name and the avatar image. -
A horizontal tab bar replaces the dashboard sidebar, on both the member dashboard and the operator console (2026-10-06). The selected tab is centred, drawn in the member's palette colour at 1.7× and rises into the header; the bar slides, swipes on a phone, stops at the first and last tab centred with empty space beyond, and shows an arrow at an end that still has tabs out of view. "Home" becomes "Dashboard" and is where login lands; the "Dashboard" word beside the logo is gone, and the logo links to the console for operators only. Arrow keys move along the bar, with a visible focus ring on every tab. The browser gates caught a mid-transition contrast reading and an unnamed second
<nav>landmark; review round 1 caught a drag that could leave the bar following a released mouse; all fixed. See Sprint AP.4. -
The Account page's layout follows the member's own card order, "MFA" replaces "Two-factor" on the Account page, and a self-service "Change Password" section joins the Security card (2026-10-06, Sprint AP.3). The MFA setup and login pages still say "two-factor"; aligning them is #208. Card order is now Profile → Colour schemes → Security → Privacy → Email notifications (operator-specified). The Profile card's status row and both Security-card mentions read "MFA" / "Multi-Factor Authentication" rather than "Two-factor" / "two-factor authentication" — same capitalisation throughout. The Security card gains a second no-JavaScript edit-mode section, styled like the Profile card's own toggle (a visually hidden checkbox +
<label for>,:checked ~CSS rules, a native<button type="reset">for "X"): current password, new password, confirm, posting to a newPOST /app/me/password(epic-control::routes::web::password). That handler verifies the current password against the stored hash, hashes and persists the new one, then — because landing back on/app/mestill signed in was the explicit ask, not a bounce to/login— revokes every session and refresh family (the same safety guaranteeroutes::auth::password_reset_confirmgives a stolen-password rotation) and immediately mints a fresh session for the browser that just changed it, so "change password" costs every other device its access while leaving the member where they were. Best-effortSecurityAlertemail; a hard audit emit (user.password_changed) for the same reasonpassword_reset_confirmtreats its own audit row as load-bearing.POST /app/security/totp/setup's successful-verify redirect changed from a separate/app/security/totp/enabledlanding page to/app/mefor the same "return to where you started, still signed in" reason — that page stays live and directly reachable, just no longer the default landing. -
Members can upload an avatar, and the standalone "Log out" button becomes a UniFi-style account menu (2026-10-06, migration 0061, ADR-074). The avatar lives in Postgres as
BYTEA(users.avatar_image+avatar_content_type, three CHECK constraints: presence-iff-content-type, content-type restricted toimage/png, body capped at 256 KiB) rather than a new object-storage dependency.POST /app/me/avatar(epic-control::routes::web::avatar) decodes the upload viaimage::ImageReader::with_guessed_format— decode failure is the format whitelist, closing the SVG-upload-as-XSS vector a naive extension check would leave open — resizes to fit within 256×256 (never upscaled), and always re-encodes to PNG regardless of source format, so what's stored is one well-understood shape rather than a pass-through of client-supplied bytes;GET /app/me/avatarserves it back. The control plane's global 64 KiB request-body cap gained a path-specific exception (AVATAR_MAX_BODY_BYTES, 15 MiB — see the hardening entry above) for this one route, on both the customcontent_length_guardmiddleware and a per-routeDefaultBodyLimitlayer. The Profile card gains an "Avatar" row, visible only in edit mode as a whole row (not just an editable cell, unlike every sibling row) with a preview, a file picker (which opens a cropper: place the photo in the square, zoom, tilt, turn), and — once an avatar exists — a "Remove avatar" button that asks first ("Are you sure you want to delete this avatar? …", with "No, keep avatar" / "Yes, remove avatar", a native popover, no JavaScript); the picker's controls use the HTMLform=attribute to submit to a separate, otherwise-empty sibling<form>, since amultipart/form-dataupload cannot nest inside the surroundingprofile-form. The header's "Log out" button is now a small avatar/silhouette trigger opening a dropdown (no JavaScript — the same checkbox-hack as the colour picker, plus a full-viewport backdrop<label>for click-outside-to-close): the member's name (falling back to "EPIC User") and email, Support, Sign out, and Terms & Conditions / Privacy Policy (each opening a temporary "In progress, come back later, email us for any questions." notice — no placeholder legal pages are published, PR #164 review D5) — modelled on a supplied UniFi reference screenshot, deliberately omitting its "Site Manager" version-number row. The dropdown's name and email come from a newepic_web::identitymodule — atokio::task_local!slot mirroringepic_web::appearance's shape, set from the same two chokepoints (routes::web::app::authenticate,admin::require_operator_session) — rather than a parameter threaded through every page function. -
The colour-scheme work's review rounds 2–3: the muted blue as the default, a contrast margin, and a dark/light switch that cannot break MFA setup (2026-10-06, Sprint AP.2). The muted dark+blue is the dashboard default by decision (ADR-073 §6);
:rootstays the public site's own. Every preset surface and every derived custom surface clears 4.6 for white text, not merely the 4.5 floor. The header switch reloads the page it was clicked on, and the MFA setup page mints a fresh secret on every GET, so switching mid-enrolment silently invalidated a scanned QR code and the written-down recovery codes; on that page, a new API key's and a new access code's, the icon now opens a short note instead, and the server refuses the setup page as a return target. The browser contrast gate now measures all nine presets — a hand-typed list had left orange, pink and slate unmeasured. Migration 0059's comments no longer claim the defaults equal:root, fixed before it is ever applied. -
A missing-chokepoint bug — seven pages rendering the dashboard's default chrome and colours regardless of the member's choice, one of them with no auth check on POST at all — fixed across every instance an audit found (2026-10-06).
routes/web/{totp_setup,keys,abuse_opt_out,diagnostic_capture}.rsandme.rs'sdelete_post/profile_post/notifications_post/appearance_postwere callingrequire_authenticated_user_pubdirectly instead of the dashboard'sauthenticate()chokepoint — the same defect class ADR-073 fixed for a different six pages the day before. The worst instance wasme.rs'sdelete_post: an unauthenticatedPOST /app/me/deletereached the full account-deletion confirmation logic before any session check ran at all.totp_setup.rsgained aGET /app/security/totp/enabledroute of its own (previously unreachable from a stateless router with no way to callauthenticate()); the appearance-reachability gate test now reads the sameall_dashboard_routes()list every router is built from, so a page missing from that list is itself a finding, not just a missing assertion. -
The Account page's two identity cards merge into one editable "Profile" card, plus a structured full name (2026-10-05, migration 0060). The read-only identity summary and the separate one-field "Profile" form (display name only) were two cards covering overlapping ground; they are now one, with a no-JavaScript edit-mode toggle — a visually-hidden checkbox (
.scheme-custom-input's own trick:opacity: 0, neverdisplay: none, so it stays focusable via its<label for>) that CSS:checked ~rules key every row's view/edit cell and the button row off. "Edit profile" is that checkbox's label; "Save profile" is a plain submit; "X" is a native<button type="reset">, which reverts every control in the form — including the checkbox — in one native behaviour, so discarding changes and leaving edit mode happen together rather than needing two mechanisms. Newfirst_name/middle_name/last_namecolumns (users, migration 0060, each nullable and independently capped at 128 chars) back a new "Full name" row — one joined line in view mode, three inputs in edit mode — kept distinct from "Profile name" (the renameddisplay_namerow, unchanged as a column). Email, Public id, and Two-factor keep no edit affordance — email needs its own re-verify flow, Public id is immutable, and two-factor setup lives on its own page. The combined save goes through one newStore::update_user_profilecall (UserProfileUpdate) rather than throughUserInsert, which stays untouched —UserInserthas roughly fifty construction sites across the workspace for fields every signup carries, and none of these three are collected at signup. Both the "Appearance saved." and "Profile updated." banners are gone: the colour picker's own selection ring and the edit toggle collapsing back to "Edit profile" already show the save took, so a banner on top of either was noise. Aphone_numbercolumn and row shipped and were removed from this same migration and card before either merged (operator request) — safe to edit the migration in place rather than carry a column nothing writes, since the branch had not merged. -
A free-form accent colour, three more curated presets, and the auth chokepoint that made the first colour-scheme feature only reach six of its thirteen pages (2026-10-05, ADR-073).
/app/models,/app/fleet,/app/playground,/app/usage,/votes/*,/app/me/exportand/app/me/deletewere callingrequire_authenticated_user_pubdirectly instead of the dashboard'sauthenticate()chokepoint, so every one of them rendered the default scheme regardless of what a member had chosen — invisible to the Sprint AP.1 gate, which drove a hand-typed six-path array that happened to be exactly the six pathsapp.rsitself registers. The gate now readsall_dashboard_routes(), the sameDashboardRoutedata every affected router is built from, so there is one list rather than a router and a second, independently-maintained idea of what it serves. Three more presets (orange, pink, slate) join the original six, same measurement discipline, same ≥ 5.5 / ≥ 4.5 floors. A genuine free-form accent sits alongside the curated set: a native<input type="color">(the browser's own wheel and eyedropper, no copied assets) submits a hex, andderive_custom_accentadjusts only its lightness — hue and saturation held fixed — to the value closest to the AA boundary, proven for every hue and saturation by a property test rather than measured for a fixed sample. (The property test only proves the contrast floor; it didn't catch the first version collapsing every dark-base input to flat white, because white clears that floor trivially — caught by looking at the rendered page, and now pinned by its own regression test.) The one dynamic<style>block this needs gets a per-response SHA-256 computed from the bytes actually served and appended to that response's own CSPstyle-src— the policy stays hash-based, not nonce-based, for every payload including this one; proven on a real response, not just on source text, by a new wire-level test that seeds a custom accent and checks the rendered body's inline block against its own header. A header quick-toggle flips only the base (dark/light), no JavaScript, and validates its hiddenreturn_toas a lookup key into the server's own route list rather than ever redirecting to request-supplied bytes. See ADR-073. -
Member-selectable colour schemes, behind the login only (2026-09-19, Sprint AP.1). Two composing axes — a base and an accent — give twelve schemes from eighteen declarations, chosen on
/app/meand stored per user (migration 0059, FORCE-RLS onapp.user_id, mirroring the notification preferences). Nothing before the login changes: the pre-login shell is a different element with its own token set, and a test pins that at the HTML rather than trusting the CSS scoping. Every value was measured against a contrast calculator validated on the four ratiosinput.cssalready documented; foregrounds carry >= 5.5 against an AA floor of 4.5, weakest of eighteen measurements 4.70. Light mode was the larger half: six colours written as light-on-dark literals measure 1.79 / 1.55 / 2.58 against white — unreadable, not merely off-brand — and are now tokens with per-base values..btn.dangerand.preview-bannerkeep their literals deliberately, carrying their own dark grounds. The scheme is carried as a per-request task-local rather than threaded through 87 call sites, which is a five-file diff instead of ~150 edits; the trade is a failure the compiler cannot see, so it is backed by a mutation-tested gate that drives the authenticated routes through the production stack and asserts a non-default scheme — seeding the default would pass against a plane that had lost the feature entirely.site.cssgrows 3.1 % and is inlined into every response, so pre-login pages carry bytes they do not use. See ADR-070 and Sprint AP.1. -
Slot purchasing & access levels — Sprint SP.3: sign-off (2026-09-18). The phase's evidence sprint. 34 §2 items across SP.1, SP.2 and SP.4, all ✅ Delivered, zero four-field deferrals. Three audits ✅ PASS: quality (workspace line 88.33 % — above the 80 % floor, above Phase 9's 86.51 %, and the highest recorded here; 2 889 tests / 137 binaries / 0 failed), security (this phase's own attack surface — webhook intake, money-adjacent authorisation, the rights gates' refusal carrying no oracle, tenant isolation under a
NOSUPERUSER NOBYPASSRLSrole, access codes hash-only at rest), architecture 77/77. The eleven §4 exit criteria were run as gates rather than inferred — 110 tests across 11 suites, named per criterion, with both concurrency properties proven against real Postgres rather than asserted from trigger text. Stage 8 returned0 regressed, 13 stableagainstlocal-3826c6c, confirming SP.4's bench re-freeze rather than re-deciding it.The gate found five real defects and all five were fixed in-sprint, and three of them share one shape: a check that had been returning the right answer for the wrong reason. The ADR-026 pipeline-invariant command in
CLAUDE.md§3.7 namespipeline.rs, andrgon a file path does not recurse into the siblingpipeline/directory — where Sprint 2.4 moved the very import it polices. A dual-stack listener test asserted "both sockets are bound" for a socket that had never bound. Andepic-meter's coverage figure had been carried by two prior phase audits as a live-substrate measurement gap when the substrate suites were running all along — the uncovered lines were production wiring nothing called, not paths nothing could reach. The lesson generalises past the crate: a coverage figure carried forward with an explanation attached is not the same as one re-diagnosed. -
Slot purchasing & access levels — Sprint SP.4: complimentary slots, slot holds, and one login per account (2026-09-16). Every slot is now a row of its own, with a kind, its four rights, a status and an end, and each right is decided per slot rather than per account. Each server has two pools: the paid places it sells, and a resizable complimentary budget operators grant from. Servers are created hidden and listed when needed. Operators grant, hold, lift and terminate individual slots from the account's page, suspend an account (which holds every active slot, and lifting releases only those holds) or close it for good. A terminated paid slot keeps its place off sale, and that place can be handed on once to another account — as a new slot with its own id and its own signed certificate, the previous holder's marked no longer valid from the day it stopped. A grant or hand-on issues a certificate; going online re-issues them dated. The Plan page lists the account's slots by server with their certificates; an operator's reasons and notes stay off it. A five-minute sweep ends temporary grants at their day and every slot at its server's term end, and expiry warnings are counted from slot rows, so a place's new holder is warned and the terminated one is not.
epic-migrate seed-org --complimentary-on <server>gives the bootstrap account a real slot instead of a bare seat. A login belongs to exactly one account: an invitation creates a new login inside the inviting account, through the whole registration pipeline, at a/invitationpage that replaces an emailed link no route had ever served. Migrations 0057 and 0058 — 0058 refuses to run while any login still has two accounts, and names them. ADR-069 Accepted; ADR-067 §1 and §3 superseded in part. Detail:sprint-SP.4. -
Slot purchasing & access levels — Sprint SP.2: one-off slot purchases on per-server pools (2026-09-15). Billing sells one-off slots instead of subscriptions. Servers fill in order. The investment step reserves slots for 30 minutes on the server they go to, is limited to the slots left, and offers a free interest list when every server is full.
PurchaseEventreplacesSubscriptionEventin the trait and in the Paddle, Lemon Squeezy and Stripe adapters, and the customer portal is gone. The database maintainsorgs.seat_countfrom paid purchases: a refund puts slots back on sale and clears the newest seats./app/billinglists purchases with their signed certificates. Operators run servers from pre-order to online at/app/admin/servers; going online dates every slot on the server and re-issues its certificates. Expiry warnings go out per organisation per server. Migrations 0054–0056; the organisation's plan dates are dropped. No merchant is live yet (tracker L1). Detail:sprint-SP.2. -
Slot purchasing & access levels — Sprint SP.1: access levels and playground access codes (2026-09-14). Only slot users reach the playground, API keys and voting. The edge refuses a non-ephemeral key whose organisation holds no slot (
403 slot_requiredon native, OpenAI and Anthropic), so keys issued before the rule stop working. Operators issue one-time playground access codes from/app/admin/access-codes(shown once, emailed best-effort, revocable). Members redeem them on/app/playground. Detail:sprint-SP.1. -
Slot purchasing & access levels — planned (2026-09-14). ADR-067 Accepted: only slot users reach the playground, API keys and voting; operators issue one-time playground access codes; billing moves from subscriptions to one-off slot purchases on servers that fill in order, with each server's go-live starting one shared ~5-year term. Plan:
sprints/slot-purchasing/(SP.1 access levels and codes, SP.2 the billing reshape, SP.3 sign-off). -
A member can ask for a new verification email (2026-09-14). Registration sends one verification mail, best-effort, and until now there was no way to get another: the signup check-email page pointed at a dashboard control that did not exist, and the verify-email error page at an account page that had none. The login page now offers "Didn't get your verification email?", leading to
/verify-email/resend(andPOST /api/auth/verify-email/resend). Every submission gets the same answer and the same page, and the answer is returned before any of the work starts — which runs on a detached task — so neither the response nor the time it takes says whether an address has an account (password-reset/requestgives the same answer but does its work first, so its timing still differs). A new 24-hour link goes out only to an existing, unverified, undeleted account, using the same mail model registration uses, and is audited asuser.verify_email_resent. Two bands: 5 / hour per IP in the middleware (AuthVerifyEmailResend), and 3 / hour per account after lookup (AuthVerifyEmailResendPerAccount, silent), because anyone can register with someone else's address and a per-IP band alone would let a caller spread across networks flood that inbox. The signup check-email page's last tip and the verify-email failure message now point at this flow. Mutation-tested: making the work synchronous, dropping the already-verified check, or dropping the per-account band each fails a test — the first via a mail server that takes 10 s to answer. -
A rate-limited form post gets the page back, not the API's error envelope (2026-09-13). The waitlist band is 5 submissions / hour / IP by design (DESIGN.md §9.7) and the limiter is middleware shared with the JSON surface, so hitting the ceiling from the landing page rendered
{"error_code":"rate_limited",…}as the whole document — the API's answer shown to a browser. The deny branch now bounces HTML form paths to their own Post/Redirect/Get target (/?joined=busy#join, a newNotice::TooManyreading "Too many submissions from this network. Please try again later."), and every/api/path keeps the envelope unchanged. The target is a fixed per-path constant rather than aRefereror a form field, because a rate-limit denial is precisely the response an attacker can summon on demand and a bounce to request-supplied input would be an open redirect with extra steps. Two tests: the map, and a cross-check that every path the limiter guards is either an API path or has a target — so a new public form cannot quietly inherit the JSON answer. -
The script's URL carries a fingerprint, and the popup centres itself (2026-09-13).
asset.rsjustified a year-longimmutablecache on the grounds that "the URL names the content" — true of the fonts and images, whose filenames change when their bytes do, and false ofbrand.js, whose name never changed. So every deploy that altered the script left returning visitors holding the old one, and the damage was not merely "no new behaviour": a cached script paired with a fresh stylesheet is a pair that never existed anywhere, and this one put the join popup as a static block at the end of the document — off-screen on a phone, a scroll away on a desktop. The layout now emits/assets/js/brand.<fingerprint>.js(eight hex of FNV-1a over the bytes, deterministic so identical bytes give identical URLs); the route serves the fingerprinted and the bare form, and the CSP guard that requires every<script src>to be a readable literal was taught this one expression and mutation-tested to confirm it still rejects an external origin. The popup is also positioned by the message itself rather than only by its scrim, so the two halves of the feature can no longer disagree about where it goes. Verified at three scroll depths and two widths, and in the scrim-less shape an older script produces: centred to 0px, fully on screen, every time. The earlier verification of this had checked onlyscrollY = 0, where a box anchored to the document's top is indistinguishable from one fixed to the viewport. -
An address typed once is not typed again, and the signup page offers the way out first (2026-09-13).
/signup?email=already prefilled;/login?email=now does too, and the signup page shows an "Already have an account? Log in" route above the form whenever an address arrived with the request, carrying it across. It says if rather than reporting a fact, and that is the design: checking whether the address is registered and saying so would answer "does this email have an EPIC account" to anyone who crafts the URL — an enumeration oracle cheaper than the onePOST /signupalready is, needing no Turnstile, no password and no policy acceptance. Offering the route costs a returning member one click and tells a stranger nothing. The already-registered error card keeps its own login button, and the pre-form route is suppressed when it is showing, so there is never one above the other.prefill_emailmoves toroutes::webfor both handlers to share, and the percent-encoder that was private to the operator console becomesepic_web::urlrather than getting a second copy. -
Four fixes from reading the preview on a real phone (2026-09-13). (1) The first screen was taller than the screen.
--brand-nav-heightclaimed 4rem while the bar rendered 77px —min-heightis a floor and the floor stopped binding when a 2.75rem panel button arrived inside 2rem of block padding — so the hero reserved 13px more than it was given and its second button sat under the fold. The bar's block padding is 0.5rem now, which makes the floor bind and the token true. The hero also subtracts the measured height of the whole pinned stack (--brand-topbar-height, published by brand.js, falling back to the token), so the preview banner is accounted for: the banner used to be deliberately unsubtracted on the reasoning that it should cost space, which made the preview an unfaithful picture of the page it exists to preview. And the hero's vertical budget is trimmed belowmd, where its content is genuinely taller than a short phone screen — measured against the fold at four viewport sizes rather than adjusted by eye. (2) The join popup is a centred modal. It was a strip at the bottom of the page and was missed; it now sits in the middle of the viewport at 18px over a 60 %-black, 6px-blurred scrim that covers the page, which is also a visible target for the click-anywhere dismissal. (3) A region with noCF-IPCountryfalls to the browser's locale before falling to USD: the header only exists behind the production proxy, so every preview showed USD to everyone, including a reader in Australia. The region subtag is read (en-AU→AU,zh-Hant-TW→TW), a region-less tag yields nothing, and a country header still outranks it. (4) The already-registered signup error carries a login button, because its message ends "Try logging in instead" and the form's own login link sits below everything the reader has just been told not to fill in. -
An unrecognised region is priced in USD, not AUD (2026-09-13). Region detection already existed —
CF-IPCountrypicks the currency, and an explicit cookie choice still outranks it — but everything unmapped fell to AUD on the reasoning that EPIC is operated from Australia. Unmapped now means USD, which is the currency the underlying prices are quoted in and so the one figure on the page that is not a conversion. Australia is listed explicitly as part of the same change, along with its external territories and the three Pacific states on its dollar:AUhad been relying on the very fallback that moved, so changing it alone would have priced Australian readers in US dollars — the one place it would have been obviously wrong. The no-country case is stated at the call site rather than deferring toCurrency::default(), so the visitor-facing fallback and the type's default can move independently. Verified end to end against the running page: US/JP/GB/NZ/CA/XX → USD, AU/KI/NF → AUD, DE/FR → EUR, CN → CNY, no header → USD, and aUSDcookie still wins over aDEheader. -
The bar's section links are centred on the page, not in the gap (2026-09-13). They had
flex: 1and centred themselves inside whatever space the two side groups left over, so the block sat half the difference between those groups away from the middle — measured at 14.7px right of centre, identically at every width, and it shifted every time either side changed: the currency control entering the bar moved it one way, leaving moved it back, and "Get Early Access" becoming "Login" moved it again. The side groups now grow equally from a zero basis, so the middle item is centred on the content box whatever sits beside it, and the content box is centred on the page because the bar's padding is symmetric. Measured 0px at 920–1900px, and still 0px with 200–300px of filler injected into either side group, which is the property that was actually asked for — centred at all times rather than centred for today's contents. -
The currency control left the navigation bar for the comparison section (2026-09-13). It spent the day at both ends of the bar — beside the wordmark, then beside the action — and is now in the heading row of the only section where the choice changes a number, which is also where
/currencyhas always redirected back to. A global control for a local effect is one people hunt for at the moment they want it. Three alignments, all of them properties of one flex row rather than measurements copied from the table, so none drifts when the table changes:space-betweenputs the control's right edge on.brand-table__wrap's right edge, the label keeps the content edge every other section's label sits on, andflex-startputs the control's top on the label's text top. Measured at 1280, 900 and 520: all three offsets are 0px.render_brand_document_forloses itscurrencyparameter andbrand_navtakes none, so every pre-login page renders the same bar. -
"Join the Waitlist" is "Join the Community" (2026-09-13). Both places it appeared — the hero's primary button and the closing section's eyebrow. No visitor-facing "waitlist" wording is left on the landing page; the word survives only in the
#joinanchor, the form's/waitlistaction and the operator console's own nav link, none of which a reader sees. It is now the section's heading ("Be part of the founding community.") and its lede ("our community portal") saying the same thing as the button above them. -
The join form answers in a popup, and its two buttons branch (2026-09-13). The three outcomes the server already produced — recorded, already on the list, refused — are reworded to the operator's exact sentences and promoted to a dismissible popup that closes on any click or after five seconds. Rendered inline first and promoted by script, not the other way round: a fixed popup rendered by the server would have nothing to close it where scripting is off, and would sit over the page until the next navigation; inline, it is a banner that needs no dismissing. The three cases the server never sees are handled in the browser, and only there: nothing typed plus
Notify Medoes nothing at all, nothing typed plusCREATE ACCOUNTopens an empty signup page, and a malformed address stays put and says so. A valid address plusCREATE ACCOUNTcarries it to/signup?email=…, which prefills the field. That value is a link anyone can send anyone, so it is trimmed, length-capped and dropped unless address-shaped — the renderer escapes, so the risk is not injection but what a hostile link can make the page say, and the answer stays "an address in a field you can see and edit".form.noValidateis set by the script rather than in the markup, so with no scripting the browser still refuses an empty or malformed address in its own words: the messages can only become ours where there is something to replace them with. -
The join section leads with the account, not the waitlist (2026-09-13). The lede is two lines as written, the optional short-message box is gone with its counter and the
maxlengthconstant that fed it — the server keeps rejecting an over-long message, which is the only one of the three checks an attacker could not skip anyway, and the field is still#[serde(default)]so the form posts fine without it. The emphasis is swapped:CREATE ACCOUNTis--primaryat 44px andNotify Meis--ghostat 88% of that, which is what the copy now argues for. The pair's heights come from one token (--brand-form-action-height) with the submit written ascalc(… * 0.88), so the ratio is the thing stated and a change to the primary height carries the proportion with it rather than silently altering it. Swapping it exposed a latent defect and fixed a live one.button, .btn— the pre-brand style the logged-in surface still uses — paints every bare<button>with--epic-accent-surface, which is blue; every.brand-btn--primarycovered it with white, so the leak was invisible for exactly as long as every<button>on this surface was primary. The compare section's no-script "Recalculate" submit was already--ghostand already blue, and renders only with scripting off, so nobody had looked at it..brand-btnnow statesbackground-color: transparent, which fixes the class rather than the instance. -
A navigation panel, and one account action instead of two controls (2026-09-12). The pre-login bar carried five section links, a "Log in" text link and a "Get Early Access" button; below the design's
mdbreakpoint the links wrapped into a second row, which made--brand-nav-height7rem there. That row is why the logo sat under the bar on a phone — the hero's--brand-hero-liftpulls its content 4rem up toward the bar to close the dead space true centring leaves,.brand-heroboth centres withalign-items: center(which overflows in both directions once content exceeds the box) and clips withoverflow: hidden, and on a short screen the lift had nowhere to go. The links now live in a<details>panel opened by a button to the left of the wordmark — the same disclosure the currency selector has always been, so it needs no script and has no focus trap — three bars coloured green-500 / green-300 / white, the hero headline's gradient stood on end. The bar is one row at every width down to 360px, and--brand-hero-liftis 0 belowmdwhere there is no dead space to reclaim. The two trailing controls became one that always reads "Login" and always goes to/login: a version that chose between "Login" and "Create an account" by reading the session cookie was built and then removed, because the cookie answers "has this browser signed in lately" rather than "do you have an account", and a button whose destination moves under the reader is worth less than one that always means the same thing. The panel is on every width, not only phones, and that is what makes the fixed label affordable — it is the only place "Create an account" appears on the pre-login surface, so hiding it at wide widths would leave signup reachable only by typing the URL, the same defect Sprint 9.8 found for/login. The inline section links now join the bar at 56rem rather than the design'smd: measured at 768px they needed ~14px more than the bar had and took a second row, the tablet form of the same defect, and deferring them costs nothing when the panel is one tap away at every width. Above 56rem they are in both places, which is the accepted cost of one panel rather than two that drift. The waitlist is unaffected: "Join" remains a section link in both places and/#joinis unchanged. -
A local dev server for the authenticated surface (2026-09-09).
cargo run -p epic-control --example dev-serverserves the realrouter_with_middlewaretower — the same CSP and response security headers production sends — on127.0.0.1:8898against the in-memory adapters, so/app/*, the playground and the operator console render with no Postgres, Ferrum, FerraVault or cluster./dev/loginplants a session cookie and redirects to/app, so a developer lands signed in rather than pasting a cookie into devtools. It closes the middle rung of the three loopscollaborator-build-and-preview.mddescribes: ~29 s against ~1.4 s for the pre-login site and ~15 min for a preview, and it is the loop that was missing where the Phase 7 F23 defects lived. Three deliberate departures from thevisual_regression.rsharness it is modelled on, each documented in place: the system clock rather thanTestClock(a dev server reading "as of 8 months ago" on/app/fleetteaches the wrong thing), theis_epic_operatorflag so/app/admin/*renders rather than 403s, and a session cookie written withoutSecure, which stops working the moment the example is bound to a LAN address to check a page on a phone. -
A workstation access runbook, and the ignore rule it turned out to need (2026-09-09).
docs/operations/collaborator-workstation-setup.mdrecords what a second owner's machine needs to reach the git server, the image registry and the preview slots from outside the cluster's LAN, and the toolchain versions that makedeploy/docker/build-image.shwork — the access layer undercollaborator-build-and-preview.md's loops. Three facts it exists to stop the next person rediscovering: the registry validates on the name, so its tailnet address is deliberately absent from the leaf's SANs and connecting by IP fails; preview DNS is not published, but the slots serve a public Let's Encrypt wildcard, socurl --resolvesubstitutes for it with full verification; and registry access is network reach plus CA trust, which is push and delete on every repository there, mitigated for substitution by digest-pinned admission but not for deletion, since a digest-pinned pod still re-pulls on reschedule. Review caught that the document asserted a protection that did not exist: it described the token file as gitignored whengit check-ignoresaid otherwise, and neither the pre-commit hook nor the secret-hygiene gate would have caught it — a Gitea token is bare 40-hex and matches no vendor prefix..git-auth-configis now in.gitignore, and the gate gained aCREDENTIAL_FILENAMEScheck matching such files by name, since their contents carry nothing to recognise; tested in both directions, so the check cannot quietly stop working. -
Raising a preview no longer needs cluster-admin (ADR-065). A preview namespace is now a slot —
epic-preview-slot-<n>, committed indeploy/k8s/preview/slots.yaml, permanent — rather than a name derived from a branch. That is what makes the provisioner credential expressible: Kubernetes RBAC ignoresresourceNamesforcreate, so a namespace-per-branch provisioner needsnamespaces: createcluster-wide and cluster-wide write on Deployments and Secrets to go with it, which is not a credential anyone can hand over.deploy/preview/mint-kubeconfig.shissues a bound token for an identity that cannot create, delete or list namespaces, holds nopods/execanywhere, reads exactly one production Secret (the preview wildcard), and holds no verb on RBAC objects. Three supporting changes:upmoved offkubectl execinto the Postgres primary onto the one-shot Job the reaper already used — and sends a SCRAM-SHA-256 verifier rather than the password, so the plaintext never crosses the API server; teardown deletes a preview's objects by slug label instead of deleting the namespace, so the slot's RoleBindings survive; and the preview hostname is slot-derived (slot-<n>.preview.<base>, two permanentAAAArecords published once bypreview.py dns), which removes the Cloudflare zone-edit token from the raise path entirely. New Stage-1 gates inci/preview/verify.py, all mutation-tested:slots.conf/slots.yaml/rbac.yamlmust agree; teardown's kind list must cover every kind the base emits (and name none it does not); and the RBAC claims — no exec, no wildcards, no RBAC verbs, namespaces bounded byresourceNames, and the namespace-scoped ClusterRoles never bound by aClusterRoleBinding— are asserted rather than written down. -
Phase 9 — Portability, rebrand & collaboration ✅ COMPLETE 2026-09-01. Tag:
phase-9-final(unsigned annotated — commit signing remains indefinitely deferred). Nine sprints, opened 2026-08-22 onphase-8-final. 90 §2 items across 9.1–9.8: 82 Delivered, 4 Partial, 4 Deferred at their sprint — and seven of the eight closed outright inside 9.9. The eighth — T-9.8.4, mail — had its whole chain proven end to end through the substitute path its own deferral named, and carries one verification step that no server-side check can make: reading a received message'sAuthentication-Results, which waits on Postmark account approval — a third-party review with no self-service path. Zero outstanding four-field deferrals, and one open verification, stated separately rather than folded into the first number. Three audits PASS: quality (workspace line 86.51 %, above the 80 % floor and above Phase 8's 85.16 %; 2 600 tests / 110 binaries / 0 failed), security, architecture 68/69 with one documented file-size deviation. Seven ADRs Accepted (058–064). Phase 6 (held open since 2026-06-20) and Phase 7 close at the same moment.The phase's thesis: three things that should have been configuration were facts of the source tree — the public domain, the host substrate, the address family. Its structural contribution is subtractive:
epichub.aiwent from 138 files to none the binaries run; five drifted client-address parsers became one resolution boundary; the manifest gained profiles instead of forks.What the gating sprint found is the part worth reading. Five defects, all fixed in-sprint, and three of them share one shape — individually invisible because the system was self-consistent around each:
- Postmark
default_fromwas read raw where ADR-059 derives it, so the client disabled itself at boot two log lines after the domain resolved successfully. A half-applied Sprint 9.2 migration whose sibling field had been converted and whose derivation had passing tests the entire time it was never called. verify_emailwas sent under two different field names, and once that was unified, under one wrong URL path: every verification email ever sent linked to a 404. Unifying the model had made the two senders consistently wrong rather than inconsistently wrong. Invisible until a message reached a real inbox.- Three DR components built, documented, and never once executed — both backup CronJobs unapplied, both network policies unapplied,
rcloneabsent from the workstation. No off-site backup had ever been attempted. grep -qunderset -o pipefailreported a loaded GPU driver as missing, becausegrep -qexits on match, the producer takes SIGPIPE, and pipefail fails the pipeline because the check succeeded. The sweep found the same shape producing a false negative in the Ferrum reseed drill, a Phase 5 gate.cargo denyadvisories failed on a yanked transitive crate (dev-dependency path).
And the Ferron dial had never been authenticated. All three EPIC→Ferron Bearer keys in FerraVault were stale and identical, returning 401 on both
/v1/admin/loadand/v1/chat/completions. It presented as one narrow broken endpoint on an otherwise-healthy dial — the inverse of the truth, because/v1/capabilitiesis unauthenticated, so the single route that did exercise auth was the only one still complaining. That is the entire content of the T-9.5.8 deferral, "the dial is re-pointed but nothing has traversed it".Closed on the deployed plane, with live traffic: a real inference over the public domain on both address families (peer addresses recorded, 26 SSE frames streaming over IPv6); per-request kill-switch enforcement end to end (
403 request_blocked→ revoke → restored in ~6 s); the first off-site encrypted backup and restore (custody → 2-of-3 Shamir → decrypt → 569 restorable archive entries, ciphertext sha256 byte-identical to the upload); and the full-verify auth SLO measured for the first time at mean 0.977 ms, p99 ≤ 5 ms — roughly 50× the standing 0.100 ms figure, which had been measured with no valid key in existence and therefore described only the reject path. See Sprint 9.9 §7.Phase 7 closed at the same moment, and closing it meant fixing what it still held. Its register carried F23 — eleven authenticated
/app/*pages scrolling sideways at a phone width,/app/admin/auditrendering 1009px into a 390px viewport — filed in 9.7 by the responsive gate the day that gate was written, and deferred there with its fix explicitly assigned to the walkthrough folded into 9.9. Tagging around it would have made Phase 7's own first exit criterion false. Fixed:.app-gridhad no narrow-viewport case at all, and.app-tablehad no scroll container. Notable that the obvious half was not the whole — containing the tables took eleven pages to three, and the last three neededgrid-template-columns: minmax(0, 1fr), because a bare1frtrack's minimum is its content's min-content width and one unbreakable model id widened the track and dragged the document with it.KNOWN_HORIZONTAL_OVERFLOWis now empty, so any page that overflows fails the build outright; the desktop pixel gate returned 0.00000 on everyapp_*page, proving the change is scoped to narrow viewports. The same run went red on six pre-login pages, and locating the diff before re-recording showed why: the differing pixels sit in one 14px band, the nav bar — the "Log in" link added the same day to a front door that had no way in. F24 (/app/securityin the sidebar 404s — never built, its functionality having landed on/app/me) is operator-deferred to L19 and filed with the detail that matters: theAuthApiError::Conflictredirect lands an already-enrolled member on the missing page, so it is an error path and invisible until someone hits it. Three Phase 7 sprint rows (7.4, 7.5, 7.6) had also claimedIN PROGRESSfor two months after shipping, corrected against their own checklists. - Postmark
-
Sprint 9.8 — mail worked in no sense at all, and each of five breaks looked exactly like the others (2026-09-01, Phase 9). Reported as an operator unable to receive a login email — which was true about the mail and false about the login: nothing gates sign-in on
email_verified, so the "check your email" screen after signup reads as a barrier and is not one. Behind it, five independent failures, every one of which alone produces "no mail" and none of which reports anything: the Postmark server token was never seeded;[postmark] default_fromwas read raw where ADR-059 derives it, so the client disabled itself two log lines afterpublic domain resolved— a half-applied Sprint 9.2 migration whose sibling field had been converted and whose derivation had passing tests the whole time it was never called;epic-controlhad no network path to Postmark at all, the Turnstile policy saying so in its own header ("Turnstile only, NOT Postmark") while the reservedepic-egressnamespace it defers to does not exist; the sender signature was domain-bound to the retired zone; andverify_emailwas sent asverify_urlfrom registration andverify_linkfrom change-email, which Postmark renders as empty rather than as an error, so one of the two paths had always delivered a verification mail with nothing to click and logged a successful send. That last one surfaced only because the merge fields had to be written down for the operator to author templates against — they are an external contract and had never been stated in one place. Fixed by construction (templates::models::verify_email, one builder, both call sites) rather than by aligning a string. Egress is a CIDR group, nottoFQDNs: Sprint 9.5 measured that any L7 DNS rule redirects every query from the selected endpoints through Cilium's DNS proxy at ~20 s per name, and because the redirect is per-endpoint that would trade a blocked mail path for a broken control plane — so the file states its own width plainly (82 AWS us-east-2 prefixes, egress-only, one port, one workload) and names the narrowerepic-egressanswer all three CIDR groups should collapse into together. DKIM and thepm-bouncesReturn-Path published and verified, with no Postmark SPF include — correcting an instruction given earlier in the same sprint, since Postmark aligns SPF through the Return-Path and a secondv=spf1TXT is a permerror that fails every message rather than only the new ones. The ADR-059 gate then caught two hardcoded domain literals in this work's own new code. See Sprint 9.8 §7. -
Sprint 9.7 — the pre-login brand surface, the waitlist, and three defects the design audit found (2026-08-26, Phase 9). The supplied
epic-ai-solutionsdesign ported to Leptos SSR: eight sections, self-hosted fonts, server-side currency selection fromCF-IPCountry(the design's browser-sideipapi.cocall — every visitor's IP to a third party with no processor agreement, on the page that also links the privacy policy — is gone), and a re-encoded backdrop at 171 KB against the source's 2.7 MB, compressed far past what a photograph normally takes because it sits under a 90 % black wash: measured against the source washed identically, the worst channel error anywhere in the frame is 10/255. An operator design audit found three things and all three were real. The "weird side panel" in Built on trust was.brand-gridkeeping user-agent list styling — this stylesheet declines Tailwind's preflight, and unlike its two sibling grids that rule never reset it, so a 40 px inset painted as a stray panel because the grid's background is what draws its gridlines. The hardware cards' editable per-kWh rate was dropped in the port; rebuilt so it works with scripting off (aGETform the server renders from, so the result is a link you can send) with live recalculation added on top — and a browser probe caught the two paths disagreeing, the script rendering350where the server rendered$350. The colour drift was four separate slips, including the two comparison columns rendering the same kind of value in two different colours and the primary button dimming on hover where the design brightens. A second report — a horizontal scroll on a phone — produced a gate..brand-orb--centreis a 37.5 rem decorative blur and only the hero clipped its own, so a 390 px page could be dragged 105 px sideways to look at it. The pixel baselines are recorded at 1280 px and could never have caught it, soresponsive_pages_do_not_scroll_horizontallychecks a property instead: every page at 390/430/600 CSS px, naming the widest offending elements, asserting on the width actually reached rather than the one requested (headless Firefox will not open a window under 500 CSS px, and a test that believed the answer would be quietly weaker on one engine). It was red on its first run, on eleven pages nobody was looking at — every/app/*and/app/admin/*page scrolls sideways on a phone, because.app-gridis a fixed 276 px sidebar at every width; filed as F23 and enumerated in a list the gate asserts exactly, so the debt is visible and can only shrink. The waitlist (migration 0051): operator-global and not RLS, because a pre-signup visitor belongs to no organisation; erasure is a realDELETEand the append-only audit log records a BLAKE3 hash, never the address, since an address written there could not be erased in turn. Its own §9.7 rate-limit band rather than sharing the abuse-report bucket, a role-address Postmark notification (the design mailed a personal Gmail through Resend), and an operator console whose CSV export defangs formulas typed into the public message field. The Postgres suite found a bug the in-memory adapter could not:erase_by_emailbound the address astext, andcitext = textresolves by casting the column down — an erasure request in a different case than the person originally typed would have matched nothing and reported success. And submitting against the live preview found that all three public forms drew a Turnstile challenge and then skipped verification when the verifier was unwired, reporting a protection that was not in force; closed with one accessor all three now read. See Sprint 9.7 §7. -
Sprint 9.7 — one site instead of a landing page and everything else, and two accessibility gates that found real dead ends (2026-08-26, Phase 9). ADR-062 Accepted. Fifteen pre-login pages moved into the brand chrome, keeping their markup: a
.brand-proseblock re-points the shared content vocabulary at the brand tokens inside that column, so the dashboard re-skin later gets the same treatment instead of a second migration. Page metadata became one value wherecanonical_path: Nonecauses thenoindexdirective rather than sitting beside it as a second switch, which makes both wrong states unrepresentable — an indexable page with no canonical is how duplicate URLs get indexed separately, and anoindexpage carrying a canonical asks a crawler to both drop and consolidate the same URL. New/sitemap.xmland aSitemap:line inrobots.txt, derived from[domain]; a preview serves a valid but empty urlset, because listing production's URLs from a branch deployment invites a crawler to attribute one to the other. The SEO gate checks both directions with an independent page list (deriving it from the sitemap would make the check circular), and a third test requiring every public path to be in exactly one of the two lists immediately caught/reset-passwordand/verify-emailin neither. Two contrast/landmark gates run in a real browser on all twenty pages, and building them correctly was most of the work: colours are canvas-resolved because Firefox returnsoklch()verbatim and anrgb()regex on it produces confident nonsense; backgrounds are resolved by walking ancestors because measuring againstdocument.bodymarks every dark-on-white button as failing at 1:1; and the first working version reported every element at 1.00:1 against itself, becausefillRectwith a transparent colour does not overwrite the pixel without aclearRect. The ramp was then re-measured against the background the page actually paints —.brand-backdropputs the visible ground at#0d0d0d, not the#000000token — which is lower contrast, and flattered exactly the two values closest to the line:ink-subtlewas documented "large text only" and used on four pieces of 12–14 px copy at a real 4.02:1, and--color-brand-ink-faintwas a token for a value this project's own comment says "FAILS AA outright, not given a token". Every/app/*page failed AA on its primary button and current nav item (white on blue-500 = 3.68:1); no single value fixes it, because the same token is a foreground on the dark ground where blue-600 drops to 3.76:1 — so the accent is two tokens now. Result: 20 pages, 669 text elements, zero below AA, with an empty known-failures list rather than a debt register. The landmark gate found a dead end:/app/usageand/app/fleethad zero landmarks and zero focusable elements, rendering a hand-written HTML string with no stylesheet, navigation,<main>or skip link — seven handlers did this, and for a keyboard or screen-reader user it is a page with no way back to the site, reached whenever a data source blips. Skip links (WCAG 2.4.1) on all four document shells, asserted to be the first focusable element; the playground's prompt textarea gained the accessible name a placeholder never was. Also:/transparency's failure is503+Retry-Afterin chrome rather than a bare500that a crawler reads as a reason to deindex; a 1200×630 social card with a test that parses the JPEG's own frame header, because a wrong aspect ratio is silently cropped rather than rejected; andci/domain/verify.pynow sees untracked files after reporting green on a tree holding thirteen literals that had not beengit added yet — a gate that only sees committed work objects one commit after the literal is already in the history. See Sprint 9.7 §7. -
Sprint 9.6 — preview environments: a branch running live, isolated by the systems that own the data (2026-08-26, Phase 9). ADR-063 Accepted.
deploy/preview/preview.py up --branch <b>deploys a branch to<slug>.preview.<base>— its own namespace, its own LoadBalancer VIP, its own Postgres database seeded fresh, a 72 h TTL and an hourly reaper. Two measurements decided the architecture and neither matched the plan's assumption: the worker has 939m of CPU free and a Ferrum node under thesingle-nodeprofile requests 500m, so a Ferrum per preview does not fit even once. Previews therefore share the substrate and are isolated by the system that owns each piece of data — a separate Postgres database withCONNECTonepicrevoked; a separate Ferrum keyspace; a separate FerraVault mTLS identity and JWT signing key; and no internet egress at all, so a preview cannot mail from the production sender. That is the stronger arrangement (a NetworkPolicy protects a port; a Postgres grant protects a database) and it is written into the manifests rather than left to be found in a packet capture. The cap is a table of addresses, not a number: a slot is a pair of addresses a human configured on the router, so running out is a refusal naming which branch holds each one, and raising the ceiling means going to the router. The signing key is the sharpest edge — internal ids are sequentialBIGINTs, so a preview signing with production's key would mint tokens that authenticate as real users; three independent things prevent it and Stage 1 asserts all three, twice.epic_store::KeyspaceScopelives onFerrumClientrather than on the adapters, because the first cut put that decision at fourteen call sites whose failure mode is silent. Two latent defects closed on the way:epic:diag:settingsis a fixed Ferrum key, so any second EPIC deployment would have overwritten production's diagnostic settings with no collision involved; anddataclon theepicdatabase was NULL, so PUBLIC heldCONNECTand every login role in the cluster could open a session on it. Previews arenoindexthree ways (robots.txt, the meta tag, andX-Robots-Tagnear the outside of the tower so it covers a 429, a timeout, a panic-to-500 and every asset with no<head>), plus a banner on every page — production byte-for-byte unchanged, the robots tag absent rather thanindex, follow. Proven live, not asserted: a preview raised end to end on the cluster serving real Let's Encrypt TLS with chain validation passing, its own keyspace and database confirmed in the boot log andpg_stat_activity; and the reaper drill run against a real expired preview, a real database and a decoy namespace — it reaped exactly the one preview, leaving the decoy, every production namespace, theepicdatabase and all seven cluster roles untouched. Ten mutations on the guard and slug and ten on the gate are all caught, and the process found four defects in this sprint's own work, each an assertion true for the wrong reason — a guard claiming three independent conditions where one was implied, a slug test that passed with case-folding removed, a banner assertion matching the inlined stylesheet, and a gate check asserting at least one key-scope application rather than one per dial site. Six more came only from execution:python3-minimalis the interpreter without the stdlib;epic-stateis default-deny so the teardown Job hung rather than failed;psql -c "A; B"runs both in one transaction andDROP DATABASEcannot; namespace-first teardown leaves an invisible orphan database;kubectl_jsonread every non-zero exit as "does not exist"; and a preview's derivedapp_originwas production's apex, so its links pointed at the live site. A post-close self-review found nine more, one critical:preview.pyused PEP 701 f-string syntax and therefore did not parse on the Python 3.11 the reaper image runs — the deployed reaper worked only because its image predated the line, and the next rebuild would have produced a reaper that died at startup every hour until the host filled. The image now compiles its scripts with the interpreter that runs them, so that class is a failed build. Also: one failing teardown aborted the entire reap run, so a single stuck preview blocked every other one forever; the teardown diagnostic could be empty (a pod that never starts has no logs) and now leads with Job conditions and container states; the branch name went raw into generated YAML, whichgit check-ref-formatproves is reachable;epic_store::KeyScopewas renamed toKeyspaceScopebecauseepic_core::KeyScopealready existed and the collision had produced a doc-link ambiguity that was patched rather than removed; a new gate checks the boundary keys actually cross rather than which files mention scoping; certificate-expiry parsing was locale-sensitive;copy_tls_secretwould apply a well-formed empty TLS Secret; and slot addresses were never validated as addresses. The in-cluster reaper had never executed its destroy path — its one scheduled run was a dry run with zero previews — so it was drilled properly, from the CronJob under its own ServiceAccount, against a real expired preview and two decoys including one with a mismatched slug. The rename also produced a guard worth keeping: because a preview default-pins to production's image, an env override the running binary does not understand is silently ignored, soupnow reads the keyspace posture back out of the pod's own log and refuses rather than reporting a preview isolated without evidence. One live production defect was caused and fixed in the same session, and it is the most instructive finding here: re-pinningepic-controlbumped the web Deployment and not the meter aggregator, which shares the web Service's selector on the explicit basis that it is the identical binary. The result was two pods behind one public address serving different builds —/robots.txtreturned200 404 200 404 200 404 404 404 200 200, an exact 50/50 split that reads as a flaky route rather than a version skew, while every pre-existing page stayed 200 because it exists in both builds. It was visible only because this sprint had just added a route that one build lacked; a behavioural difference would have presented as an intermittent bug with no pattern.check_image_paritynow requires everyepic-controldigest pin underdeploy/k8s/to be identical, because an invariant living in three files cannot be held by a comment in one of them. Two of the ten mutations on the new gate checks survived the first pass, both real: one check was satisfied by its own explanatory comment, and another's vacuity floor sat below the real count. See Sprint 9.6 §7. -
Sprint 9.3 — IPv6 readiness, code plane: the address family becomes a configured value (2026-08-23, Phase 9). ADR-060 Accepted. New
epic_core::net(pure, no I/O) —BindSet(validated non-empty bind list; parses a string, a comma-string or a TOML array; rejects duplicates and wildcard/specific conflicts at config load rather than atbind()), the single v4-mapped normalisation point, the forwarding-header element parser, the trusted-hopClientIpPolicy, andIpv6PrefixLen+rate_limit_identifier— plus a newepic-netcrate that binds one socket per family withIPV6_V6ONLYset explicitly, so behaviour never depends on the host'snet.ipv6.bindv6onlysysctl. Every listener on both planes (edge h3 + h2, control h2 + h3,/metricson both) now takes a bind list; the h3 listener binds its ownUdpSocketand hands it toquinn::Endpoint::newrather than lettingEndpoint::serverinherit the sysctl.EPIC_*_BINDaccepts the comma form, since env is the one layer without TOML arrays. ADR-026 invariants re-measured under v6, not inferred: 9 allocs/token over IPv6 (vs 11 over IPv4) and cancellation p50 8.5 ms / p99 10.7 ms with the peer family alternating across the sustained-load drill; the dual-stack listener proved on real sockets, both families on the same port, each completing a TLS request through the real router. T-9.3.10 verified against sibling source (record): Ferrum and FerraVault both accept bracketed IPv6 and already carry an SNI override; Ferron was the gap and it was EPIC's —tonicderives its SNI name fromUri::host(), which for a bracketed IPv6 authority includes the brackets and is not a validrustlsserver name, sohttps://127.0.0.1:50051configures andhttps://[::1]:50051does not. Fixed with[ferron] tls_server_name, matching the field both other siblings already take; not hypothetical, since the deployed edge dials Ferron at an IP literal (ADR-039). Also recorded for Sprint 9.4: Ferrum's client returnsv4.or(v6), so a dual-stack Ferrum Service is dialled over IPv4 — deliberate upstream behaviour, not a defect. Deployed ConfigMaps stay IPv4-only deliberately (the cluster isSingleStack; binding[::]on a pod with IPv6 disabled crash-loops atsocket()), with the list syntax documented so Sprint 9.4 is a one-line flip. See Sprint 9.3 §7. -
Sprint 9.2 — Domain indirection: the public domain becomes a configured value (2026-08-23, Phase 9). ADR-059 Accepted.
epichub.aiappeared in 138 tracked files, 484 times; it now appears in five, none of them code the binaries run — the parity proof, one comment in an applied (checksum-immutable) migration, the generatedepic-domainConfigMap plus the gate that reads its source, and FerraCert's own config. No Rust source file, no Kubernetes workload manifest and no configuration either binary reads contains it. Newepic_core::domainderives every host, origin, cookie domain, role mailbox and ACME SAN from one[domain] base, resolved once at boot into an immutableDomainSnapshot(newtypedDomainBase/Host/Origin;Host::is_withinis label-aware, so the origin allowlist refuses a look-alike registration thatends_withadmits). Deployed values arrive asEPIC_DOMAIN_*from the generatedepic-domainConfigMap viaenvFrom— no templating step, so ADR-016 §3's vanillakubectl applyposture is unchanged;deploy/domains/check.pyverifies the deploy-layer half (FerraCert SANs, brand identifiers, Cargo homepage) that cannot be substituted. K8s label namespaceepic.epichub.ai/→epic.internal/across 35 files (ICANN-reserved, so it can never track a rebrand again). No behaviour change, proven bycrates/epic-config/tests/domain_parity.rs, which pins every derived value against the literal it replaced. New Stage 1 gateci/domain/verify.py(22 tests incl. the end-to-end negative) makes the next hardcoded literal fail the build, and already policesepic-ai.solutionsahead of the Sprint 9.8 cut-over. Four latent defects found and fixed: a Kyverno cosign policy naming a registry host that has never existed (would have admitted every image unverified while reporting healthy); a chaos runbook left selecting the old labels;Cargo.toml/OCIimage.sourcepointing at a non-existent GitHub repo; andepic-config'sdiff_bootsilently not covering the two Sprint-3.3 signing sections. A post-close self-review found five more, all fixed rather than recorded: the visual-regression gate was failing on six pages (the SSR identity was never installed in the harness — now it resolves production's, and all 20 pages return pixel-delta 0.00000 against pre-sprint baselines, which is the real no-behaviour-change proof);AppState::wirefell back toexample.testwith a warning and now fails closed, because a control plane that boots without a domain comes up healthy and serves a wrong site that no readiness probe catches;allows_browser_originallocated on the per-request path while four documents said it did not (rules unified into one allocation-freecheck_host, pinned at exactly zero by a counting-allocator harness);Originserialised to a string but deserialised from a struct, so it did not round-trip; anddeploy/domains/check.py --env dev|previewreported 19 failures for manifests that were never theirs (a check that goes red when used as documented teaches people to ignore it). Because both planes now fail closed,check.pygained a seventh check — every plane workload must receiveEPIC_DOMAIN_BASEand its namespace must hold the ConfigMap — with a vacuity guard, and every check was mutation-tested to confirm it is not inert. See Sprint 9.2 §7. -
Sprint 9.1 — collaboration & git-server enablement (2026-08-22, Phase 9). EPIC becomes a two-owner project: ADR-058 (review authority, why the sprint documents stay authoritative over the issue tracker, the break-glass override, the fast-loop/ceremonial-gate split) and ADR-064 (a phase takes its number when it opens and never gives it up; unopened phases are named — ending three renumbers in ten weeks). Ships
CODEOWNERSwith aci.shlint against inert rules, PR + issue templates,.gitea/workflows/README.md, aCONTRIBUTING.mdsecond-developer path executed literally on a clean container, the gate on push-to-mainwith deduplicated issue-on-failure,.env/credential sweeps in Stage 1, andtools/sprint-sync— a one-way sprint-document → Gitea bridge (9 milestones, 102 issues) with a drift check proving the board cannot diverge from the contract. A self-audit pass then found and fixed eight defects in that work, three of them live — most seriously that Gitea caps an issue page at 50, so the failure reporter's dedup was blind to any report older than the 50 newest and would have filed a duplicate on every run (the nightly carried the same bug); and that the secret sweeps gave different answers under GNU grep and ugrep, so they now run on one engine with unit tests. See Sprint 9.1 §7. -
Sprint 8.2 — shim tool surfaces + compat hardening (2026-07-03, Phase 8). The OpenAI + Anthropic shims carry tool-calling end-to-end and land the generic guided/structured-output surface (ADR-055 Accepted). Native
guidedsubstrate onInferRequest(illegal states unrepresentable; threaded to Ferron'sguided(18) field). OpenAI:tools/tool_choice/response_format;message.tool_calls(content null on tool-only turns); streamingdelta.tool_calls[]fragment forwarding (never buffered); assistanttool_calls+role:"tool"history round-trip. Anthropic:tools/tool_choice/output_format;tool_use/tool_resultcontent blocks (Phase-3.2 reject arm removed); six-event SSE machine extended with lazy dynamic block indexing (content_block_start(tool_use) →input_json_delta→content_block_stop);POST /v1/messages/count_tokens(no Ferron dial) + lenientanthropic-version. Conformance gates extended (tool-calling + structured output; SDK re-pin openai 2.44.0 / anthropic 0.116.0 — 15/15 + 11/11 green); real-SDK agentic-loop drill recorded (ci/drills/phase-8-harness/). ADR-026 re-asserted on both shim tool streams (zero per-token alloc, cancel < 100 ms, ~256 ns/~120 ns per-fragment reframe). See Sprint 8.2. -
Sprint 8.1 — native tool-calling substrate (2026-07-03, Phase 8). EPIC's native API surface gains tool-calling end-to-end against Ferron's grammar-constrained engine (ADR-054, Path A): re-vendored Ferron proto (sha256 provenance guard); native
tools/tool_choice/tool_calls+StreamEvent::ToolCall{Start,Delta,End}; edge renders tools into the model chat template;translate.rsmaps the structured tool frames (Malformed stub removed);usage.prompt_tokenssourced fromFinish.prompt_tokens. Live drill PASSED against the real Qwen2.5-3B-Instruct-FP8 tool-calling build. Roadmap renumber: new Phase 8 (tool-calling) inserted, Launch → Phase 9, hardware → Phase 10+. See Sprint 8.1.
Changed
-
A due OFAC re-screen no longer fails every open PR at once (2026-10-25).
seed_re_screen_is_not_overduefails at 30 days after the list's# published:date by default, and at 20 in the nightly sweep (EPIC_OFAC_RESCREEN_DUE_DAYS), whose failure issue is now the prompt. From day 20ci.shStage 1 also warns in every gate's log without failing. The guard still refuses every signup at 35 days. On 2026-10-24 the old 20-day test failed main and every open PR, with 15 days still to go before production was affected. -
/pricingand/transparencysay account and pool, not group (2026-10-23, #400; Glossary #381). Both pages carry the owners' wording (Melvyn, 2026-10-23; wiki Glossary §8) with typos and light polish fixed./pricingis four sections — Pricing, Slots (Term, Capacity, Sharing), Models, What is not in scope — and still names no price: the configured slot prices are seeded, in AUD, and not read by this stateless page, so it says a guideline price will be given, in the landing page's words, until the owners decide to publish one./transparencycounts and suppresses accounts, keeps its live threshold, day counts and figures, and says the page fills in once the first server comes online. Tests pin the new text and that no price figure or "group" appears. -
orgs.seat_countisslots_held, and a ballot'sseat_countisvote_weight(2026-10-23, #381). The owners' names for what the two columns hold since ADR-069: an account's slots that are active or on hold, of any kind and with any rights; and a ballot's weight when it was cast (Org::vote_weight). Migration 0094 renames both, their three constraints and PG18's NOT NULL constraint names, and recreatesenforce_seat_limit()andslots_recount_orgs(), whose bodies named the column.Org::seat_count,BallotInsert/BallotRecord/CastRequest.seat_countand the test builders follow. For one release each old name stays as a read-only generated column, with the edge's column grant copied to it in the same transaction, so the planes can roll one at a time and roll back. The migration refuses to run while a vote is open, since a control plane on the release before casts into the old ballot column. The next release drops both (docs/operations/migration-runner.md§5). Wire names stay: the Postmarkslots_remainingkey and theseat_countkey inmodel_vote.castaudit payloads. ADR-014 and ADR-069 gain amendments. -
The avatar upload limit is 10 MiB, down from 15 MiB (2026-10-22, owners' decision on the #372 review; ADR-074 §2, amended). The large platforms cap a profile photo at about 5–10 MB (Google about 5, LinkedIn about 8, Discord 8–10), and a stored avatar is at most 256 × 256, so the extra 5 MiB a body could hold bought nothing a member sees. ADR-074 chose 15 MiB so that every default phone JPEG (5–12 MiB) fit; most still do, but a phone photo over 10 MB now gets the "too large" page. That page now names the limit, read from
AVATAR_MAX_BODY_BYTESso it cannot drift, and says how to fit under it: make a smaller copy of the photo, or choose a different picture. With JavaScript the avatar cropper uploads a 512 × 512 square, not the photo, so only a browser without JavaScript, or a file the browser cannot open, meets the limit. Memory: at most 4 bodies in flight per replica is now 40 MiB, not 60, and the replica's worst case ≈ 560 MiB, not 580.body_limit::the_avatar_limit_is_ten_mibpins it: 10 MiB passes, one byte more and 15 MiB get the page. -
One word for the account on screen, and the stale fair-share formula fixed (2026-10-22, #381). The member portal and the operator console called the thing that holds slots a "group"; they now say "account": the console's Accounts tab, Suspend account and Unsuspend account, "account not found", the checkout's Account name (optional), and the Fleet, Security, keys and audit-log lines. The signed-in Group tab is now Pools (Melvyn, 2026-10-23), the tab ADR-082's PL.4 turns into the pools tab: its label in the tab bar and in Profile's opening-tab list, its page title and hidden heading, its search section, its name in the console's paused-boxes list, and the Members box's line, "Everyone in your pools, and their role." Its stored ids (
MemberTab::Group, page keygroup) and its route/app/orgare unchanged. Its "Group details" card is Account summary, because the Account tab already has an "Account details" card. The pre-login invitation page keeps "An EPIC group", and the public pages are unchanged. The console's per-server "Pools" card is now Paid and complimentary places, and the test-key option says "sandbox models" (owners, "for now"), leaving "pool" to ADR-082's pools. Stale definitions fixed:fair_weight = seat_countin DESIGN.md (§6.3, §8.3, §9.5, §10.4, §12.3, Appendix A) and in code comments, and ADR-045 and ADR-048 marked superseded in part by ADR-067 §1 and ADR-069 §4 with their text kept; the weight ismax(slots_capacity, 1)(Org::fair_weight). Also Appendix A's slot entry ("2,500 will be sold") and ADR-066'sSubscriptionEvent(nowPurchaseEvent). On-screen text and docs only: no stored key, route, error code or column changes. -
Changelog entries are one file per PR in
changelog.d/, so PRs no longer conflict onCHANGELOG.md(2026-10-22, #376). Every PR used to add its entry at the top of the same section, so each merge tomainleft the other open PRs conflicting on the changelog alone: six of the eight conflicts on 2026-10-22 were nothing else, each costing a merge-in and a fresh PR Gate run. A PR now addschangelog.d/<PR number>-<words>.<section>.md.ci/changelog/compile.pyfolds the fragments intoCHANGELOG.mdand deletes them, only onmain, as the first step of building a production image:deploy/docker/build-image.shrefuses a production build while any fragment is committed or staged, because the public/changelogpage embedsCHANGELOG.mdat build time and a build cannot commit; a preview build passes--preview(tag-preview, labelepic.internal.image.preview=true). Each entry is placed by the date in its title's closing parentheses among the entries already in its section, newest first, and same-day entries by PR number, highest first; it ends with a hidden<!-- changelog.d/<name> -->marker, so a fragment that comes back after compiling is refused even if the entry was edited, and a later compile knows the entry's PR number. The gate refuses a missing or future title date, two dates, a title that does not close on its date, a title already inCHANGELOG.mdor in another fragment; only files git tracks are compiled, and trailing spaces are kept.ci.shStage 1 checks every fragment (ci/changelog/verify.py, 89 unit tests); the domain gate treats a fragment likeCHANGELOG.md.CLAUDE.md,sprints/CLAUDE.md,CONTRIBUTING.md, the PR template and the preview and migration runbooks say so. -
The dashboard modules' test routers become one test-only helper, and the h3 body-limit test drives the mounted tree (2026-10-22, #358 review). The 21 dashboard modules'
router()s, called only by tests since #358, give way to one#[cfg(test)]routes::web::test_router(routes)that folds a module'sroutes()asweb_routerfoldsall_dashboard_routes(), so production cannot mount a dashboard module apart from the list (referral::public_router()andemail_change::public_router()stay).integration_h3_body_limitservesrouter_with_middlewareinstead ofavatar::router(), and a new test stores a real image over 64 KiB through the mounted avatar route over h3 and refuses one over 15 MiB; removing the route'sDefaultBodyLimitfails it. No route changes:ci/routes/verify.pyreads 56 controls against 221 routes, its dashboard fixture now in the tree's new shape. -
A newly published RustSec advisory no longer fails every open PR (2026-10-24).
cargo auditandcargo deny check advisoriesrun on the PR Gate only whenCargo.lock, aCargo.toml,deny.tomlorci.sh(the two ignore lists) changed since the base (ci/advisories/should_run.py,EPIC_ADVISORIES=changed); the log says why either way.cargo deny's bans, licenses and sources still run on every gate. The nightly sweep and a local--fullrun the advisories every time, and the nightly's failure issue is where a new advisory is now reported. Prompted by RUSTSEC-2026-0387, which failed main and every open PR on 2026-10-24 within an hour of its publication, for a crate only FerraVault's test server pulls in. -
The PR Gate does less avoidable work (2026-10-22, #380). It had grown from ~65 to ~145 minutes. Four changes: (1)
cargo geiger, ~45–70 minutes and informational, runs on the gate only when a dependency manifest changed orunsafewas added since the base (ci/geiger/should_run.py,EPIC_GEIGER=changed); the log says why either way, and the nightly sweep and a local--fullstill run it every time. (2) Thetestprofile carries line tables instead of full debuginfo, which makes a test binary ~60% smaller (anepic-controlsuite went from ~1.6 GB to ~0.6 GB). (3) Each crate's integration suites build as one binary,tests/integration/main.rs, with each suite a module under its old name: 130 test binaries become 38. The 16 that keep their own process say why in their header (a#[global_allocator], a latency measurement, a WebDriver, or process-wide state), andci/test-layout/verify.py(Stage 1) fails on an unexplained second binary or a suitemain.rsnever declares. Before and after list the same 788 tests by name. Run one suite withcargo test -p <crate> --test integration <suite>::. (4) The runner goes from 4 CPUs / 16 GiB to 6 / 48 (EPIC_CI_JOBS6,EPIC_DOC_JOBS4);docs/operations/host-resource-split.mdrecords the memory trade. -
The pre-login pages in the owner's logo pack (owners' second landing round, 2026-10-21). The hero shows the owner's website badge, served as his own
Badge.svg(26.8 KB over the wire, sharp at any zoom); the pre-login bar and the comparison table's EPIC column show theEPICletters alone, as a vector (epic-header-mark.svg, extracted from his text-only logo bytools/make-header-mark.py). The pre-login pages take the logo's palette, defined once on.brand-page: the hero headline word by word in the strapline's colours, section headings in a blue–green–orange sweep, a green accent, blue links, orange figures, and a call to action in the badge frame's band. Every pair clears WCAG AA (tools/palette-contrast.py), andink-subtlemoves to oklch(66%) so small print stays above 4.5:1 on the new ground. The photograph behind the pages is replaced by a star field like the owner's social banners (tools/make-starfield.py: two depths of seeded stars over the banners' teal glow, 6.5 KB where the photograph was 340 KB), whose stars brighten around a hovering pointer (starfield.js; nothing on touch screens, no fade under reduced motion). Two extras, on the landing page only and behind one switch (pages::landing::fun::FUN_EXTRAS): an occasional shooting star, and the badge turning to face the pointer; neither runs under reduced motion. The dashboard is unchanged. -
The Dashboard's Models box: the owners' intro, and which access applies to you (2026-10-21, #367; issue #276). The intro is Maran's wording as the owners accepted it, and a line under it says whether the viewer's slots give unlimited access, an access code gives access until a stated time, or neither yet, with a link to the playground's code form and, for an owner or admin, to the slot options (the new
access::can_buy_slots, which/app/billingnow uses too, so the box offers a purchase only to those checkout lets buy). The API keys section and the playground page follow the same rule (owners, 2026-10-23): their "See slot options" is shown only to an owner or admin, and anyone else is told slots are bought by an owner or admin of the account. The box says that too, and the header search offers "Buy slots" only to an owner or admin, in its suggestions and its results (filtered per request, like paused boxes). It is decided by the newepic_control::access::model_access(the API right, else the playground's ownplayground_access), and the box and the playground page show the access code's end through the same formatter, so the two cannot disagree. Suspension wins over a running access code (owners, 2026-10-22): a member of a suspended or closed account gets no access from one on the playground page, the key mint or the box, and a code redeemed for such an account is left unused with a notice saying why (ADR-067 and ADR-069 amended, DESIGN.md updated). Theapp_homevisual baseline needs re-recording. -
The member dashboard is mounted from the list the appearance sweep reads (2026-10-21, #320 item 9; issue #199 C14).
web_routermerged each dashboard module'srouter()by hand, whileall_dashboard_routes()listed the same modules for the appearance sweep, so the two could disagree.web_routernow foldsall_dashboard_routes()itself: a dashboard route is served exactly when it is listed. The two modules that also serve a public page get apublic_router()for it (referral's/r/{code},email_change's confirm and cancel pages), mounted beside the list; each module's ownrouter()stays, for its tests.ci/routes/verify.pynamesall_dashboard_routesa collector, as it does the console'sall_admin_routes: what it lists is mounted, and a module it leaves out serves nothing and fails the gate (mutation-checked on the real tree withkeysdropped; two new fixture tests). No route changes: the same 221 routes are served. -
Tests: the cancel-budget test bounds its median against the runner's own baseline (2026-10-21; issue #357).
cancellation_budget_holds_at_100_rps_sustainedfailed a PR Gate run with no code change: its median, measured from the batch start, is mostly how fast the runner moves 100 requests to their first chunk, and on the 4-CPU runner that sat at 21–25 ms against a fixed 21 ms bound. Seconds now alternate (ABBA) between cancel windows and control windows running the same batch, where the control samples the same streams' first chunk and drops them after the window. A cancel sample is time-to-first-chunk plus the cancel path and a control sample time-to-first-chunk alone, so the test bounds the difference of the medians, cancel p50 − control p50 ≤ 10 ms, which isolates the cancel path and does not widen on a slow runner (a ratio would have allowed ~21 ms there). It keeps the 200 ms per-iteration bound and an absolute 50 ms backstop on the median, looser than the old 21 ms because time-to-first-chunk alone reaches 21–25 ms on the runner. Both windows now settle before the next starts, and every window must observe all of its cancels (or first chunks) inside the sampling window, so a late or lost cancel fails instead of dropping out of the median. Every run prints both medians, the difference and the bounds past libtest's capture, so passing PR Gate runs record the runner's numbers. On the laptop the difference read −2.8 to +1.4 ms on unchanged code; a 12 ms delay in the cancel arm fails it, a 7 ms delay does not, and 1 ms of blocking per cancel sits at the bound (failed 3 of 6). Each peer family now carries half the cancels (they all went over IPv4). The default run is 20 s, keeping ~500 cancel samples. Test-only. -
One electricity rate for the six "real cost of going local" cards (owners, 2026-10-20). The per-card rate fields are replaced by one field at the end of the introduction (
?rate=, still a GET form that works without JavaScript); each card now reads "Annual electricity @ " in the currency's own unit ($/kWh, €/kWh or ¥/kWh). The rate is bounded at 10 per kWh (was 100) and a single decimal comma is accepted. The hint names what the default is an average of, per currency: a US, Australian or Chinese national average, and a eurozone average for EUR (it said "a national average" for all four). The field takes the 2px accent focus ring the other brand controls have (it had a 1px border colour and no outline). A rate it cannot use is said in words under the field, and the field isaria-invalid; the red border followsaria-invalid, so colour is never the only signal (WCAG 1.4.1, 3.3.1). A repeatedrateorjoinedparameter keeps its first usable value, where it had become a plain-text 400. The landing page's wording is updated where the owners rewrote it (Why EPIC heading and hardware footnote, How it works step 3, two principles, the join lede), and the join lede's two lines no longer run together as text. GB300 is no longer considered (owners, 2026-10-22):/aboutnames the hardware the landing page does, the NVIDIA Vera Rubin NVL72 and the AMD Helios; the model-proposal form says "the fleet's hardware envelope"; and a test holds the landing, about, pricing and security pages and the proposal form to never naming GB300. -
Tests: RLS cases for the four user-scoped tables that had none (2026-10-16, #307; issue #200). The tables from migrations 0059, 0062, 0066 and 0068 each gain a positive and a negative case under the
NOSUPERUSER NOINHERIT NOBYPASSRLStest role; each fails with the policy dropped, RLS disabled, orUSING (true). Test-only. -
Tests and docs: the appearance review follow-ups (2026-10-16, #309; issues #199, #201, #203, #207, #210). The AA contrast proof uses
ThemeBase::grounds()itself; new tests holdgrounds()andappearance.js's live-hint grounds to the stylesheet's--epic-bgvalues; the account menu's identity and the console scheme gain assertions; references left stale by the ADR-073 split are corrected. The logout test's doc comment, which had fused onto the legal-items test, moves back onto its own test: the last open part of #210, whose file sizes and other fused doc comment were already fixed onmain. No behaviour change. -
Tests: the avatar pipeline on WebP, a real downscale, and uploads over 64 KiB through the production tower (2026-10-16, #317; issue #204). Three tests drive the real
process_avatar_upload(format sniffing, decode limits, Lanczos3 resize, PNG re-encode). Test-only. -
Tests: the signed-out sweep covers every dashboard route and method (2026-10-16, #318; issue #199 C17, C18). The sweep derives its routes from
all_dashboard_routes()and sends GET and POST to each until the handler itself answers, and the scheme gate's floor counts what it renders. Test-only. -
Production Ferrum rolled
sprint-9.16→0.3.0-fix-121: idle CPU 1.27 → 0.15 cores (2026-10-20, #293, ferrum#121). An idleferrum-0spent ~1.27 cores in its twoferrum-raftthreads, woken ~63,600×/s, because Ferrum's slot-migration controller waited on a fresh clone of openraft's metricswatcheach time and so never slept. Ferrum fixed it in chart 0.5.0 (tagferrum-chart-0.5.0,4d17d639). Measured on the live pod the way #293 specified (process CPU over 30 s, per-threadvoluntary_ctxt_switchesover 5 s), ten minutes after the roll: 0.150 cores process total, 0.004 of it Raft, Raft wakes ~70/s. What remains is the two shards and the wasm-epoch thread on their ~920/s tick, unchanged by this release. Image4d17d6392f5a-0.3.0-fix-121, pinned to Ferrum's own manifest digestsha256:6633aadc…. From 0.5.0 Ferrum publishes to the private Gitea registry, and the chart's default needs a pull secret. EPIC instead mirrored the image intoregistry.epic.internal:5000withcrane copy, which keeps a single-arch manifest byte-identical, so the cluster holds no Gitea credential and the pin is the upstream digest. No template, config, wire, client-crate or on-disk change: the rendered release differed from live only in the image digest, and the server-side dry run passed.helm upgrade --rollback-on-failure(release revision 8). Ferrum was down ~25 s (07:14:11–07:14:36 UTC). Both edge pods, control and the aggregator re-seeded in-process within 20 s (ADR-126), with no pod restarts. The CI test-server pin (ci/ferrum/test-server.pin) moves with it. The six Ferrum-backed suites ran strict against the new binary: 20/22 pass locally, and the twointegration_adapter_stackcases that need a Postgres testcontainer could not start one because the workstation's Docker is down. The release's other fix also landed: shards now pin only within the process's CPU mask, so those suites no longer abort on this workstation'suser.slice(sched_setaffinity failed for core 0). Rollback:helm rollback ferrum 7. Record:deploy/k8s/epic-state/ferrum-values.yaml. -
The second sign-in factor has one name: Multi-Factor Authentication, MFA for short (2026-10-16, #320 item 11; issue #208). #197 renamed it on the Account page, but the setup page, the page after enrolling, the login step and their error messages still said "two-factor", so "Set up Multi-Factor Authentication" led to "Set up two-factor authentication". They now use the Account page's wording, the login page included. A new test,
the_mfa_setting_has_one_name, fails on "two-factor" in any string literal ofepic-webor the control plane's web routes. The search index keeps "two-factor" and "2fa" as synonyms. None of these pages has a visual baseline. -
Transactional mail verified on a received message (2026-10-14, #91). Postmark approved the account; a real password reset reached an outside (
gmail.com) address with SPF, DKIM (d=epic-ai.solutions) and DMARC all passing. Closes Phase 9's open verification (T-9.8.4 / matrix row 13); T-9.8.4's role-address routing remains unverified — Sprint 9.8 §7, Sprint 9.9 §7. -
The member's avatar has a versioned URL, cached for good (2026-10-16, #320 item 10; issue #209). Every authenticated page showed
<img src="/app/me/avatar">, revalidated on each view (private, no-cacheand an ETag), so each page view ran a second authentication and an image read. Migration 0085 addsusers.avatar_hash, a generated column holding the first 128 bits of the image's SHA-256. Pages now name/app/me/avatar/<hash>, read with one primary-key lookup (Store::user_avatar_version). That URL is servedprivate, max-age=31536000, immutable, so the browser fetches it once. A new upload is a new URL, and an old version or the bare URL redirects to the current one. The leaderboard's avatars keep their fixed URL and ETag. ADR-074 §3 is amended.- Signing out clears the browser's cache (#346, the owners' decision). Under
immutablethe browser no longer asks for the avatar, so a shared computer would keep showing it after sign-out. Every response that signs the browser out now sendsClear-Site-Data: "cache"with the clearing cookie:POST /logout(on the 303, which browsers honour),POST /api/auth/logout, the account deletion (POST /app/me/deleteandPOST /api/me/delete, which now also clears the cookie) and the email-change confirmation. The header names only the cache."cookies"would clear every subdomain's cookies, and nothing in browser storage is the member's (the launch figures are the same for everyone). ADR-074 §3 and DESIGN.md §9.7a record it.
- Signing out clears the browser's cache (#346, the owners' decision). Under
-
The HTTP/3 comments match ADR-026 (2026-10-14, review notes on #267). The workspace
Cargo.tomlblock andepic-edge's module docs still described an io_uring-driven,quinn-proto-only edge listener; since Sprint 2.3 both planes serve HTTP/3 throughquinn+h3-quinn+h3. The #265 entry under Security now names every path quinn-proto arrived through (ferrum-client/ferrum-protocolas well asquinn). Comments and changelog only. -
The rest of the "quinn-proto-only edge" wording follows ADR-026 (2026-10-14, Maran's review of #271).
epic-edge's listener module docs, its package description and its tokio comment (which listedlistener/h3.rsamong the runtime-free pipeline modules),epic-control'sCargo.tomlandmain.rs, and the Talos worker patch still described a hand-rolledquinn-protoedge listener. ADR-011 now carries a "superseded in part by ADR-026" status note, and ADR-024's listener bullet a dated annotation; both records are otherwise unchanged (append-only). The workspace note names all five ADR-026 invariants, io_uring portability included, and says the pipeline modules import no runtime crate beyondtokio::sync. Comments and records only. -
No more
cfg(loom)claim (2026-10-15, #302 item 1, Maran's review of #298).epic-edge's tokio comment said the pipeline modules compile under acfg(loom)swap-out forcrossbeamchannels; neither exists inepic-edge. It now points at the check that does run, the CLAUDE.md §3.7rginvariant. ADR-026's table, where the claim came from, gains a dated note: row 5's swap-out was never built, and row 4'srgreturns the one sanctionedtokio::sync::mpscimport, not 0. Comments and records only. -
Production Ferrum rolled
sprint-9.14→sprint-9.16, client crates 0.2.0 → 0.3.0 (2026-10-06, #212). Imageb2be29c4c0d2-sprint-9.16(sha256:7a44b43a…, Ferrum's own manifest digest, mirrored digest-preserving), rendered from Ferrum chart 0.4.0 (63b9994c). 9.15–9.16 are Ferrum's Kafka exactly-once rebuild, and EPIC uses none of it (kafkaTxn.enabledstaysfalse). The roll keeps the server inside the pairing Ferrum supports for the 0.3.0 client ("server ≥ sprint-9.16"). The client change is additive for EPIC, and no EPIC source changed. Before production moved, the six Ferrum-backed suites passed strict against the 9.16 binary (22/22), and the meter benches were A/B'd against 9.14 in one session because 9.16 rewroteCONSUMER.SUBunder the meter consumer: no regression, frozen as baseline tagferrum-9.16. A server-side dry run caught the chart's version bump being rejected, becausevolumeClaimTemplatescarried versioned labels and are immutable. A plainhelm upgradewould also have left the new[kafka_txn]ConfigMap beside a 9.14 pod that rejects it. Ferrum fixed it in chart 0.4.0 (ferrum#119, after the missing image and chart bump in ferrum#117). Done as: clean shutdown snapshot, volume tarred to/data/backups/ferrum-9.16-roll/, one-timedelete statefulset --cascade=orphan,helm upgrade --rollback-on-failure(release revision 7). No data step: the AOF and snapshot formats are unchanged. Ferrum was down about 2 minutes, 13:06–13:08 UTC. Both edge pods, control and the aggregator reconnected in-process (ADR-126 reseed) with no pod restarts. Rollback ishelm rollbackto revision 6, chart and image together, because 9.14 rejects[kafka_txn]. Record:deploy/k8s/epic-state/ferrum-values.yaml. -
Dashboard copy and tracker fixes from the owners' site-text round 1 (2026-10-14, #278). The first round from the site editor: 14 texts reworded as written (the marketing slot pool, Usage, Models, Q&A); the free give-away bar shows what is left in green from the left and what is gone in red at the right-hand end, so it empties from the right as slots are given, with the subtitle "Anyone can win these slots!"; no tracker draws a marker line at its end; the Fair-share "?" pulses until opened, the standard for every help mark; the Project stages bar is a wave with the stages on its hilltops and in its dips. The round's robot mascots and stages drag editor follow in their own PRs; the Models box question is #276.
-
Production Ferrum rolled
sprint-9.11→sprint-9.14(2026-10-04). Ferrummain5f305866, imagesha256:79cc8c7f…, viahelm upgradewith the chart from the same commit — not an image-only roll: 9.14 rejects the old chart's config (unknown field audit) and refuses a pre-ADR-134 AOF, so the old AOF files were removed after a clean shutdown, with the volume tarred to/data/backups/ferrum-9.14-roll/first. Brings the fix for #156 (ferrum#2) plus the production fixes that mattered more: counters no longer double-applied on restart,EXTTLs no longer early, arena memory returned. Encryption at rest is live for the first time — the 9.11 chart rendered a key the server never read — so a Ferrum snapshot restore now needsmaster.key(disaster-recovery.md Part D). Both planes and the aggregator reseeded in-process, no pod restarts. Record:deploy/k8s/epic-state/ferrum-values.yaml. -
FerraCert pinned to 1.0.0, its first tagged release (2026-10-05, #137). The pin moves from the August
0.1.0build (c8f306f) tosha256:b937625a…(tagv1.0.0, mirrored digest-preserving from the lab registry). It bringsferracert deliverand theserveredelivery sweep (FerraCert ADR-029), so a certificate that issued against a stale delivery-key pin is recovered without spending a Let's Encrypt duplicate-certificate slot;stack-relocation.mdnow says so. Config is additive only and migration 009 applies at store open; FerraCert's kustomize base is unchanged since the prior pin. -
FerraVault crates 1.12.1 → 1.14.0, and the bearer-key rotation chain is now tested over an mTLS-only Watch (2026-10-05, #182). v1.14.0 changes no proto and no config; the client side needed no code change. FerraVault's published manifests require their sibling crates only at
^1.12.1, so all sixteen in the lockfile are moved together — updating just-client/-proto/-test-utilsresolves a 1.12.1ferravault-serveragainst a 1.14.0ferravault-proto, which does not compile. The newepic-ferron-clientsuiteintegration_bearer_rotation_mtls_watchdrives the edge's chain end to end against v1.14.0's mTLS test listener: a certificate-onlyVaultClientbound to the deployedepic-edge.tomlpolicy,spawn_watcher,spawn_rotation_watcher, two successive rotations observed in theBearerAuthholder with no restart; a negative control proves a policy withoutwatchdoes not rotate. The production fix is the FerraVault server roll:ferravault-values.yamland the backup CronJob are re-pinned to1.14.0(sha256:b2736fb1…) and1.14.0-debug(sha256:8047efc5…), mirrored digest-preserving from the lab registry; a render against the live release differs only in those two images. -
Ferrum client crates 0.1.0 → 0.2.0 (2026-10-05, #178). The registry's 0.1.0 (
afdb8d9d, 2026-08-18) predated every Sprint 9.14 client fix: its smart router chose a shard byslot % shard_countwhile the server places keys byxxh3 % shard_count, so about half of EPIC's commands took an intra-pod forward hop underthreads: 2, and three reply-desync fixes were unpublished. Ferrum published the 0.2.0 closure fromsprint-9.15-final(ferrum#113). No EPIC code change; the six Ferrum-backed suites pass strict (21/21) against a ≥ 9.14 server. Client 0.2.x needs server ≥ sprint-9.14, which production runs. -
Signup asks for credentials and consent only; the investment step asks for the organisation's profile (2026-09-14). ADR-066. The signup form loses its country, organisation-name and separate age fields; the Acceptable Use Policy box now reads "I am 18 or older and accept the EPIC Acceptable Use Policy", so one tick is both. Signup's sanctions check now screens the country Cloudflare resolved for the connection (
CF-IPCountry, andCF-Region-Codeso the list's occupied-region entries can match at all), believed only from a[client_ip]trusted proxy; a connection with no trusted country is refused under the new[boot.sanctions] unknown_country = "block"(declared in production) and accepted-but-recorded under"allow"(preview slots, viaEPIC_SANCTIONS_UNKNOWN_COUNTRY). What was screened goes into theuser.registeredaudit row./app/billing/checkoutgains a required country, an optional organisation name and a required 18+ box, enforced increate_checkoutfor the form andPOST /api/billing/checkoutalike: the declared country is validated and OFAC-screened, and the organisation updated andbilling.checkout.profile_confirmedaudited, before any payment provider is called.orgs.nameis optional (migration 0052,orgs_name_not_blank):Org::label()falls back to the billing email where an organisation must be identified (admin console, audit summaries, billing email, the slot certificate), profile pages show "(not set)", and the JSON org APIs returnnull. One normaliser (normalise_org_display_name) now governs names at registration, the investment step and rename. Wire:RegisterRequest.country_coderemoved (ignored if sent);CheckoutRequestrequirescountry_code+age_affirmed. Production signup now depends on the zone's IP Geolocation setting —cloudflare-zone.md§5b.
Security
-
The production edge no longer links the test-fixture crate (2026-10-25, #414).
epic-test-utils, and through it a whole in-process FerraVault server, is now optional behind a non-defaultdev-stackfeature. Only that feature compiles in the in-memory adapter stack thatdev.allow_in_memory_stack = trueboots. Without it, the edge refuses that config with a message saying how to rebuild. The production edge's dependency graph drops from 526 to 438 crates. Advisories in test fixtures no longer read as production advisories: RUSTSEC-2026-0387 reached the edge only this way. -
tokio 1.52.3 → 1.54.1 for RUSTSEC-2026-0386 (2026-10-23, #408). The advisory, published 2026-10-23, found that
tokio::io::unix::AsyncFd's safe constructors accept anyAsRawFd. Safe code can therefore register a closed or reused file descriptor, which violates I/O safety and can lead to a use-after-free in tokio's driver. EPIC's code never usesAsyncFd, but the advisory failedcargo denyon every PR Gate. Lockfile only. -
ctutils0.4.2 → 0.4.5 for RUSTSEC-2026-0385 (2026-10-23, #406). The advisory, published 2026-10-23, found thatctutils::Choice's mask generation and itsu128comparisons could compile to branches on secret values, so they were not reliably constant-time.ctutilsreaches the graph throughdigest0.11 →hmac0.13, including FerraVault's client crates. The patch bump is lockfile-only, and it turned every PR Gate's Stage 2 green again. -
Blocked email addresses and the restricted abuse record (2026-10-23, #365, #378 comment 6374; owners' decisions of the day before). An address EPIC blocks for a policy or terms breach stays blocked until an operator lifts it:
- What it blocks: sign-up (
register,/signup, the invitation sign-up), being invited (403 email_unavailable) and an email change, each checked where the request comes in. Sign-up answers a blocked address exactly as a taken one (409 conflict, one fixed message, which also stops the duplicate case passing Postgres's own message through), and an email change reads as one to a taken address, so neither reveals a block. Phase 10's federated sign-up calls the sameEmailBlockService::is_blocked. - Storage (migration 0093,
blocked_emails): a BLAKE3 keyed hash of the normalised address under a FerraVault key (epic/email-block-key-v1; ops step in the runbook), and the readable address for one year; an hourly sweep then clears it unless held, and the hash keeps blocking. Operator-only (FORCE-RLS deny-all). - The restricted abuse record (
abuse_records): kept when EPIC blocks or deletes an account for abuse: ids, email, keys, the abuse reports and kill-switch decisions about it and their audit row ids. One year from the decision unless an operator sets a legal hold; the account's erasure leaves it alone; operators only, every read audited. GDPR Art. 17(3)(b)/(e). - Console: Abuse triage → Blocked emails (block, find, lift, legal hold) and Abuse records (list, record, legal hold); "Block this email" and "keep an abuse record" on the member page's suspend and delete. Every change audited;
EpicAbuseRetentionFailing/EpicAbuseRetentionStalealerts. - A member's own deletion blocks nothing: the address signs up again once the account is erased.
DESIGN §10.6, §13.4 and §14.1 updated.
- What it blocks: sign-up (
-
An erasure no longer fails on an audit row, and "Keep my account" always wins (2026-10-23, #403). Two ways an account erasure could go wrong:
- An audit row naming the account or the person stopped the erasure, every hour, for good. The append-only trigger refused the
org_id → NULLthe account's deletion sets, and the erasure'sactor_user_id → NULL. A member who reached the investment step while the pool was sold out, then deleted their account, was never erased; an untouched own account carrying such a row could not be replaced by an invitation. Migration 0091 now letsaudit_log_immutableallow exactly an UPDATE that sets those columns to NULL, every other column unchanged. Neither is in the chain hash, so everyrow_hash, the chain and every signature still verify. Everything else is still refused. - "Keep my account" between the sweep's listing and the erasure could still end in an erasure. The sweep now erases through
Store::erase_user_if_due, which re-checks under the person's row lock thatdisabled_atis set and past the cutoff, and leaves the account as it is when it is not (counted askeptin the pass's report).
Tests against Postgres 18, as the application role: an erasure with audit rows naming the account and the person, the chain and signatures verifying after; an invitation replacing an own account named in the audit log; the trigger refusing every other change; a cancel between the listing and the erasure. DESIGN §10.5 and §14.1 and the erasure runbook updated.
- An audit row naming the account or the person stopped the erasure, every hour, for good. The append-only trigger refused the
-
An admin key works only for an EPIC operator (2026-10-23, #402, #384). Any slot user could mint an
admin-scoped key and use every/api/admin/*route, because the routes checked only the scope. With #374'ssuspend-userthat meant a lasting lockout of every operator.POST /api/keysrefuses theadminscope (403 forbidden) to anyone but an operator (users.is_epic_operator, ADR-049), and the web form offers it to operators only.- Every
/api/admin/*request reads the key's issuer from the database and refuses it (403) unless they are an operator whose account may sign in. Keys minted before, and an operator who lost the flag or was suspended, are refused on the next request. - The routes are listed once and mounted behind one route layer that runs the check before any extractor. A test sweeps every listed route by every method through the plane's router; a source scan fails on an
/api/adminpath registered outside the list. epic-migrate seed-orgflags the user it gives the bootstrap admin key as an operator, as ADR-049 already said it did. A deployment seeded before runsgrant-operatorfor that user once.- The
diagnosticscope gates no route today, so it is unchanged.
-
Account erasure runs, covers everything, and knows who asked (2026-10-22, #365; owners' decisions the same day). Members were told their account was deleted after 30 days, and it never was:
- The sweep runs:
ErasureSweepis spawned hourly inepic-control, audits each erasure (user.erased) and is counted (epic_control_erasure_sweep_*), withEpicErasureSweepFailing/EpicErasureSweepStalepromtool-tested (runbook). - It no longer skips sole Owners, and nothing blocks it: one transaction per account (
epic_store::erasure) handles every foreign key tousersandorgs, listed with its disposal and pinned by a test againstpg_constraint; erased as the application role, not the superuser. An Owner's account goes with them; its other members keep their sign-in, each getting a new account of their own with no slots if it was their only one (owners' decision on #378, same day). - One exception, for the owners to confirm: purchases and slot certificates stay, with the name, email and id replaced; the account row stays closed and nameless as their anchor.
- Who asked matters (migration 0091): a member who deleted their own account can sign in for 30 days, only to "Keep my account" (keys stay disabled); an account EPIC deleted (new console action and
POST /api/admin/delete-user) cannot sign in./app/me/deleteno longer promises a restore on written request. - Losing one's account never disables or erases the member: removal from one's only account now gives the member a new account of their own with no slots instead of disabling them (which the now-running sweep would have turned into an erasure). Migration 0091 re-enables the logins the old removal disabled and gives every login without an account one of its own; login does the same as a net.
403 no_groupis gone. - An untouched account may accept an invitation (owners' decision of 2026-10-22, #378 comment 6374): a login whose own account is exactly as it was made (no slots, purchases, keys, usage, other members, invitations, name or settings) joins the inviting account, and the empty one is deleted in the same transaction (
InviteStore::accept_invite). That covers a new sign-up as much as a member who lost their account. An account that holds anything still refuses the login with409 account_exists(ADR-069 §1), and a refused or mismatched acceptance no longer uses up the invitation. - EPIC's deletion sends its own email (same comment):
account_closed_by_epic, with the day it was closed, the day it is erased and where to appeal (the[domain]abuse and support mailboxes), body indeploy/postmark/. A member's own deletion still sendsgdpr_erase_scheduled. It has to be pasted into the Postmark console (#294).
DESIGN §9.2, §10.1 and §14.1 updated.
- The sweep runs:
-
Sign-in state that should end now ends (2026-10-22, #366; owners' decision the same day). Three ways it didn't:
- Sessions end on the server: 30 days after sign-in (the cookie's
Max-Age, now derived from the same value) or after 7 days unused, whichever comes first ([control] session_max_age_secs/session_idle_timeout_secs). Before, a copied cookie worked until someone revoked it. An hourly sweep deletes revoked and ended sessions, their refresh tokens and expired refresh tokens a day later (auth_session_purge, migration 0090;epic_control_session_sweep_*metrics). - A disabled account can't sign in: the login refuses
disabled_atwith403 account_disabled, and a session it still holds is revoked on its next request. - The user kill switch is a durable suspension: firing it records
users.kill_switched_at+kill_switched_reason(migration 0090) before revoking keys and sessions; the login refuses it with403 sign_in_suspended("sign-in", not "account": ADR-069 uses suspended account for an account-level hold that still lets the member sign in; owners' approval on #381), the session lookup signs it out, andis_firedreads the column once the 24 h tombstone is gone. An operator lifts it from the member's console page (/app/admin/orgs/{org}/members/{user}) orPOST /api/admin/unsuspend-user; both surfaces can now fire it too (suspend-user), which no route could before.
Both refusals come only after the password and the second factor verify, so a wrong password still answers like an unknown address. DESIGN §9.2, §10.1, §10.3 and §13.4 updated.
- Sessions end on the server: 30 days after sign-in (the cookie's
-
Every signed-in form and script post carries a token bound to the session (2026-10-21, #356 item 1; ADR-081). A state-changing request that carries the
epic_sessioncookie must carryBLAKE3-derive_key(context, cookie), as a hidden_form_tokenfield or theX-Epic-Form-Tokenheader, orepic-controlrefuses it with403 form_expired(logged, counted inepic_control_form_token_refusals_total). A tab rendered for one account can no longer post as the account that signed in after it, with or without JavaScript, and a same-site subdomain can no longer forge a request on the cookie. One layer inrouter_with_middleware, before routing, stateless, compared in constant time. Exempt by exact path: the pre-login forms and their JSON equivalents (/api/auth/{register,login,password-reset/*,verify-email,verify-email/resend},/api/abuse/report), which render no token; and/logout, so "Sign out" works from any tab, however old (owners' decision on the #372 review;SameSite=Laxkeeps a cross-site sign-out cookie-less), which is why no sign-out form carries a token, #354's on/loginand/signupincluded. The avatar upload's body is never read by the layer, which runs before authentication: the handler checks the token in it after authenticating the member and taking an upload slot and the account's claim, with the layer's own check, so the #197 bound on upload memory holds (proved through the production tower and over HTTP/3). The refusal page says what happened in plain words: the page was opened under an earlier sign-in (the browser signed in again, or the password was changed, since), nothing was saved, reload it; and a reader who sent nothing may have had another site try to. Every signed-in<form method="post">and the three scripts that post (layout.js,menu_editor.js, the playground's key mint) carry it. Router sweeps cover 64 dashboard routes, 84 console routes, 24 other cookie routes, a path no route serves and every exempt path; a source gate, quote- and expression-aware, covers every POST form and refuses a token rendered only under a condition. The gatedplayground_browser_e2enow runs through the production tower, with its stale#pg-outputselector fixed. -
The edge role's check sees memberships, and the grants apply in one transaction (2026-10-17, #340 items 1, 2 and 4; follow-ups to #330). Because
epic-edge-rois NOINHERIT, the privilege checks behindedge-grants.allowignored role memberships: aGRANT epic_admin TO "epic-edge-ro", after whichSET LOCAL ROLE epic_adminreadsapi_keys, or apg_read_all_datamembership, still matched the list.- The check now also reads the role's attributes, every role it is a member of (directly or not), and everything
pg_shdependrecords against it beyond the grants already listed: objects it owns, large objects included, policies naming it, default privileges, and grants in other schemas or databases. A new test makes twelve such changes and checks each one is reported. - In production CNPG sets the attributes and memberships from
postgres-cluster.yaml, which that test never reads. A new Stage 1 gate (ci/postgres-roles/) fails if the role's entry there declaresinRoles, any attribute butlogin, or leaves outinherit: false(CNPG defaults it to true). - The migrate Job ran
postgres-roles.sqlstatement by statement, so a failure after the edge section's REVOKE ALL left the edge with no grants: an edge-wide auth outage. It now runs psql with--single-transaction, and the file opens withSET LOCAL lock_timeout = '10s'.preview.pywraps its cut of the file in its own transaction, since a COMMIT in the file would be cut off. - The gate also requires
--single-transactionandON_ERROR_STOP=1on the Job, and keeps transaction control and psql meta-commands out of the file. The file's header now says so; it had claimed the DO blocks only grant (item 4). - Rolling back means re-applying the previous file the same way:
docs/operations/migration-runner.md§3. - #350 review. The check now reports a privilege the role may pass on (
… WITH GRANT OPTION), MAINTAIN (PG17+; it lets a roleLOCK TABLE … ACCESS EXCLUSIVE, which would stall auth on both planes) and everyALTER ROLE … SETfor it (settinglines). A dependency row is now hidden only when every privilege in it is one the lines report, judged fromaclexplode, so a privilege type the query does not know surfaces as adependsline rather than vanishing. Five new drift cases cover them. The query moved intodeploy/k8s/epic-state/edge-effective-privileges.sql, which the test loads withinclude_str!and the runbook (§2) runs on the cluster with psql's\bind, so both check the same text. The gate also checks the role as each substrate profile renders it (kubectl kustomize deploy/k8s/profiles/*), and refuses arole,dependsorsettingline inedge-grants.allow, which would otherwise let one pass beside the grant that causes it. Its SQL scanner moved intoci/postgres-roles/sql_scan.py.
- The check now also reads the role's attributes, every role it is a member of (directly or not), and everything
-
A new account confirms its email with a code before it can log in, and a browser holds one account at a time (PR #354; Melvyn's request of 2026-10-20; the multi-tab report in #356). Sign-up already sent a 24-hour confirmation link, but nothing asked for it: an account logged in whether or not its owner held the address.
- Confirmation by code (owners' decision 2026-10-22, #354 review). The mail carries a six-digit code, not a link: a link confirmed whoever opened it, so someone who registered another person's address with their own password could have that person confirm the account for them, and ADR-068 §6.3 auto-links federated sign-ins on a verified email. The code (CSPRNG, stored only as a keyed BLAKE3 hash bound to the account, compared in constant time) works for 24 hours, and only together with a pending-confirmation token that EPIC gives the browser that just set the password (sign-up) or proved it (a login, with any second factor, refused only for the unconfirmed email): 32 random bytes, stored hashed, one per account, one hour, in a
__Host-HttpOnly SameSite=Lax cookie on the site and aspending_confirmation_tokenin the JSON API. Each try is counted before it is compared, atomically, and the fifth wrong one voids the code; "Send a new code" (/verify-email/resend, which now needs the token) replaces it. An account is mailed at most three codes an hour and ten a day, sign-up's included (owners' decision 2026-10-23); past either, the answer is429with when the next can be sent. With five tries a code that holds someone who has the password but not the mailbox to 50 guesses a day, about 0.15 % over a month, while the address's owner receives every code. Code entry also has its own per-IP band. A password change ends the token. Migration 0092 addsemail_verification_codesandemail_verification_pendingand drops 0024's link table. Sign-up and an unconfirmed login go straight to/verify-email, the code page; the?token=link, the address-taking resend form and both check-email pages are gone. The Postmarkverify_emailtemplate indeploy/postmark/renders{{code}}and{{valid_hours}}and must replace the one in Postmark when this deploys. - Confirmation before login.
POST /api/auth/loginrefuses an unconfirmed account with403 email_not_verified, only after the password and any second factor, so a wrong password still gets the sameunauthorizedas an unknown address. Only new logins are gated: an existing session keeps working. A completed password reset does not count as confirmation. There is no grandfathering and no migration of the flag: an existing account that never confirmed logs in, gets the code page and confirms like any other, soemail_verifiedkeeps meaning that the owner of the address proved it from the browser that holds the password. The owners confirm their own accounts the same way; preview accounts are already confirmed byepic-migrate ensure-accounts. - One account per browser. A browser holds one session cookie for all its tabs, so a second login replaced it under the tabs still showing the first account, and their forms acted for the new one (Melvyn saw the interest totals mix across tabs, #356).
loginandregisternow read the request's own session first and refuse409 already_signed_inbefore anything about the account asked for is looked up; a session lookup that fails is503, never "signed out"./loginand/signupshow which account the browser is signed in to, with a sign-out button, and logging in again as that same account goes on to the dashboard with no second session. Every dashboard and console page carries its account (data-account, only when there is one) and a small guard script (account_guard.js, CSP-hashed) that records it inlocalStorage: a tab whose account another tab signs out of or replaces is blocked behind a modal that offers only "Reload". The login and sign-up forms, shown only when the server found no session, remove that record (account_signed_out.js, CSP-hashed), so a session that ended without a sign-out (expiry, a deleted account) leaves none on a shared machine; a failed session lookup still shows the form but leaves the record, so an outage never blocks the other tabs. - No cross-site pre-login posts (#354 review). Login, sign-up, the code and the resend (web forms and JSON API alike) refuse a request whose
Sec-Fetch-Siteis notsame-originornone, or whoseOriginis not the site's own, with403 cross_site_request, before anything is looked up; a request with neither header (the CLI) passes. Without it a page elsewhere could post the attacker's own credentials and sign a visitor's browser in to the attacker's account (login CSRF): that post carries no session cookie, so the one-account rule saw a signed-out browser. "No session is issued" now holds cross-site too. The site's origin is[domain]'s app origin, or[control] public_originwhere a deployment overrides it (a preview, a local run). - Tests: the code service (issue, confirm, five wrong tries void it, the code's day and the token's hour, a resend replaces the code with fresh tries, no token, one account's token with another's code, a password change ends the token); the handlers and pages, both ways in, through the whole router with Postmark mocked so the code entered is the one mailed; anti-enumeration unchanged; every cross-site case on every pre-login POST, with nothing created, issued or mailed; a real session-row and account-row lookup failure answering
503on login and sign-up with a cookie present; the daily cap reached and its window rolling over, with sign-up's code counted and logins counting nothing; Postgres under theNOSUPERUSER NOBYPASSRLSrole (the lifecycle, 24 concurrent tries getting exactly five, the sweep, an email change, the edge role seeing neither table); the h2 end-to-end suite confirms with the mailed code; and the earlier one-account tests, including the two-tab Firefox run indashboard_browser_e2e. The visual set swapssignup_check_emailforverify_email(the code page's "log in to continue" state), which needs a baseline recorded.
- Confirmation by code (owners' decision 2026-10-22, #354 review). The mail carries a six-digit code, not a link: a link confirmed whoever opened it, so someone who registered another person's address with their own password could have that person confirm the account for them, and ADR-068 §6.3 auto-links federated sign-ins on a verified email. The code (CSPRNG, stored only as a keyed BLAKE3 hash bound to the account, compared in constant time) works for 24 hours, and only together with a pending-confirmation token that EPIC gives the browser that just set the password (sign-up) or proved it (a login, with any second factor, refused only for the unconfirmed email): 32 random bytes, stored hashed, one per account, one hour, in a
-
rustls-webpki 0.103.16, webpki-roots and webpki-root-certs 1.0.10: no trust in the distrusted emSign intermediates (2026-10-21; RUSTSEC-2026-0381/0382/0383, CVE-2026-111943). The private keys of several emSign intermediates are now publicly known.
rustls-webpki0.103.15 could accept a certificate path through them (0382), andwebpki-roots1.0.7 /webpki-root-certs1.0.9 bundledemSign ECC Root CA - G3, which reaches them (0381, 0383).Cargo.lockmoves those three crates and nothing else. All three are transitive, throughrustls,hyper-rustls,reqwestandsqlx, so this is lockfile-only likecryptokiin #263, andcargo deny/cargo auditkeep any lockfile from resolving back. Found whencargo denyfailedci.shStage 2 on #350's gate; it fails onmainand every open PR from the advisories' publication. -
The checkout hands merchants absolute return URLs only (2026-10-16, #320 item 12; #315 review). A return path such as the form's
/app/me#billingis now resolved against the public origin, so a merchant only ever receives an absolute URL on this site and can never resolve a relative one against its own domain. With no public origin configured the checkout stops with a 503 before any slot is held. An explicit[control] public_originoredge_public_originloses a trailing/when it is resolved: kept, it refused this site's own return URLs and made email links such as{origin}/verify-emailstart with//. New cases pin an uppercase host, a trailing dot, an explicit default port and an IPv6 origin with a port as refused, as they already were. -
The edge's database role holds an explicit allow-list, checked in CI (2026-10-16, #320 item 8; #242 item 7).
epic-edge-rohad SELECT on every table, default privileges on every future one, every sequence and two INSERTs, so a stolen edge credential could readabuse_reports,waitlist_signups,playground_access_codesand every other member's data. Through grants that migrations make to PUBLIC, it could also run every SECURITY DEFINER wrapper, includingauth_session_insert: a session for any member.postgres-roles.sqlnow grants the edge only what its code runs. That is SELECT on the columns of the five tables it reads and the API-key lookup function; it writes nothing. The file also gives each other callable definer function to the control role andepic_adminand revokes it from PUBLIC, and revokes PUBLIC'sTEMPORARY.edge-grants.allowpins the role's complete effective privileges, and two tests inintegration_postgres_rolesfail if they drift or if any query the edge makes no longer works under the role. -
A recovery code is spent only with the change it authorises (2026-10-16, #320 item 5; issue #205). The password change, the email change and the two-factor disable each removed the code first and made their change after, so a failure in between (hashing the new password, a database error) spent the code for nothing.
AuthFlowStore::spend_recovery_codenow removes the code with a compare-and-set and makes the change in the same transaction: a change that fails rolls the spend back, and of two requests presenting one code only one makes its change (8 concurrent spends against Postgres: exactly one wins). An email change whose mail cannot be sent is withdrawn together with giving its code back, but only if the withdrawal removed that very request, and only while MFA is still on with the same secret, so a re-enrolment never revives an old code; this retires the residual ADR-075 accepted. The withdrawal (AuthFlowStore::withdraw_email_change) is keyed on the request's confirm-token hash, with or without a recovery code: a send can report failure after the mail was accepted, so the link may already have been used, or another tab may have replaced the request, and then nothing is deleted, nothing is given back, and the member is told to check the Account tab rather than that nothing changed (review of #322). Each withdrawal is audited asuser.email_change_withdrawnwithrecovery_code_restored, anduser.email_change_requestedis written after the spend, so a request that lost its code to another never reads as requested (#337).RecoveryCodeUse::RequestEmailChangecarries no user of its own, so a change can only be recorded for the user spending the code.consume_recovery_codeis gone, so no caller can spend a code apart from its change. -
Passwords: 15 to 256 characters, and never a known breached or common one (2026-10-16, #320 item 1). NIST SP 800-63B-4 requires at least 15 characters where a password is the only factor, and a check against known compromised and common passwords.
epic_core::password_policynow holds both: the length in Unicode code points after NFC, an embedded blocklist of 255,874 entries of 15 or more characters (SecLists' xato, Pwdb and NCSC lists, folded and sorted, searched in place with no index), and NIST's context check against the account's email and the site's domain. Register, reset and the Account tab's change call it, and each refusal carries its own stable code and a reason. Sign-in never does, so existing passwords keep working, and the login form drops itsminlength. A reset now reads its token without spending it (AuthFlowStore::find_password_reset_token), so a refused password keeps the link working. The sign-up form's hard-coded 12–128 now reads the policy, like every other form. -
The operator console's routes are gated and tested from one list (2026-10-16, #320 item 4; issue #147). Every
/app/admin/*handler calledrequire_operator_sessionitself, after its form and query extractors, so a signed-out POST reached a form extractor before any check. The tests that proved each route gated read hand-kept lists, which had drifted twice: the waitlist routes, then the questions email route. Each console section now lists its routes asAdminRoutes, andadmin::routermounts exactlyall_admin_routes()behind one route layer that runs the gate before any handler or extractor. The authorisation tests sweep the same list by every method, through the plane's whole router, signed out (303 to sign in) and as a non-operator (403), so a new route is gated and tested by construction, and a handler another router registers on a listed path fails the sweep. The routes gate (ci/routes/verify.py) holds the console to that shape (#342 item 1).all_admin_routesis a declared collector: its calls mount the sections only once it is itself reached from the root router, so a section left out of it fails as unmounted, and every other listing still mounts nothing. A path under/app/adminregistered any other way than as anAdminRoutein a section underroutes/web/admin/fails, and so do a.nest(…)whose prefix covers the console, a section's routes mounted other than through the collector, and anything merged, nested or routed after the gate's.route_layer(…), which would be served ungated. -
The checkout's return URLs must be on this site (2026-10-16, issue #149).
POST /api/billing/checkoutpassedsuccess_urlandcancel_urlto the merchant unchecked, so an owner or admin could make the merchant redirect to any address after payment. Each must now be a path on this site or an absolute URL on its public origin; anything else is a 400 before any slot is held. The path rule is the currency form's (is_safe_return_path: no//,/\, control characters or over-long values). Low severity: the caller must already own or administer the organisation, and no merchant is live yet (tracker L1). -
A plane that cannot reach Postgres at boot no longer prints the database password (2026-10-14, issue #275, found investigating #91). Both planes put the full DSN into their connect error's context, so the
epic-controlcrash of 2026-10-12 wrote the live password into its pod log. Oneepic_config::sections::redact_dsnnow masks the userinfo password and a libpq?password=parameter on both planes, and withholds an unparseable DSN whole. The exposed password still needs an operator rotation — see #275. A test on each plane drives the real connect path with a password in the DSN and asserts the printed error chain does not contain it. -
rustls 0.23.46: four advisories published 2026-10-14 (2026-10-15, #300; RUSTSEC-2026-0346/0347/0348/0349).
Debugon client session values printed TLS secrets (0346); buffered application data went to the outer server after an ECH rejection (0347); Brotli certificate decompression could allocate up to 1 GiB before authentication (0348); an unauthenticated ServerHello could downgrade later connections to classical key exchange (0349).Cargo.lockmoves rustls 0.23.45 → 0.23.46 and nothing else; the workspace entry rises from0.23to0.23.46. Unlike quinn-proto before #265, this floor binds: seven member crates inheritrustls = { workspace = true }, so no lockfile can resolve below it.cargo audithad failedci.shStage 2 onmainand every open PR since the advisories were published. -
The quinn-proto 0.11.19 floor is now enforced (2026-10-13, issue #265). The workspace entry #258 added bound nothing: no member crate inherited it, and quinn-proto arrived only transitively — through
quinn0.11.9 (which accepts 0.11.12+) andferrum-client/ferrum-protocol0.3.0 (^0.11) — so only the lockfile andcargo denyheld the line against RUSTSEC-2026-0338…0342.epic-edgeandepic-control, which run the HTTP/3 listeners, now declarequinn-proto = { workspace = true }for the floor alone, and acargo updatecannot resolve below it. Also from Maran's review: the two copies of the kube read timeout cross-reference each other, and theunsafe-libyamlnote says what RUSTSEC-2024-0449 records. -
A TOTP code is accepted once (2026-10-13, issue #239; RFC 6238 §5.2). A code valid within its ±1-step window could be used again for up to 90 seconds — by anyone who saw it over a shoulder, relayed it from a phishing page or read it from a log — at sign-in and at every step-up a code guards (change password, change email, disable MFA, owner transfer).
TotpService::verifynow says which step a code matched (still constant-time over the three candidates), and the newTotpService::verify_oncespends that step throughAuthFlowStore::accept_totp_step: one compare-and-set on the newusers.totp_last_step(migration 0080) that records it only if it is later than the last step accepted, so of two concurrent submissions of one code exactly one passes. A replay isAuthError::TotpReplayed, answered on the wire exactly as a wrong code. A new secret or disabling MFA clears the history. All five verification sites (login, enrolment confirm, disable, the shared web step-up, owner transfer) use it. Tests:verify_onceunit tests (replay, steps only forward, a wrong code spends nothing, new secret, backend failure), a Postgres suite under the app role (spent once, forward only, resets, 20 rounds of a concurrent double submit), the edge's hidden user columns, and login and step-up replay tests. -
quinn-proto 0.11.15 → 0.11.19: five remote denial-of-service flaws in both planes' HTTP/3 listeners (2026-10-13; RUSTSEC-2026-0338/0339/0340/0341/0342). A peer could grow the receiver's memory without bound (empty DATAGRAM frames past the receive-buffer limit; gapped STREAM frames past the reassembly cap; repeated NEW_CONNECTION_ID frames past the retirement limit) or panic it (zero-length STREAM frames in unordered reads; a min/max ACK delay inversion). Found by the PR Gate's newer advisory database; the workspace floor is now 0.11.19 so no lockfile resolves below it.
quinnstays at 0.11.9 — the FerraVault test server pins it exactly — and takes the new quinn-proto unchanged. -
kube 3.1 → 4.2 and k8s-openapi 0.27 → 0.28: no YAML parser written in unsafe C-translated Rust (2026-10-13).
cargo denybegan failing Stage 2 on every branch,mainincluded, with RUSTSEC-2024-0449:unsafe-libyaml(a c2rust translation of libyaml, archived, no fixes) came in throughkube-client3 →serde_yaml(deprecated), which reads a kubeconfig. kube 4 reads it withserde-saphyr, which forbidsunsafe, and the two crates leave the lockfile.k8s-openapitakesv1_36, the lab API server's own version, in place ofv1_35within the skew. One behaviour change in kube 4 is undone on purpose: it dropped the client's 295 s read timeout (for exec and port-forward sessions, which EPIC never opens), soepic-clusterandepic-secret-syncset it themselves —epic_cluster::kube_clientandepic_secret_sync::sink::with_read_timeout— and a stalled API server still fails a call instead of hanging the reconciler; a test in each pins it. No code beyond that changed: both crates compiled unchanged against kube 4. -
The sign-in email changes only through a verified flow;
PATCH /api/meno longer changes it (2026-10-08, ADR-075).PATCH /api/meset a new email at once on nothing but a session, so a stolen session cookie could move an account to its thief's address and then reset the password from it. It now refuses an email change withemail_change_requires_confirmationbefore writing any field. The change takes the current password and the MFA step-up (shared with Change Password, including one rate allowance for both forms), a single-use 24-hour link to the new address (only its hash stored, migration 0064), a confirm button on the page the link opens (a mail scanner opening the link changes nothing), and a notice to the old address; confirming signs the member out everywhere. An address another account holds gets the same answer and no mail. The notice to the old address carries a cancel link that needs no password, the Account tab shows a pending change with a Cancel button, and any password or MFA change cancels a pending change — otherwise "change your password" would not have stopped an attacker who already held the link (found by an independent review before merge). Confirming also kills every reset and verification link mailed for the old address, and a disabled account cannot confirm. Two Postmark templates (deploy/postmark/email_change_*) must be added to the console before the form is used in production. -
The control plane's HTTP/3 listener holds a bounded amount of unread request data (2026-10-07, PR #225). quinn's defaults allowed 1.25 MB per stream and no limit per connection — about 125 MB of unread data on one connection with 100 streams open. The QUIC receive windows are now 1 MiB per stream, hyper's HTTP/2 default, and 4 MiB per connection — larger than HTTP/2's 1 MiB because QUIC flow-controls a request's headers, so with the two equal a single upload the app is not reading would stop every other request on the connection from starting. Ordinary requests never meet either window, and a large upload is paced by the app's own reading as over HTTP/2. Three tests over real QUIC pin the stream window, the connection window and the room left for the next request; each window moved in either direction, the setters swapped, or quinn's defaults restored fails at least one.
-
The control plane's HTTP/3 bridge streams the request body into the app instead of collecting it first (#198, 2026-10-07). It used to read the whole body into memory before dispatching to axum — so over HTTP/3, on any path and before authentication, a request body was bounded by nothing but the sender's patience. The body now streams in as QUIC delivers it, capped at the path's limit (64 KiB, 15 MiB for the avatar upload), exactly as on HTTP/2: authentication, the rate bands and the avatar upload slots all run before any of it is held, and the peer is told to stop sending once the answer is out. Tested against the production
serve_h3over real QUIC, including an unauthenticated upload and one with no free slot, each answered while the body is still arriving. -
cryptokipatched for RUSTSEC-2026-0286 (2026-09-17). An out-of-bounds read:cryptoki0.12.0 treated theCKA_ALLOWED_MECHANISMSulValueLenbyte count as an element count, so a valid nonempty attribute read back through the safeSession::get_attributescould build an out-of-bounds slice. Caught by the SP.3 gate sweep, which was failing Stage 2 on every branch includingmain— the reason PRs #129 and #134 were red with no defect of their own. Lockfile only,0.12.0 → 0.12.1, two lines, no other crate moved. It reaches EPIC transitively and only through test fixtures (cryptoki → ferravault-server → ferravault-test-utils → epic-test-utils → epic-edge), and nothing in EPIC opens a PKCS#11 session, so the vulnerable path is linked but never called. The last edge of that chain is a[dependencies]edge, not[dev-dependencies]— so the fixture crate compiles into the production edge binary. That is pre-existing (Sprint 2.3) and is recorded with a concrete remedy as Q77 of the architecture review rather than reshaped inside a sign-off sprint. -
The website's login, signup and password-reset forms are rate limited (2026-09-14). The DESIGN.md §9.7 bands were matched on
/api/auth/*only, but/login,/signup,/forgot-passwordand/reset-passwordpost to server-side handlers that call those API handlers in process — so the middleware saw only the form path, matched nothing, and the forms the site actually offers members had no anti-abuse band at all. Password guessing through/loginwas bounded by the Argon2id cost and Cloudflare's one-a-second volumetric rule, i.e. ~3,600 attempts an hour per address against a band of 40. Each form now shares the band of the endpoint it calls, so alternating surfaces does not double the allowance. Two related defects closed with it: the check ran for every method, so opening the public/abuseform five times in an hour locked a reader out of filing a report and a CORS preflight cost a login attempt (only submissions count now —GET/HEAD/OPTIONSpass through, matching Cloudflare's own rule); and a refused website form, or one refused because the limiter's backend was down (still fail-closed), gets a page with the wait and a fixed link back — or, for the landing page's join form, a new?joined=unavailablepopup — instead of the JSON envelope. Mutation-tested: un-listing/loginor removing the method check fails seven tests. -
The reaper's
namespaces: get, list, deletecluster-wide grant is gone (ADR-065). It was the most dangerous grant in Sprint 9.6 and could not be narrowed withresourceNames, because a preview's namespace name was derived from a branch and unknown ahead of time — solib.guard_namespacewas carrying the whole weight of bounding a process that deletes namespaces on a timer. The reaper now empties a slot rather than deleting a namespace, and every verb it holds is bound by name; itsClusterRoleis an empty rule set, kept rather than deleted so applying it to a cluster carrying the old binding replaces the rules instead of leaving them in force. The guard stays — the operator runs the same tool with cluster-admin, where RBAC refuses nothing — and its first condition is now membership of the slot set rather than a name prefix, which also refusesepic-preview-notes. Writing the tests for it found two real defects: a hand-editedpreview-slot: "0"raisedValueErrorrather thanNotAPreviewand so escaped every caller'sexcept, andint()accepted" 1 ","+1"and"1_0"as slot 1, meaning three label values nobody wrote satisfied "the label reproduces the name". -
OFAC monthly re-screen — 2026-09-10, no change to the six codes (out-of-band). The embedded seed was attested 2026-08-18;
seed_re_screen_is_not_overduetripped at 20 days on 2026-09-07 andmainhad been red since, withMAX_LIST_AGEdue to be crossed on 2026-09-22 — at which pointOfacGuard::checkfails closed on every country includingUSand blocks every signup. The tripwire did exactly what ADR-051 built it to do, and the failing gate is the deliverable rather than an inconvenience. Re-screened 2026-08-18 → 2026-09-10 against OFAC's Recent Actions feed and Sanctions Programs index: ten actions, every one an SDN designation/delisting or a licensing change, none altering the scope of a country- or territory-wide embargo — which is the only thing the file encodes.CUstands (2026-09-03 added six SDN entries and issued GL 4A; a general licence authorises conduct within an embargo and the CACR, 31 CFR 515, is untouched — the action tightens).IRstands (2026-09-08 designated 43 aviation entities, suspended GL J-1, issued wind-down GL DD; net tightening inside the ITSR, 31 CFR 560).KPstands, no action in window. The two judgement calls the file asks to be re-made each cycle were re-made: Kherson and Zaporizhzhia stay off — E.O.s 13660/14024/14065 authorise blocking persons who materially support the occupation, which is SDN screening rather than a territorial embargo, and neither oblast has been determined a Covered Region under E.O. 14065 — andSYstays off, its case strengthened by OFAC removing Syria's State Sponsor of Terrorism designation on 2026-08-24. Research assistant-performed against primary sources and countersigned by the operator, matching the provenance the 2026-08-18 cycle recorded. Merging this does not unbrick production: the list isinclude_str!'d into the binary, so the running pods carry their image's seed untilepic-controlis rebuilt and rolled. Automating the cycle remains tracker item L8 (OfacGuard::spawn_refresh). -
Sprint 9.7 — response security headers, on both planes (2026-08-26, Phase 9). The public surface served none: no
Content-Security-Policy, noStrict-Transport-Security, noX-Content-Type-Options, noReferrer-Policyand nothing preventing the site being framed — found while checking a claim in this sprint's own T-9.7.2 rationale that a CSP already existed. Both planes now stamp them from a layer mounted outside the panic catcher, which is the position that makes "no response leaves without them" a property rather than a description of the happy path: a panic converted to a 500, a timed-out request, a body over the cap and a rate-limit 429 are all responses no handler produced, and they are the ones easiest for an attacker to provoke. Pinned by a test that panics a route, and mutation-tested by moving the layer one position inward — it fails only that test, because every other response shape would be decorated either way. The document policy carries no'unsafe-inline'and no'unsafe-eval'and a test fails the build if it ever does: the inlined stylesheet and playground client are admitted by the SHA-256 of their content, chosen over a per-response nonce because every payload is aninclude_str!of a repository file, so its digest is a property of the build rather than of the request. That failure is silent — a hash that does not match is not a build error and not a server error, the browser drops the element and the page returns 200 with all its text and no styling — so it gets three gates: a source scan requiring every inline element inepic-webto interpolate a declared constant rather than a literal, a wire test checking one real response's inline blocks against that same response's own header, andvisual_regressionnow rendering through the production middleware tower, which makes the twenty existing pixel baselines a CSP gate at no extra cost.style-src-attris'none', which cost 42 inlinestyle=attributes (five declarations repeated across sixteen files) — style attributes are governed separately from style elements and there is no per-attribute hash, so one anywhere forces the exception for the whole document; proven visually inert, with 17 of 20 baselines at pixel-delta 0.00000 and the three that differ reproducing identically on stashed-clean HEAD. Turnstile was verified rather than trusted: Cloudflare documents onlyscript-srcandframe-src, and the failure mode if that is wrong is signup, so an env-gated browser test loads the realchallenges.cloudflare.comscript against Cloudflare's test sitekey under the real policy and asserts the widget produces a token — written as a controlled comparison (re-run with the policy removed) so "the policy blocked it" and "this machine cannot reach Cloudflare" are different outcomes. That design earned itself immediately: the first probe looked for an iframe, found none, and would have read as a CSP failure — the real cause was that Turnstile builds itself inside a shadow root, and a less careful test would have weakened the policy for nothing.epic-edgewas added to the task's scope (it was written control-only): it renders no documents so it takes the transport-level half, asserted on health probes, an auth failure, a 404, a CORS preflight and the streaming path, whose headers are written before the first token; values are shared inepic_core::security_headers, layers stay per-plane, ADR-026 re-checked at six header inserts per response with all six zero-alloc suites unchanged. HSTS carries nopreload— an operational decision, not a side effect of a constant. DESIGN.md gains §9.7a. See Sprint 9.7 §7. -
Sprint 9.3 — three client-IP trust defects, found by the IPv6 survey and closed (2026-08-23, Phase 9). None of the three is an IPv6 problem; the survey walked into them because making the address family a configured value meant reading every site that touches a client address. Recorded in full in ADR-060 §Context.
epic-edgehonouredX-Forwarded-Forfrom any caller. The edge is a DNS-only grey-cloud record (ADR-015 §2) — QUIC cannot transit an HTTP reverse proxy, so nothing sits in front of the data plane and the transport peer is the client — yetip_from_proxy_headerspreferred the header and never consulted the peer. A singleX-Forwarded-For: <address>therefore defeated both the per-API-key IP allow-list and the IP/CIDR kill-switch. The pipeline had documented the intended design all along ("IP allow-list runs in the route handler because it needsConnectInfo") and both listeners had been installingConnectInfosince Sprint 2.3 — nothing consumed it.epic-controlkeyed every DESIGN.md §9.7 anti-abuse band on a caller-supplied header.extract_client_ippreferredConnectInfoand fell back to the firstX-Forwarded-Forhop — but none of the control plane's three serve paths installedConnectInfo(notaxum::serve, notaxum_server::bind_rustls, not the h3 bridge), so the first branch could not fire in production and the unvalidated fallback ran on every request. Rotating a fabricated header bought a fresh bucket per request: effectively no rate limit on login brute-force, registration spam, password-reset flooding, TOTP-verify brute force or abuse-report flooding. The same unvalidated value also reached Turnstile'ssiteverify, theabuse_reports.reporter_ipevidence column, the hash-chained audit log, and the admin API key's IP allow-list check — whose127.0.0.1fallback additionally meant an admin key allow-listing loopback was usable from anywhere.- Login wrote a literal
127.0.0.1into every session row. A Sprint 1.1 placeholder ("the proper extraction lands in T-1.1.1 when the listener is wired") that survived eight phases, sosessions.ipcarried no information at all. Nothing renders that column today — there is no session-listing surface — so this was a security record that recorded nothing rather than a broken member-facing feature: the column an incident review reaches for to answer "where did this session come from" answered127.0.0.1for every session ever issued, and any future device-management surface would have been built on a constant.
Closed by one model on both planes (ADR-060 §4): the client address is resolved once, at the ingress boundary, under a configured
[client_ip]trusted-hop policy — the transport peer is the client unless the peer is inside a configured trusted-proxy CIDR, in which case the first parseable address from the first present configured forwarding header wins.epic-edgetrusts nothing (empty set, the safe default);epic-controltrusts the Cloudflare ranges and readsCF-Connecting-IP. All three control serve paths now installConnectInfo— two in the sprint, the h3 bridge in the post-close self-review below; v4-mapped addresses are normalised at that single point (usingto_ipv4_mapped, notto_ipv4, which would let::203.0.113.5impersonate203.0.113.5); and five hand-rolledX-Forwarded-Forparsers — which had already drifted apart — collapse into one typed extractor. Both planes fail the boot on a[client_ip]that does not compile, matching Sprint 9.2's[domain]posture, because a half-applied trust list is not a degraded service any probe would catch.No test failed when the edge's old behaviour was removed. That is the finding rather than an aside: it is the sprint's own premise — that fixtures which only ever exercise
127.0.0.1are what let this class hide — confirmed against the tree. The new gates were mutation-tested against the behaviour they replace (restoring the old resolver fails exactly the four bypass tests; removingset_only_v6fails the two listener tests), and that pass caught an inert UDP coexistence check and replaced it with a direct socket-option read. -
Sprint 9.3 post-close self-review — six further defects, four of them in the client-IP work itself (2026-08-23, Phase 9). Recorded rather than folded in, because the pattern is the finding: every one is a place where configuration could mean something other than what it said, which is the class ADR-060 §4 was written to eliminate. Closing the code path is not the same as closing the class. Full write-up in ADR-060 §4c–§4f.
- The control plane's h3 listener never installed the transport peer (§4c). The sprint fixed two of the three serve paths and missed the QUIC one, which computed
incoming.remote_address()for a log line and discarded it. The effect on HTTP/3 was worse than the defect being fixed: with no peer, resolution trusts no header and returnsUnresolved, so every h3 request fell to the"unknown"rate-limit identifier and shared one anti-abuse bucket across the whole transport, while no session row, Turnstile call or abuse report recorded an address at all. Fixed by threading the peer through, with abuild_axum_requestseam extracted so the property is unit-testable without a live QUIC connection (5 tests; 4 fail under mutation). - The deployed control plane declared no
[client_ip]at all (§4d). The policy was built, wired, tested — and pointed at nothing.epic-controlis orange-cloud proxied behind anexternalTrafficPolicy: LocalLB, so its peer is always a Cloudflare edge address and thedirect()default keys every §9.7 band on the proxy: all members behind one Cloudflare node share a login bucket, and the proxy's address is what reaches session rows, Turnstileremoteip, and admin keys' IP allow-lists. Pre-9.3 that plane read the firstX-Forwarded-Forhop with no trust check — spoofable, which is defect (2) above, but it did yield a distinct value per honest client. So the sprint closed a security hole and opened an availability one. Closed three ways: the section is now declared explicitly on both planes (empty-and-written on the edge — omitted means nobody asked, empty means someone answered); both planes log their posture at boot; and the newci/client-ip/verify.pyStage 1 gate fails the build when a plane's declaration stops matching its ADR-015 exposure. Cloudflare's ranges (15 v4 + 7 v6) retrieved from the published lists, with the refresh procedure indocs/operations/cloudflare-zone.md§5a. ClientIpPolicy::newrejected trust-with-no-header but accepted its mirror (§4e). Headers-with-no-trust is equally inert and more dangerous to leave un-rejected, because it fails safe — the headers are never read, which at run time is indistinguishable from a correct direct posture. Now symmetric.- RFC 7239's own
Forwardedheader could be configured but could never parse (§4f). Its elements arefor=192.0.2.43;proto=https, not bare addresses, soforwarded_headers = ["forwarded"]validated, deployed and then silently fell back to the peer forever. Now parsed, readingfor=and ignoringby=/host=/proto=—by=being an address that is emphatically not the client's. - The Ferron IPv6 fix replaced a transport bug with a deployment trap (§4h). T-9.3.10 correctly found that
tonicderives SNI fromUri::host(), which for a bracketed IPv6 authority includes the brackets — sohttps://127.0.0.1:8443configures TLS andhttps://[::1]:8443does not — and fixed it by adding a[ferron] tls_server_nameknob, required for any IPv6 endpoint, on the reasoning that "there is no name to derive from an address literal". The same test's IPv4 case disproves that:rustlsaccepts an IP literal as a server name and matches it against the certificate'sIP:SAN. The deployed edge dials Ferron at an IP literal (ADR-039) and Sprint 9.4 is the sprint that repoints it at a v6 address, with nothing in the manifest prompting the knob and an opaquetransport errorat boot as the symptom. EPIC now derives the name for every family (epic_core::net::tls_server_name_for_uri_host);tls_server_namesurvives as an override and is normalised identically. - The control plane held two client-IP trust policies, and one of them made a test lie (§4g).
RateLimitMiddlewarecarried its ownArc<ClientIpPolicy>and re-resolved the address separately from the ingress layer — initially as an optional builder step defaulting todirect(). Making it a required parameter was not enough, and mutation testing is what showed it: a new tower-level test asserting two members behind one Cloudflare node get independent buckets still passed when the tower's policy was swapped todirect(), because the limiter consulted its own copy. A test that appeared to exercise the composed stack exercised a private field. The duplicate is now removed rather than parameterised —rate_limit_anti_abusereads theResolvedClientIpthe ingress layer already resolved, so the plane has one trust policy and one resolution per request, and the limiter keeps only the IPv6 bucket prefix. The same mutation now fails, naming the symptom.
- The control plane's h3 listener never installed the transport peer (§4c). The sprint fixed two of the three serve paths and missed the QUIC one, which computed
Fixed
-
Two rustdoc links that turned
main's doc gate red after #371 (2026-10-23, #406).cargo doc --workspace -D warningsrefused a module-doc link to the privatedrops_event(epic-meter) and a link toFerrumStream::ensure_stream_configured, which exists only on #359 (epic-store); the first is now plain code, and the second namesConsumerSettings::dead_lettersandFerrumStream::ensure_consumer_with, which apply those settings onmain. No code change. -
Logging in, signing up and confirming an email work from
www,adminandapitoo (2026-10-23, #401). #354's same-origin rule on the pre-login POSTs comparedOriginwith the apex alone, but the control plane serves the same pages on thewww,adminandapihosts, with no redirect, and the session cookie's domain is the base, so a sign-in there is a real one. A member onwwwwas refused every login, sign-up, confirmation code and new code with403 cross_site_request. The rule now takes the site's own origins: the one the plane serves itself from and, when that is[domain]'s app origin, thewww,adminandapiorigins, all derived from[domain](AppState::site_origins). Each is compared whole, so a look-alike (https://www.<base>.evil.example,https://evil<base>), another scheme or port, andnullare still refused;Sec-Fetch-Siteis checked as before. A preview or a local run, which sets[control] public_origin, has only that origin. Tested on every pre-login POST, JSON and form, through the bare router and the production tower. -
A vote's quorum is measured against the weight that can be cast (2026-10-22, #385). Quorum compared turnout with Σ
orgs.seat_count, every slot active or on hold with any rights, while each ballot weighs only the account's active slots carrying the voting right (Org::vote_weight, ADR-069 decision 9). Slots on hold and complimentary slots without the vote inflated the denominator, so a vote every eligible account joined could still fail quorum.VoteStore::sum_eligible_seatsis nowsum_eligible_vote_weightand sumsslots_votingover the active founding-member accounts. ADR-014 §4 is amended. A Postgres test run as the application role pins the denominator to ΣOrg::vote_weight, and anepic-controltest closes a fully attended vote at a 100 % quorum. The fleet test that pins the shown fair-share weight to the edge's annotation now uses slots on hold and slots without the capacity right, so it fails if the page shows the slot count or the raw capacity count. -
The interest total updates live in every tab, and "since your last visit" is per account (2026-10-21, issue #356 items 3 and 4). The Interest box's "N slots of interest from M entries", its bar and the note past one server's slots were drawn once, so each open tab kept the total it had loaded. They now redraw from the
GET /app/launch/figuresanswer the scenes already ask for every 20 s (assets/scenes/counts.js, which now also holds the bar drawing the coin desks used). The likeliness chart, split and average stay as drawn until the next page load: the figures carry no breakdown. The figures' last-visit baseline was onelocalStorageentry per browser; it is now kept under the account's public id (data-accounton the Dashboard's<body>, from a typedIdentity.account: Option<UserPublicId>, left off when there is none), and the old browser-wide entries are removed unread. Because those entries say which accounts used the browser, they are kept only while it is signed in (#368 review, the owners' decision): a sign-out from a dashboard or console page removes every one as the form is sent (assets/figures_on_sign_out.js), and the login and sign-up forms when the server finds no live session, and the pages an email change and an account deletion end on, remove them at once (assets/figures_signed_out.js), so an expired session leaves nothing either from the next such page on. The cost: after a sign-out the next visit's "since your last visit" starts fresh. ADR-074 §3 (amended), DESIGN.md §9.7a andepic_control::cookielist what the browser keeps and when it goes;epic_web'sbrowser_storageholds both scripts. What the robots say (item 2) is unchanged, left for the robot-scene branch. -
Diagnostic captures and review samples are stored, travelling from the edge to the control plane over Ferrum (2026-10-21, issue #329; ADR-079). The production edge never set its capture or review-sample sink, so both fell back to no-op sinks: every capture an org consented to and every review sample an abuse opt-out required was dropped, while ADR-032 and ADR-037 said the Postgres stores were wired. Since #330 the edge cannot write those tables, so the edge now seals each record (
Vault.Wrap+ AES-256-GCM, the ADR-008 envelope unchanged) and publishes only the ciphertext toepic.diagnostic.captures/epic.review.samples(epic_protocol::sealed, versioned protobuf with no plaintext field, fuzzed). The control plane's consumer re-checks consent (or the opt-out) under a row lock, drops a record withdrawn in flight, inserts the restON CONFLICT (public_id) DO NOTHING, then ACKs; a Postgres outage leaves records in the stream for redelivery.- The sinks are required:
AppStateBuilder::newtakesContentSinks, so a build without them does not compile; tests and the dev stack sayContentSinks::discarding(), and a discarded record is counted. Every record is counted onepic_edge_content_records_total{kind,outcome}and a dropped one logged with public ids, never content. - The consumer (
epic-control-content-ingest,[control] run_content_ingest, on the singleton aggregator workload) has one stable name,max_deliver1024, and resumes a paused Ferrum consumer at boot, after a failed pass and on every idle pass, as #325 does for the meter;FerrumStreamkeeps #325's one consumer API, withensure_consumer_withfor these settings. Metrics:epic_control_content_ingest_{records,failures,consumer_resumes}_total. Runbook:docs/operations/content-ingest.md. - Tests: an end-to-end suite through the edge over real Postgres (deployed roles; the edge still holds no INSERT), Ferrum and an mTLS FerraVault with wrap-only and unwrap-only identities; the per-case real-service suite; the zero-allocation streaming suites with both tees firing (3 allocations per token, as without).
- #359 review: the tees' work is bounded per pod and kind (
[content_tees], 16 in flight, 10 s; over itdropped_backpressure, past itdropped_timeout); the stored expiry is never past capture + 7 d / sample + 90 d; a record whose key or user is not its org's is refused (org_mismatch); renewing consent while capture is on keeps the window, so captures from just before a renewal are no longer dropped; the edge holds a wrap-onlyKeyWrapperand the control plane an unwrap-onlyKeyUnwrapper; records Ferrum gives up on go to{<stream>}.dlq, are replayed each minute, counted (epic_control_content_ingest_dead_*) and alerted on (content-ingest-rules.yaml). - Maran's 2026-10-22 review of #359: the consent audit's
renewalflag is the store's answer (EnabledWindow), so a first enable is no longer called a renewal because Postgres keeps microseconds and the clock nanoseconds (F1); a record claiming a time more than five minutes ahead of the control plane's clock, or whose clamped expiry has passed, is refused, acknowledged and counted asrejected(IngestOutcome::{DatedAhead, ExpiredOnArrival}), and the streams keep six days plus one in their dead-letter streams (SEALED_DEAD_LETTER_SETTINGS), so ciphertext leaves Ferrum within a capture's seven days (F3); Ferrum counts no retention eviction, soepic_control_content_ingest_backlogandEpicContentIngestBacklogstand in, and steady edge drops (dropped_ferrumnow documented to include Ferrum refusing a publish past its flow-control stop) alert asEpicEdgeContentRecordsDropped(F2);dead_letters_waitingreads Ferrum's contiguous acknowledged position, pinned by a test that fails on the "highest acknowledged" reading (F4); a record of a newer version waits unacknowledged instead of being dropped as malformed, and the control plane rolls first (F5);org_mismatch,malformedandrejectedalert (EpicContentIngestRecordsRefused) and the consumer logs the record's org asclaimed_org(F6). ADR-079 carries a dated amendment. - Production stays fail-closed: FerraVault 1.15 lets only
sudoon**create an org's KEK, and the policies are not changed, so turning capture on and approving an opt-out still fail before anything is written, until ferravault#11 (the request). Deploy note: rollepic-control(the aggregator Deployment now setsEPIC_CONTROL_RUN_CONTENT_INGEST=true) andepic-edge, in either order: a record published before the consumer runs waits in its stream. No migration, no policy change.
- The sinks are required:
-
The landing page's logo was cut off under the bar at high zoom (owners, 2026-10-20). From 768px wide, any viewport too short for the hero — a laptop at 100 %, a desktop zoomed to 125–200 % — lost the top 32px of the logo: the hero lifted its content with a negative margin, which let the content out of the top of the box it was centred in, and the hero's
overflow: hiddencut it off. The lift is now padding moved from the hero's top to its bottom: the same picture where the screen has room, and never clipped where it has not. A browser gate,the_hero_logo_is_never_cut_off_or_covered, checks it at twenty zoom-level viewports. -
A help "?" stops pulsing only once it is opened on purpose (2026-10-17, #320 item 13; #306). #306 marked a "?" as seen whenever it opened, so passing the pointer over it, which opens it for a look and closes it again when the pointer leaves, stopped the pulse for good in that browser. It is now marked from the click on the "?": a click on a closed "?", the click that pins a look, or Enter or Space on it, which the browser delivers as a click. A look that ends when the pointer leaves, or with Escape or a click in the box, leaves it pulsing. The
toggleevent no longer marks anything: it says that the box opened, not why, and it arrives from a queued task, by which time a look may have been pinned or ended.dashboard_browser_e2ecovers hover then leave, hover then click, a click with no hover before it, and Enter and Space, each across a reload. With the oldtogglemarking, the hover-then-leave case fails. -
Every control-plane write goes through the cached store (2026-10-17, #320 item 3; follows #209 and #314). #314 moved the profile and avatar saves onto
AppState::cache_aware_store(), but many more writes to cached rows still went through the barePgStore, so theLayeredStorenever dropped its copy and a page reading the row through the cache showed the old one for up to the cache TTL (60 s by default). Not a security problem: the edge checks keys and orgs uncached. Moved onto the cached store: the/api/mename change and erasure (and the keys erasure disables); the/api/orgsrename, member removal (and the account it soft-deletes) and the other membership writes; the admin JSON kill-key, suspend and unsuspend; the operator console's suspend, unsuspend, close, kill-switch key and kill-switch org; ending a member's playground access; the checkout profile; every/api/keyswrite, the playground mint and the admin-key use stamp (theApiKeyServicewas built over the bare store); the kill switch's cascade (NCMEC quarantine and abuse auto-disable), which was built over aPgStoreof its own, so the DESIGN.md §13.4 claim that a firing invalidates the cache was not true until now; and the ephemeral-key reaper.Store::purge_expired_ephemeral_keysnow returns the deleted keys' public ids, soLayeredStoredrops their cached copies. The rule is now kept by the compiler: the bare store is a private field ofAppState, reachable only throughAppState::uncached_reads(), a newUncachedReadsview that has theStorereads and none of the writes. Reads that were uncached on purpose stay uncached: membership and role checks, an org's standing before an operator acts on it, and org rows, whose seat and right counts the database keeps from slot rows that no cache invalidation sees. Guards:compile_faildoctests (a write throughUncachedReads, and the field from outside the crate) and a test that fails if anything inepic-control/srcoutsidestate.rsreaches the field or builds its ownPgStore. Tests: each kind of write caches the row through a realLayeredStore, runs the handler and reads it back through the cache;LayeredStoregains one test per cached row type covering every write to it. -
Avatar and profile saves clear the cached account; two org-admin pages read the session once (2026-10-16, issue #209). The avatar upload and removal and the profile save wrote through the bare
PgStore, so theLayeredStoreinvalidations written for them never ran in production, and a cached account kept its old name andhas_avataruntil the cache TTL (60 s by default). They now write throughAppState::cache_aware_store(). The diagnostic-capture and abuse opt-out pages usedauthenticate()and then read the session and the primary org again in their admin check (the diagnostic POST three times); the check now uses the sessionauthenticate()resolved.MemSessionLookup::find_countlets tests pin one session read per request. -
The meter aggregator counts each event once, even after a crash (2026-10-16, #320 item 2; issue #185). A crash after the rollup committed and before the Ferrum ACK redelivered the batch, and the additive UPSERT counted it twice. Migration 0086 adds
meter_applied_events, a ledger keyed onMeterEvent.request_id.MeterAggregateStore::apply_meter_events(replacingupsert_meter_aggregates) records each event's id and adds only the events it had not recorded, in the same transaction. It is keyed on the event's own id rather than the stream offset, because Ferrum redelivers on a timer, after later offsets may have committed, and a stream reset renumbers offsets. An event with no usablerequest_idis dropped and counted as malformed.events_already_appliedon the aggregator health route counts the redeliveries absorbed.- #325 review: no replay counts twice. The aggregator's Ferrum consumer was named after its pod, so every roll created a new consumer, which reads the stream from its start: each roll re-counted the whole history (the production rollups are over-counted by every past roll) and left a dead consumer that Ferrum keeps for good and whose position its publish flow control waits on. The consumer is now one configured name,
[boot.meter_aggregator] consumer_name(defaultepic-meter-aggregator), withevent_horizon_days(default 30: an older event, judged by its UUIDv7request_id, is dropped and counted onevents_dropped_beyond_horizon) andinitial_deliver(all,neworby_sequence:<offset>, used only when the consumer is created), each with anEPIC_METER_AGGREGATOR_*override. The ledger keeps ids for the horizon plus a day and is pruned byrequest_idrange, 10 000 rows a tick; observers see only newly applied events; ledger and rollup rows are locked in one fixed order. - Ferrum pauses, and leftover consumers. Ferrum pauses a consumer whose pending records go un-ACKed past the stream's stuck threshold (150 s by default; a Postgres outage that long), and only
CONSUMER.RESUMEclears it; the aggregator now resumes it on an idle tick (consumer_resumes) and counts the other consumers on the stream hourly (stream_other_consumers).GET /api/admin/meter/consumersandPOST /api/admin/meter/consumers/{name}/delete(admin key, reason, audited; the aggregator's own consumer refused) are how an operator sees and removes them, since the Ferrum image has no CLI. - Auto-disable counts each reject at the time it happened, and ignores rejects older than its hour, so a backlog or a rebuild cannot disable a key.
- #325 review, questions 3 and 5: a failed lookup retries, a backlog drains, and a paused consumer reads on. A failed lookup of an event's organisation or user (a database error, not a missing row) fails the tick instead of dropping the event and ACKing it, so Ferrum redelivers the batch and the ledger counts it once; a missing row, or ids that do not parse, still drop the event. After a full batch the loop pulls again at once instead of waiting 60 s, so a backlog or a §4b rebuild drains at about 1 000 events a second (measured locally) instead of 1 000 a minute. And after resuming a consumer Ferrum had paused, it pulls at once too: Ferrum pauses it again within 500 ms unless the paused records are read and ACKed first, so with the 60 s wait the rollup never moved again after any outage over 150 s. The drop counters now count each event once, when its batch is ACKed. Maran's 2026-10-22 review: a failed lookup holds back only its own event (
events_held_lookup_failed; the tick fails withLookupsFailed), and the rest of the batch is counted, so one organisation row this image cannot read no longer stops everyone's metering. Ferrum gives an event up aftermax_deliverdeliveries, about one a minute under such a failure; owners' decision (Melvyn): the consumer is created withmax_deliver10 (Ferrum's default 5 dropped the batch after five minutes, the first proposal's 1 024 held the stall for 17 hours, past Ferrum's 16 384-record publish stop) and a dead-letter stream on the source's shard,{epic.meter.events}.dlq(METER_CONSUMER_SETTINGS,METER_DEAD_LETTER_SETTINGS: 30 days, 256 MiB), which keeps such an event rather than dropping it.epic-storegains #359's unifiedConsumerSettings(dead_letters: Option<StreamSettings>) and creates the dead-letter stream before the consumer. Ferrum fixes both at creation, so they ship before #325's switch-over createsepic-meter-aggregator; the replay of dead letters, the console'sdlq_total, alerts and a check of an existing consumer's settings follow the deploy. The ledger prune and the consumer audit now run on a failed tick too (a persistent failure skipped them), and a resume in the middle of a drain no longer logsaggregator caught up. Seedocs/operations/meter-aggregator.md§2a. - Deploy note: the first deploy needs the one-off switch-over in
docs/operations/meter-aggregator.md§4; with the defaultinitial_deliver = allthe new consumer would re-count the last 30 days. The aggregator therefore refuses to create its consumer atallwhile any counting inside the horizon was done without the ledger, saying what it found and the fix in the health route'slast_errorand retrying every tick, unless[boot.meter_aggregator] allow_full_replay(EPIC_METER_AGGREGATOR_ALLOW_FULL_REPLAY) is set. It reads that from the stream (other consumers over an empty ledger) and, since Maran's 2026-10-22 review, from Postgres: rollup hours inside the horizon that end before the ledger's oldest event, or another consumer's cursor row that counted inside the horizon before the ledger's first apply (MeterAggregateStore::{oldest_applied_meter_event, meter_rollup_hours, meter_cursors_ticked}, replacinghas_applied_meter_events). So deleting the old consumers before the switch-over, or re-creatingepic-meter-aggregatoratallwithin 30 days after it, is refused too; §4c re-seeds it, and the guard letsallthrough once the ledger spans the horizon.
- #325 review: no replay counts twice. The aggregator's Ferrum consumer was named after its pod, so every roll created a new consumer, which reads the stream from its start: each roll re-counted the whole history (the production rollups are over-counted by every past roll) and left a dead consumer that Ferrum keeps for good and whose position its publish flow control waits on. The consumer is now one configured name,
-
An API key's "Last used" is real (2026-10-16, #320 item 6; issue #148).
api_keys.last_used_athad no writer for inference keys, so the Account tab said "never" for every key, including keys in use. The meter aggregator now takes each key's latest event in a tick (MeterEvent.api_key_public_id,max(emitted_at)) and moveslast_used_atforward with one forward-onlyUPDATEasepic_admin(MeterAggregateStore::record_api_key_last_use). The edge pays nothing, a redelivered batch never moves a key back, and an event stamped in the future counts as the tick's own time. A key used before this deploy but not since still shows "never" until its next use. -
The menu mini bar can be dragged more than one place on Chromium (2026-10-16, #320 item 8; #242 item 8). The drag captured the pointer on the dragged tab, and moving that tab within the strip could release the capture on Chromium, which ended the drag after one step.
menu_editor.jsno longer captures; it listens for the move and the release on the document, as the dashboard boxes' drag (layout.js) does. A new browser step drags a tab left past two others and checks it lands two places earlier and is saved. On Chrome 155 it fails against the old code (one place) and passes now, and it passes on Firefox as part of the full suite.EPIC_CHROME_BINARYlets the suite launch a Chrome for Testing build.- A drag no longer swallows the next click (#326 review). The click that follows a release was marked by a flag that only that click, on reaching a tab, could clear. Without capture, a release off the strip sends its click to what the press and the release have in common, outside the strip, and Chromium sends none once the pressed tab has moved, so the flag stayed set and ate the member's next click on a tab. It now lasts for that one release (cleared by a 0 ms timer, which runs after the release's click and not before it, as a microtask would), and it is checked before looking for a tab.
- A drag that is not dropped puts the tabs back and saves nothing (#344 items 1 and 2). A cancel kept the half-done order and saved it. And with no capture to lose, nothing ended a drag whose release went elsewhere (a context menu opened mid-drag, a switch to another window): the tabs followed a pointer with no button held, and the next click anywhere saved where they had got to. A move with no button held, a fresh press of the same pointer, or the window losing focus now abandons the drag the way a cancel does. Restoring rather than keeping is deliberate: a release that never arrived is not a drop, and nobody chose the order the tabs were in when it went missing.
- The dashboard boxes had both defects, and get both fixes (
layout.js). There the stuck flag also ate a keyboard press of a grip, since the grip is a button and Enter and Space are clicks; a lost release could also leave the page scrolling under the box near an edge. A box drag that is cancelled already put the boxes back. - New browser steps: a click on a tab after the two-place drag, and after a drag released below the strip; a press of a grip after a box is dragged up (which moves the pressed grip itself); and each of the four ways a drag can end without a drop, for the tabs and for the boxes, checking the order is put back, nothing is left lifted and nothing is sent. Run on Firefox (the whole suite) and on Chrome for Testing (the drag steps).
-
A details save over another tab's is refused, not written over it (2026-10-16, #320 item 8; #242 item 6). The Account details and Profile forms compared against values read in the same request, so a save from a page opened before another tab saved silently overwrote it, despite the 409 ADR-075 promised. Both forms now carry the row's version as rendered (
UserRowVersion,users.updated_atto the microsecond) in a hiddenseenfield, andStore::update_user_details(id, seen, next)writes only while the row is still at it. Otherwise, or with no version, the member gets the 409 page saying to reload and make the change again, with the posted form's own tab (Account or Profile) selected.Store::user_row_versionreads the version uncached, and each page reads it before the values it shows rather than after them (#341 point 3): read last, a save landing between the reads paired the old values with the new version, and saving the page then wrote them back over that save. Any write to the account row moves the version, so an avatar or password change in another tab also asks for a reload; the owners chose that over a narrower details-only version (#327), and the in-memory store's version now includes the avatar so it moves in the same cases. The Profile card takes its avatar from the same read as its version. A page rendered before this deploy carries no version, so its first save after the roll asks for a reload. -
Follow-ups from the #241 re-review: names, caches and the browser checks (2026-10-16, issue #242 items 4, 5 and 9, plus nits).
PATCH /api/merefuses a display name with control or invisible formatting characters, as the Account tab's form already did, so a bidi override can no longer reach the rest of the group through the JSON API (item 4). The email-change confirm writes through the cachedStore, so the cached account is cleared with the change; it wrote through the barePgStore, and the comment saying otherwise was wrong (item 5).LayeredStore::update_orgclears the cached org around its write instead of writing the new org over it, so a read that raced the update cannot leave the old org cached. The hidden-formatting set gains U+00AD, U+034F, U+115F, U+1160, U+17B4, U+17B5, U+3164 and U+FFA0.ci/visual/run.shwith no pixel baselines skips only the pixel comparison:csp_browser_e2eanddashboard_browser_e2eneed none and now run (item 9). DESIGN.md'semail_change_tokensentry, ADR-075's "no sweeper" and the billing module's/app/billingreference brought up to date. Two paths that audit before an irreversible step now also audit a failed outcome, so the trail never ends at an intent that did not happen:user.email_change_failed(with the reason) andslot_interest.clear_failed. -
The SSR control gate requires every router to be mounted (2026-10-16, issue #170).
ci/routes/verify.pyproved a.route(…)was declared, not that its router was mounted: with the.merge(logout::router())line commented out,/logout404'd and the gate stayed green. A registration now counts only if its function is reachable fromstate::routerthrough.merge(…)/.nest(…)calls (module paths,useimports andpub use … as …re-exports resolved; a mount it cannot resolve fails rather than guesses), and a router nothing reaches fails on its own, including API routers no control links to. Routes declared only in test code no longer count. Mutation-checked on the real tree (logout, the billing API, avatar, epic-web's router, the kill-switch console) and by eight new gate tests. -
The launch animations, before release: the rest of the review (2026-10-15, issue #301 items 3–8, 10–20).
- Privacy and load. The GDPR export carries the animations switch. The figures route checks the session only, serves the figures from a shared 10 s cache, and leaves out paused boxes. The scenes stop polling once signed out, and poll only while a figure scene is on screen.
- Motion. Nothing in a scene moves under reduced motion. The scenes follow reduced motion as it changes. Switching animations off stops the same transitions reduced motion does, and a test keeps the two in step.
- Correctness. The appearance controls change the stored row under its lock, so concurrent changes all survive. Queued figure changes are acted out as one net change. Clerk's prices are grouped like every other figure. The give-away text names the same pool as its bar. The "?" stops pulsing once it has been opened.
- Size. The robots' stylesheet goes only to the signed-in pages (
site.cssis back to 128 KB on every page).scenes.rs,scenes.js,interest.rs,promo.rsandstages.rsare split under 500 lines each. - Docs. DESIGN.md and ADR-077 §5 document the deposit.
-
The launch animations, before release: the desk bar, migration 0084 and the console header (2026-10-15, issue #301 items 1, 2 and 9).
- The desk bar. The desks' bar now carries its figures as plain digits (
data-value,data-goal), and the count line is the figure and the rest of the sentence in spans of their own.scenes.jsredraws the bar from these asmeter::geometrydoes, past the goal too, rather than reading figures back out of the label. As the page opened with a sale since the member's last visit, the scene had added that sale to the figure the page already showed (2 sold, 1 seen: the bar went to 3). The bar is now set back to the last visit's figure as the page opens, and Strider pulls it on to today's. - Migration 0084. It adds the deposit column, lowers the deposit to a row's multi price where that is below 50, and only then adds the
deposit_aud <= multi_audCHECK, so a database with test prices no longer aborts the roll. - The console header. The header switches' forms are flex boxes, so the operator console's header keeps its height and its pages no longer sit a fraction of a pixel lower.
- The desk bar. The desks' bar now carries its figures as plain digits (
-
Password reset and login find an account whatever the capitalisation of its email (2026-10-14, issue #91).
find_user_by_emailcompared theCITEXTcolumn against atext-typed bind, which Postgres resolves astext = text: case-sensitive. A member who signed up asAda@…silently got no reset email asada@…, while signup'sUNIQUEindex (case-insensitive) told them the account existed. Every other email match already cast$1::citext;MemStorematched case-insensitively, so no in-memory test saw it. Real-Postgres regression test, mutation-checked. -
A control plane that boots while the vault is sealed now heals its Postmark token and its AAAA reconciler (2026-10-14, issue #222). Both read FerraVault once at boot and never retried: Postmark sent every email with an empty token and
/readystill said"postmark": true; DDNS never started. The F19 vault reconciler now carries them as pending boot secrets, fetched each pass until installed — the Postmark token into a lock-free slot every clone of the client shares, the Cloudflare token by starting the reconciler late./readyreports"postmark": falsewhile the token is empty, and every send in that window is refused before reaching Postmark (PostmarkError::TokenPending) rather than going out unauthenticated. Only a vault that is unavailable is waited out: a fetch refused asAccessDeniedorNotFoundis configuration, logged once and dropped — otherwise a preview slot, which mounts production's config but reads onlyepic-preview/**, would retry its denied Postmark read every 30 s for the life of the pod (#277 review). Tests pin the pending-secret decision, the spawn condition, the refusal path and the fail-closed send; each fails when its fix is reverted. -
The interest tracker's five-answer privacy floor counts only answers (2026-10-13, issue #242 item 1). The breakdown — the individual/team split, the average likeliness and the chart — is held back until
INTEREST_DETAIL_MIN_ENTRIES(5) members have answered, but the gate counted entries carried over from the old interest list, which state neither a kind nor a likeliness. With four carried over, the first member to answer opened the breakdown and it showed that one member's split and likeliness exactly. The gate now countsInterestView::answered_entries(individuals + teams); an empty tracker still shows its invitation. A test with four carried-over entries and one answer fails under the old gate. -
An affiliation code is taken only under READ COMMITTED (2026-10-14, Maran's review of #269; ADR-077 §8). Migration 0081's claim re-checks
retired_affiliate_codesafter waiting on the name's lock, and sees a retirement committed meanwhile only because READ COMMITTED takes a fresh snapshot per statement. Under REPEATABLE READ or SERIALIZABLE the snapshot predates the wait; serializable isolation does not catch this case either (one read/write edge, no cycle). Every writer runs at READ COMMITTED today; migration 0082 makes it a rule, refusing a write that takes a name — an INSERT, or an UPDATE setting a new, non-null code or personal code — at any other level (SQLSTATE 25000). Giving names up (leaving, an operator's removal, an erasure) still runs at any level, and READ UNCOMMITTED, which Postgres runs as READ COMMITTED, is accepted. A Postgres test, all as the application role, sees an UPDATE and an INSERT refused under both stricter levels (it fails without the guard), an operator's removal and a leave go through under REPEATABLE READ (it fails if removals are refused too), and the stores' own claims succeed. Also: "Email this member"'s docs now call aquestion.emailrecord an attempt, since a stopped process, a failed second record or a lost Postmark reply can each leave it standing alone. -
Two console actions no longer act before they are recorded (2026-10-13, issue #264, found reviewing #245; ADR-077 §4, §5). A slot-price save committed the prices, then wrote its audit record; "Email this member" sent, then recorded — so a failed record left the action standing unrecorded, against CLAUDE.md §8.2.
SlotPriceStore::set_slot_pricesnow takes the record's builder, andPgSlotPriceStoreappends it withPgAuditSink::emit_inin the price transaction (the sameemit_inas #261, identical so the two merge cleanly), so the change and its record commit together or not at all;epic-controlhands the store the deployment's signing sink. An email is recorded before it is sent and not sent if the record fails; a failed send addsquestion.email_failed. Tests: the console refuses a save whose record is refused and leaves the prices; a Postgres test revokes the app role's INSERT onaudit_logand sees the change roll back; the email test checks nothing goes out unrecorded and a failed send is recorded. -
A code being given up can no longer be claimed before it is retired (2026-10-13, issue #262, found reviewing #245; ADR-077 §8). Migration 0074's namespace trigger locked only the names a row was taking, and only on insert and update. A member leaving the programme, an operator removing a personal code, or an account erased gave its names up without the lock, and their retirement is written later in that same transaction. A member claiming the name in that window passed the retired check, waited on the unique index, and held a retired code once the leave committed. Migration 0081 locks every name a write touches — given up or taken, on delete as well — until commit, in lock-key order (a 32-bit
hashtextcan give two names one key, and ordering by name could then deadlock), so the claim waits and then sees the retirement. Three Postgres tests hold each give-up open, block a claim behind it under the NOSUPERUSER NOBYPASSRLS role, and commit; all three failed against 0074 alone, with the claim handed the retired code. -
A colour scheme picked after a custom colour shows at once (2026-10-11, ADR-073). A custom colour reaches the page as an inline style block, which is unlayered, so it outranked the presets'
@layer baserules whatever their specificity; a preset picked afterwards (applied without a reload) changeddata-accentbut not the colour until the next reload — the owners saw it as "doesn't change until you hit the light/dark icon". The block now applies only while the custom colour is the choice, on its own base (body.app-shell[data-base=…][data-accent=custom]). The Dashboard browser test now runs under the production middleware (a bare router rendered every page in the default scheme, which is why this was invisible) and reproduces the sequence: it fails without the fix. -
Clicking a colour scheme no longer reloads the page and jumps to the picker (2026-10-08, Sprint AP.6). The swatch script posted
new FormData(form), which goes out asmultipart/form-data; the server's form extractor answered 415 and the script's fallback reloaded the page to#colour-schemes— on every preset click since the picker shipped. It now posts url-encoded and the page does not move. The reloads that must remain (a custom colour, the header's dark/light switch) put the page back where it was. The selected swatch's ring is now an outline with nothing painted in the gap, removing the stacked-shadow seam that could show as a light edge around the swatch. -
The control plane's HTTP/3 bridge could panic after answering a request whose body it had not read in full (2026-10-07, #226). When a handler answered without reading to the end — the
413for an over-limit body, the sign-in redirect, the "no free upload slot" page — the bridge told the client to stop sending with h3'sstop_sending(H3_NO_ERROR). Inh3-quinn0.0.10 (the latest release) that unwraps a QUIC stream it no longer holds whenever a read is parked, and h3 leaves one parked even after a read that returned data, so the state cannot be checked from outside. The panic was in the request task after the response was out, so the client saw a correct answer and every test passed through it. The bridge now drops the unread stream instead, and quinn sendsSTOP_SENDING(application code 0) itself.integration_h3_body_limitnow fails on any panic in the bridge and checks that a refused client is told to stop; withstop_sendingput back the suite fails on every run, and with the stream kept alive the three "told to stop" checks fail. -
FerraVault client-certificate renewal: a missed renewal is seen whichever one it is, a sealed vault no longer reads as a refusal, and the alert rules are tested (#221 / #195 review follow-ups, 2026-10-07). Melvyn's non-blocking review comments, all closed:
EpicVaultIdentityNotRenewedages the certificate in use, not the pod. It compared pod age with a renewals counter that resets on a process restart, so a container restarted after its day-60 renewal fired it falsely, and it never saw a missed second renewal. Every FerraVault client now exports thenotAfterof the certificate it is using (epic_<plane>_vault_identity_not_after_seconds,epic_vault_audit_identity_not_after_seconds, read with the renewal digest in one pass by the newepic_net::serving_cert::pem_pair_identity). The rule tickets under 21 days and the newEpicVaultIdentityExpiryImminentpages under 7, the serving certificate's thresholds. That catches every renewal, survives restarts, and also catches cert-manager never renewing. Both live renewal tests pin the gauge to the mounted leaf'snotAfterbefore and after the switch.- The renewal probe reads an answer by where it comes from (FerraVault v1.15.0 layers
Sealed→RateLimit→Auth→ handler).INVALID_ARGUMENT, a handler answer, is accepted;FAILED_PRECONDITION(sealed) andRESOURCE_EXHAUSTED(rate-limited) are undecided (IdentityUndecided): retried like a refusal but not counted as one, so a seal longer than an hour no longer raises the renewal-refused alerts. Same rule inepic-vault,epic-secret-syncandepic-vault-audit; each has a live test that seals an in-process FerraVault mid-renewal, and countingFAILED_PRECONDITIONas refused fails all three. - An owed JWT reload is due at a fixed instant. Each loop turn used to restart its backoff sleep, so events the task ignores (a KEK rotation, another key's) arriving more often than the backoff held the retry off indefinitely.
- The alert rules are tested, in Stage 1. The new
ci/alert-rules/gate runs the pinnedpromtool3.15.0 (sha256-verified, cached underEPIC_BUILD_TOOLS, now/cache/build-toolson the runner) ascheck rulesover every PrometheusRule andtest rulesover committed tests for all eleven vault alerts. The "promtool-tested" claims in the #195 and #221 entries below had no committed test to reproduce them; now they do. Restoring the old pod-age rule, or loosening the threshold, fails the gate. - Docs. Runbook §9 step 4: positional selection changes which records v1.14 grandfathers, never how many, so an equal count proves nothing on a forked node, and the baseline line is logged only when something is grandfathered (
accounted_orphans/unaccounted_orphansconfirmed againstferrav1.15.0's source).EpicVaultAuditRenewalRefusedsays what happens when a transport blip drops the auditor's connection while a refused renewal is on disk. ADR-025 gains a dated revision. The relay test's threshold comment now counts TLS records rather than HTTP/2 frames. - The ~30-day floor the expiry rules assume is enforced (#228 review, 2026-10-08). Both identity-expiry rules now exclude
epic-preview-*namespaces, whose FerraVault identity is a 30-day leaf renewed with 7 days left and would otherwise sit under 21 days for most of its life.ci/ferravault/verify.pyreads each rule's threshold and exclusion fromvault-watch-rules.yamland fails when an identity the rules age is renewed less than a week before one would fire. The rule tests pin each threshold to within an hour from both sides, where before any ticket threshold from 20 to 35 days passed. A newepic-nettest writes leaf + CA totls.crtand asserts the leaf'snotAfter; taking the last certificate instead now fails it, where before every suite passed.
-
FerraVault clients now pick up renewed certificates, and the vault watch resyncs wherever it cannot replay (#216, #217, #186, 2026-10-06). No FerraVault client ever re-dialled with a renewed certificate:
VaultClient::rotate_identityhad no production caller, and ADR-025 §7's rotation pipeline was never built. The edge, control,epic-secret-syncand the auditor kept their boot certificate, and FerraVault re-validates it on every request. So between day 30 and day 90 of pod uptime, control logins would all have returned 503, the edge's Ferron key rotation would have stopped, and Secret syncing would have gone stale, all masked so far by deploys. Each client now polls its mounted certificate by content with the one digest the serving-certificate reloader uses (epic_net::serving_cert::pem_pair_digest), re-dials on a change, and reopens its Watch on the new channel. Renewals are exported asepic_<plane>_vault_identity_renewals_total. Separately, the watch session lost events wherever it had nothing to resume from: the first open after boot (#186), a refused pattern's readmission, andOUT_OF_HISTORY, which it had retried with the same revision forever, as didepic-secret-sync(#217). FerraVault gives a read-only client no current revision, so the session now resyncs instead. It drops the KV cache, nudges revision subscribers, and sends a newKeyEvent::Resync, on which control re-reads its JWT public keys. Live tests (renewal on a seconds-long certificate forepic-vault,epic-secret-syncand the auditor; resync on first open and readmission) are each mutation-checked. ADR-025 Amendment. Review fixes (#221): (1) A renewal is switched to only once FerraVault accepts it. Under TLS 1.3 a certificate FerraVault will refuse (an issuer missing from its client CA bundle, or no binding for the DN and exact SAN set) still dials, and it was swapped in with its digest recorded. Every replica would then fail within a minute, all at once, and never go back to the still-valid certificate. Now the new channel first opens one live-onlyVault.Watchon the plane's own patterns and drops it. Only an answer from an authenticated handler counts: the stream, orPERMISSION_DENIED.UNAUTHENTICATED, a TLS failure or a timeout isVaultError::IdentityNotAccepted, which keeps the current channel and records no digest, so the next poll retries it.epic-secret-syncverifies the same way on its own watch paths, and a config with no[[secret]]mapping is now refused (it would have nothing to verify against). Newepic_<plane>_vault_identity_renewals_refused_total, and two promtool-tested tickets:EpicVaultIdentityRenewalRefused(refused for an hour), andEpicVaultIdentityNotRenewed(a pod over 61 days old that has never switched to a renewal, viakube_pod_start_time, covering the auditor too). (2)rotate_identity(cert, key)is gone. Nothing outside a test called it, and a call with other paths would have been undone by the next renewal poll. The swap is the crate-privateVaultClient::redialof the configured identity, and half an identity is refused atVaultConfig::new(now with a unit test). (3) A readmission from the "every pattern refused" branch owes a resync, as one while streaming does. Before, it resumed past whatever happened under the other pattern. (4) The readmission test now tests readmission: an event first moves the revision past 0, so the reopen resumes and only the owed resync can explain one. A new live test drives the all-refused path by unbinding the subscriber. (5) A JWT verifier reload that outlasts its quick retries is owed, retried on a 1 s → 60 s backoff alongside new events until it succeeds, instead of waiting for the next rotation. (6) Secret-sync subscribes live before itsOUT_OF_HISTORYresync, not after. All six arms of the watch session and the identity reloader shareshutdown_requested. Stale docs fixed (Watcher::changed,rotation.rs, ADR-025's amendment revised). Mutation-checked: swap unverified (both crates),UNAUTHENTICATEDaccepted, the all-refused and the streaming readmission resyncs, the dropped JWT reload, and the uncounted refusal each fail a test. -
FerraVault rolled to v1.15.0; a vault restart no longer silences the edge's key rotation, and the auditor's first run found four kinds of live drift (#183, #193, 2026-10-06).
- #183. Edge, control (with its aggregator) and secret-sync were rebuilt from one revision onto
ferravault-client1.15.0, whose HTTP/2 + TCP keepalive ends a Watch to a vanished vault pod within about 30 s. Then the daemon was rolled (single voter, Ready about 78 s after the upgrade, unsealed by the reconciler). Both edge pods, not restarted across it, delivered the rotation probe at its revision. The "roll the edge after every FerraVault restart" step is retired in three runbooks and the alert comment. - Found on the first run, all fixed.
epic-vault-audit's first check found that:epic-control,epic-edgeandepic-previeweach still held acert:rule their files dropped in ADR-071 / #168. They are re-registered, and the auditor cleared on thePOLICY_PUTevent, 2.1 s after the write.- The identity registry listed no SANs for any identity, while FerraVault keys a binding on the DN and the exact SAN set and every EPIC certificate carries a SPIFFE URI. The auditor itself, bound per the old recipe without
--san, was refusedmtls binding not found. The registry is corrected,verify.pynow fails when registry SANs differ from the Certificate manifest (mutation-tested), and every documentedferra mtls bindgains its--san.
- First v1.15.0 start. It reported this node's three historical off-chain audit records. Each was the last record before a pre-v1.13 daemon stop (a denied health-probe read whose commit never landed; the next boot record chains from the same predecessor), and they are acknowledged with that evidence;
ferra audit verifyis OK. - Backup CronJob. It was found still on 1.12.1-debug (the 1.14.0 pin was never applied); it is now on 1.15.0-debug and proven by a one-off run.
- Live drill. A throwaway binding was flagged as unregistered 2.1 s after
MTLS_BINDING_PUTand cleared on delete.ferravault-verify-live.shexits 0.
- #183. Edge, control (with its aggregator) and secret-sync were rebuilt from one revision onto
-
One refused FerraVault watch pattern no longer silences the rest, and vault access is now checked, not assumed (#168, ADR-072, 2026-10-06). FerraVault refuses a whole
Vault.Watchon the first pattern a policy does not grant, and the watcher retried that set forever — the edge's Ferron bearer-key rotation was dead for two weeks (#167) with every podReady.epic-vaultnow probes a refused set one pattern at a time, keeps the granted patterns streaming, re-probes the refused ones so a policy fix lands without a restart, validates patterns against FerraVault's grammar at boot (WatchPattern), and records every pattern's state on aWatchStatusBoardexported asepic_<plane>_vault_watch_pattern_state{watch_pattern,watch_state}+epic_<plane>_vault_watch_connected, withEpicVaultWatchPatternRefused(page) andEpicVaultWatchDisconnected(ticket). The control plane's signing-key rotation watch is derived from[boot.auth_signing] jwt_signing_key_idinstead of restated inwatch_patterns; that restatement had left previews subscribed to production's key path andkv/epic/**, so every preview's watch was refused outright — previews now setEPIC_VAULT_WATCH_PATTERNS=kv/epic-preview/**and follow their own key. New Stage 1 gateci/ferravault/verify.pyderives every vault request each identity makes from the deployed config (registry:deploy/k8s/epic-state/ferravault-identities.toml) and evaluates it with a port of FerraVault's evaluator: a refused request, an unused grant, an ignored policy key, an unbound policy, a subject-DN/certificate mismatch, an unknown cert spec, or a preview policy reaching production fails the build. It removed the preview policy'scert:epic-preview-wildcardrule, which named a path FerraVault never authorises.deploy/k8s/epic-state/ferravault-verify-live.shcompares the live vault's policies and bindings with the repository inside the operator ceremony (a scheduled check needs full admin in FerraVault 1.14.0; the least-privilege read it waits on is ferravault#8, tracked as #193). Deploy: order-independent; then re-registerepic-previewand runferravault-verify-live.shonce. -
The CI build cache filled the disk the cluster runs on, and every pod was evicted for about four hours (2026-10-06). The PR Gate and nightly sweep build into a persistent
CARGO_TARGET_DIR(/srv/ci-cache/targetonlab-git) that nothing pruned, with incremental compilation on: it reached 1.5 TB (1.1 TBdeps, 324 GBincremental)./srvalso holds the Talos nodes' container filesystems; at 88 % used the kubelet'simagefs.available < 15 %threshold tainted both nodesdisk-pressurefrom 16:28 UTC, and edge, control, Postgres, Ferrum, FerraVault, FerraCert and cert-manager were evicted and could not reschedule while every node reportedReady. Recovered by deleting the incremental cache (cache only). Fix: both workflows setCARGO_INCREMENTAL=0(CI never benefits from it, and sccache — already configured — refuses incremental builds, so it had been bypassed for the workspace crates, the only ones Cargo builds incrementally), and a new first step,ci/cache-guard/guard.py, empties the target cache whenever its filesystem is at least half full. It is a deletion tool, so it refuses any target that, after resolving symlinks, is not atargetdirectory sitting directly in a cache root that is not/; inside it, it walks byO_NOFOLLOWfile descriptor, unlinking symlinks rather than following them and stopping at any mount nested in the cache; it warns when emptying the cache does not bring the disk under the threshold (something else is filling it), and a failed deletion exits 3 with a one-line reason rather than a traceback (26 tests, in Stage 1, each check mutation-tested). It empties the directory rather than removing it because in CI the directory is a bind mount: the first version'srmtreedeleted every file and then failed the job withEBUSYon the mount point, which the new test reproduces (and the old code was confirmed to fail against a real bind mount inlab-git/ci-rust:1.95.0). It is only safe because the runner iscapacity: 1; the workflows README now says a second job slot needs its own target directory. Separating CI's disk from the cluster's is #219. -
The PR Gate leaked two test Postgres containers per run, and nothing reclaimed them (2026-10-05). This was the real cause of the
cargo docOOM kills the two entries below capped job counts for. Theepic-dbharness's#[dtor]reaper andci/docker-hygiene/sweep.pyboth shelled out todocker, but the gate's job image has nodockerCLI: the tests reach the daemon through the socket act_runner mounts. So every gate run loggedleaked test container …: could not run dockertwice, and the sweep crashed onFileNotFoundErrorbehindci.sh's|| true. 273 containers had accumulated inside the runner's 16 GiB cgroup (8.4 GiB of it in swap, host swap 100 % full), so each run had less memory than the last, and rustdoc died on a different crate every time, even at-j 2. Removing them took the runner from 4.9 GiB to 1.2 GiB and its swap from 8.4 GiB to 42 MiB. Both now speak the Docker Engine API over the daemon socket ($DOCKER_HOSTunix://, else/var/run/docker.sock), with only the standard library: a new blockingepic_test_utils::docker::remove_containerfor the reaper (it runs after the tokio runtime is gone; it falls back to the CLI only for a non-UnixDOCKER_HOST), and anhttp.clientUnix-socket client in the sweep.ci.shnow warns when the sweep fails instead of hiding it. Proven with thedockerCLI shimmed out ofPATH:schema_invariants13/13 leaves 0 containers and never calls the shim; the old reaper leaves 1 and calls it. 8 new Rust tests (a fake daemon on a temp socket) and 10 new sweep tests, includingmain()end to end against a fake Engine API. -
cargo docruns at two rustdocs, because four were still OOM-killed (2026-10-05). The entry below capped every cargo step at 4 jobs and credited that with ending thecargo dockills; it did not. Four concurrent rustdocs were killed in all three gate runs after it merged (its own,mainat4d7683c, #176), while #190'scargo doc -j 2passed the full gate.ci.shnow runscargo docwith-j ${EPIC_DOC_JOBS:-2}(Melvyn's #190 change), never above the cargo job count, and an invalid value fails the run. The workflows README replaces #190's request for a runner memory limit, which already exists (16 GiB), with the measured account. -
The PR Gate OOM-killed its own
cargo doc(2026-10-05). Every open PR's required gate was red for a reason unrelated to the PR: rustdoc died withsignal: 9, SIGKILLon a different crate each run. The runner container is capped at 4 CPUs / 16 GiB, but the gate runs in a job container nested inside it whose cgroup shows no limit and whosenprocreports the host's 8, so cargo ran eight compilers in 16 GiB (the runner cgroup had logged 21oom_kills).ci.shhad documentedEPIC_CI_JOBS("default: nproc") from its first version and never read it. It now does: the newci/cargo-jobs/jobs.py(15 unit tests, run in Stage 1) resolves the job count as the smallest ofnproc, the cgroup CPU quota and the cgroup memory limit / 4 GiB per job, exported asCARGO_BUILD_JOBSunless the caller set it; an invalid value fails the run. The per-job budget is measured: rustdoc onepic-controlpeaks at 1.6 GiB and compiling its library tests at 4.7 GiB. Because the nested job cannot see the runner's limits, the PR Gate and nightly workflows setEPIC_CI_JOBS: "4"explicitly, and the workflows README now records the runner's real limits. -
Both planes now serve a renewed TLS certificate without a restart (#169, ADR-071, 2026-10-04). Edge and control loaded the public certificate once at boot with
with_single_certand ignored every renewal; the served*.epic-ai.solutionscertificate expires 2026-11-23, and on that day every handshake on both planes would have failed with every podReady. The certificate is now chosen per handshake byepic_net::serving_cert::ReloadingCertResolverand re-read from the mountedepic-serving-tlsSecret every 60 s; a replacement is installed only if it parses, its key matches the leaf, and it is inside its validity window, otherwise the current one keeps serving. Per-planeepic_<plane>_tls_certificate_{expiry_timestamp_seconds,installs_total,reload_rejections_total}and threeEpicServingCertificate*alert rules. The edge's[tls] vault_path— a "primary FerraVault fetch" no code ever performed — is removed (fallback_cert_path/fallback_key_pathbecomecert_path/key_path, plusreload_interval_secs); control gainstls_reload_interval_secs. The edge stops watchingcert:epic-wildcard, and that rule leaves both planes' FerraVault policies: neither used the private-key grant, and the denied pattern was what took the edge's kv watch down for two weeks (#167). Deploy: ConfigMap and image roll together; re-register the edge policy only after every edge pod runs this build. -
Ferrum-backed tests and benches run against the Ferrum production runs (#156, 2026-10-04). The harness used to take the first
ferrum-serverit found in a shared target dir — a stale debug build, or none at all, in which case local runs silently skipped every Ferrum-backed test.ci/ferrum/test-server.envnow pins the binary to production's image digest;ci/ferrum/fetch-server.shextracts it by digest and verifies its SHA-256; a Stage 1 gate (ci/ferrum/verify.py, mutation-tested) fails the build if the pin andferrum-values.yamldrift;ci.shStage 4 and the Stage 8 bench runner use it; and the meter benches refuse any unpinned server (require_pinned_ferrum_test_server). Against it,meter_record_warmis 8.67 µs (was a 541 µs baseline that measured ferrum#2's recv leak) andaggregator_tick_warm9.55 ms (was 49.7 ms); both re-frozen under the new tagferrum-9.14. -
Ferrum-backed tests and benches run against the Ferrum production runs (#156, 2026-10-04). The harness used to take the first
ferrum-serverit found in a shared target dir — a stale debug build, or none at all, in which case local runs silently skipped every Ferrum-backed test.ci/ferrum/test-server.pinnow pins the binary to production's image digest;ci/ferrum/fetch-server.shextracts it by digest and verifies its SHA-256; a Stage 1 gate (ci/ferrum/verify.py, mutation-tested) fails the build if the pin andferrum-values.yamldrift;ci.shStage 4 and the Stage 8 bench runner use it; and the meter benches refuse any unpinned server (require_pinned_ferrum_test_server). Against it,meter_record_warmis 8.67 µs (was a 541 µs baseline that measured ferrum#2's recv leak) andaggregator_tick_warm9.55 ms (was 49.7 ms); both re-frozen under the new tagferrum-9.14. -
A graceful shutdown could cancel a meter-aggregator tick mid-commit and double-count its batch (2026-10-05).
Aggregator::runraced the wholetick()against shutdown in abiasedselect!, so a shutdown landing between the UPSERT and the ACK dropped the tick: the deltas were in Postgres but the events stayed unacknowledged, and the next start re-added them.tick()is now split into a cancellable pull and an apply that always runs to completion; shutdown still interrupts the blocking pull and the sleep. The SP.3run()test had started failing onmain(10/10 on the post-reboot workstation) because it was catching exactly this; a new gated test holds the UPSERT open while shutdown fires and fails 3/3 on the old code. The module doc's claim that the UPSERT and cursor commit in one transaction was false and is corrected — a crash in that window still over-counts, which needs aMeterAggregateStorecontract change and is tracked separately. -
A FerraVault fetch could cache a key that a concurrent rotation had already replaced (2026-10-05, #182).
epic-vault's secret cache was filled as miss →KvGet→ insert, with nothing stopping an invalidation that landed while the RPC was in flight. The Watch event of a rotation clears the cache, but a rotation pass already reading the previous key would then insert the old value after the clear, and the next pass would serve it from cache for the rest of the TTL (300 s on the edge). That is the shape of a Ferron bearer-key rotation, which writes three keys back to back. Every invalidation now advances a cache generation, a fetch records it before the RPC, and the insert is skipped if it moved. The bug was unreachable while #182 kept the Watch silent; it is fixed before FerraVault 1.14.0 makes the Watch work. -
Ferron 0.2.0 pickup: the window check, tool history, and an empty
tool_choice: noneanswer (2026-10-05, epic#138 / epic#140). Ferron fixed all three of EPIC's open FRs on 2026-10-05 (ferron#1–#3) and publishedferron-tokenizer/ferron-core0.2.0; the EPIC side is four commits. (1) Completion budget: Ferron now enforcesprompt + max_tokens <= context_window, which exposed that the edge checked an absentmax_tokensas 0 while the adapter sent 2048 — such a prompt near the window was refused upstream as a retryable-looking 503ferron_rejected. The newcompletion_budgetresolves the budget once (absent ⇒min(DEFAULT_MAX_TOKENS, window − prompt)) and stamps it onto the Ferron-bound request; Ferron's owncontext_length_exceededrefusal maps to the same 400context_window_exceeded. (2) Tool history through the model's template: earliertool_callsandtool_call_idbind structurally; the hand-written Hermes/Qwen<tool_call>fold is deleted. A new golden test matchestransformersbyte-for-byte on the deployed Qwen2.5 template; request-sidetoolsnow bind in HF's nested shape, andferron-tokenizerturns onserde_json/preserve_order, so schema key order is kept (every JSON hash EPIC computes is key-order independent). (3) Context-window fallback retired: Ferron advertises the window it enforces (live: 16 376 forQwen2.5-3B-Instruct-FP8, clamped by its KV pool — the ConfigMap's 32 768 had become wrong), so the[ferron.context_window_fallback]bridge is removed; apply the ConfigMap before rolling the image (deny_unknown_fields). (4)tool_choice: nonereturned"": the adapter treated any request carryingtoolsas a tool stream and dropped its raw tokens, but Ferron only emitsToolCallContentwhentool_choiceconstrains — present since Sprint 8.1, caught by the live drill's new tool-history turns;translate::uses_tool_contentnow mirrors Ferron's predicate. Live tool drill 17/17 on all three surfaces. ADR-041 and ADR-054 amended; the three FR docs marked resolved. epic#139 (credentials in Ferron's logs) needed no EPIC code: Ferron's fix is deployed and its keys rotated. -
Two tests that passed for the wrong reason (2026-09-18). Both found by the SP.3 sign-off gate; neither is a product defect, and that is the point — each was a check being read as evidence of something it did not test.
The dual-stack listener test took its port from the ephemeral range by binding
[::1]:0and releasing it, so anything on the machine could win that port before the two real listeners claimed it — rare on an idle box, routine under a loaded parallel run. The spawnedserve_h2results were discarded, soEADDRINUSEsurfaced as a timeout rather than as itself, and the panic then asserted "both sockets are bound", naming the one explanation that had been ruled out. It also carried no client timeout, so a port won by a socket that accepts and never speaks TLS would hang the test rather than fail it. It now takes another port and retries, distinguishing the two cases by the fact that aserve_h2whose bind failed has already returned while one that bound is still accepting.ci.sh --fullhad passed this test an hour earlier; its conditions were never adversarial enough to lose the race.The
epic-meteraggregator'srun()— the loop production starts — had no test at all, because every test drivestick()directly. What that left unproven was the behaviour on the way down: the two biasedselect!arms that make shutdown win over a tick in flight and over the sleep between ticks. An aggregator that finished its minute before noticing shutdown would pass every other test in the file and hold a rolling deploy for a minute per pod. Both fixes are mutation-tested. -
A slot certificate whose holder's name spelled PDF syntax could not be verified (2026-09-16). Found by a new property test over the certificate renderer, not by the task that was being worked on. The verifier located the
/Infodictionary by searching the file for the first>>, and a printed value can contain one — so an account named "Acme >> Labs" got a certificate that rendered and signed correctly and then failed its own verification as malformed. Searching fortraileror6 0 objwas steerable the same way: those are keywords a name can also spell. The reader now anchors on line boundaries (certificate text never contains a newline) and steps over string values when looking for a dictionary's end. The shrunk failing case is kept as a regression seed. -
The SDK conformance gate passes again (2026-09-15). Since Sprint SP.1 the edge refuses API keys whose organisation holds no slot (
403 slot_required), andepic_test_utils::seed_inference_orgstill seeded an organisation without one. Its seven users are the conformance server, the live-edge and live-Ferron drill fixtures and four edge benches, none of whichcargo testruns, so the gap surfaced only inci.shStage 4: 17 conformance tests failed with 403 on both SDKs. The helper now grants one slot. OpenAI 15/15 and Anthropic 11/11 pass over plaintext and TLS. On a machine withoutpython3-venv,ci/conformance/run.shleft a venv without pip and then failed with a misleading error on every later run; it now recreates the venv and names the missing package. The live OTel collector test also boots its container before its 30-second deadline, so a first-time image pull no longer fails it. -
Wide tables no longer push the dashboard sideways on a desktop screen (2026-09-15).
/app/admin/auditoverflowed a 1024 px window by 189 px, and the access-code console overflowed at 1280. The Phase 7 F23 fix had capped the grid column only on phones. The column is now capped at every width, and every.app-tablesits in a.app-table-scrollbox, so a wide table scrolls inside its card while narrow tables keep their full width. A source test fails on any table written without the box. The browser overflow gate now also checks 1024 and 1280 px, and was shown to fail with the old stylesheet. A table can opt into wrapping long values instead (app-table--wrap), and columns can be sized with Tailwind utilities on their header cells. Twenty of the 22 pixel baselines were unchanged. -
Leaving signup's "optional" country blank refused the signup (2026-09-14). The field read "Used for OFAC sanctions screening. Optional but recommended", and
RegisterRequestdocumented a missing country as "not blocked" — butOfacGuard::checkreturnsMissingfor an absent code and everyOfacErrormaps tocountry_blocked, so a blank field answered "EPIC cannot accept signups from your jurisdiction". Closed by ADR-066, which removes the field and screens the connection's country instead (see Changed). -
A new migration could be silently missing from an already-built binary (2026-09-14).
epic_db::MIGRATORissqlx::migrate!("./migrations"), which embeds the directory at compile time, and nothing told cargo to rebuildepic-dbwhen a file was added. Adding migration 0052 left one cachedepic-dbartifact embedding the old set, so a test binary linked against it ran without 0052 and failed on the constraint 0052 removes.crates/epic-db/build.rsnow emitscargo:rerun-if-changed=migrations, the fix sqlx documents. -
Renaming an organisation to whitespace would have been a 500 (2026-09-14).
PATCH /api/orgs/{id}checked a new name's length but not whether it was blank after trimming; with migration 0052'sorgs_name_not_blankit would reach Postgres as a check violation. It is now a typed400, and a name with a control character — which the slot certificate refuses, after payment — is refused at every surface that accepts a name. -
The password-reset email linked to a page that does not exist (2026-09-14).
password_reset_requestbuilt the link inline as{origin}/auth/password-reset/confirm?token=…; nothing serves that path — the form is/reset-password, and the JSON confirm endpoint is a POST under/api/auth/— so every reset mail carried a link that 404s. It is the defect already fixed for the verification mail (VERIFY_EMAIL_PATH), which survived here because this model was built at its one call site rather than inepic_postmark::templates::models. It is built there now (RESET_PASSWORD_PATH,models::password_reset), the Postmark field names are unchanged so the template needs no edit, androutes::web::reset_passwordgains the test that crosses the mail and the router — mutation-tested: restoring the old path fails it. Mail sending is still gated on Postmark account approval, so no member is known to have received a dead link. -
FerraVault had been refusing every audited RPC for 2 d 19 h while reporting
Active(2026-09-14).epic-state/ferravault-0wedged at 2026-09-11 07:45:12 UTC and stayed1/1 Readyuntil the pod was deleted — so member login, signup, TOTP and API-key minting were down that whole window acrossepic-controland both preview slots, with the site itself serving 200. One audit-sink emit exceeded the hardcoded 200 ms per-sink timeout inferravault-audit; underfail_closed = truethat becomes a failed StateMachine write, openraft treats a state-machine storage error as fatal, and the RaftCore died permanently. Withreplicas: 1the daemon's own advice — "restart this node's daemon; survivors keep serving" — had no survivors and no restarter. The filesystem was not full (26 % used) and no I/O error was logged: a transient slow fsync on a disk shared with Postgres, Ferrum and FerraVault's own hourly 8.7 MB snapshot build produced a permanent total outage. Nothing detected it, for three separate reasons, and the first is the one that mattered:Vault.Health.raft_leader_knownreads openraft's last-published metrics watch, which keeps naming a leader after the core dies, soferra status --require-active— the probe added specifically to eject a leaderless voter — exited0throughout;epic-observabilityis running nothing, so no rule was being evaluated at all; and none of the four FerraVault alert rules covers this shape anyway (sealed == 1is false, and no-leader is false for the same stale-gauge reason). The retry loop then wrote ~2 MB per 11 minutes of identical errors and rotated its own root-cause logs away within hours, so the trigger had to be dated fromaudit.log's mtime and its last record'sts_ms. Repaired by restarting the daemon; all four control planes re-wired themselves with no roll via their ownvault_reconciletask (epic_control_vault_services_wired 1), which is this incident doubling as the live validation Sprint 7.5's self-heal drill was still carrying as unverified. Filed as a FerraVault FR with the one-line upstream fix (RaftMetrics.running_state, which their openraft 0.9.24 already carries) plus asks to retry a sink timeout before failing the apply, expose the timeout inAuditConfig(deny_unknown_fields, three fields — an operator on a shared disk currently cannot widen it), exit on a fatal core, rate-limit the rejection log, and prunesnapshots/(202 files / 1.1 GB, nothing reclaimed since the cluster was initialised). -
FerraVault has never had an off-site backup, and the CronJob said so for 12 days (2026-09-14).
epic-ferravault-backuppinnedferravault:1.10.3-debug, a tag that is not in the registry, so every run since the job was created hitImagePullBackOffand thenDeadlineExceeded—status.lastSuccessfulTimewas empty, not stale. The siblingepic-dr-snapshotjob that was succeeding nightly covers Postgres only, so the vault holding every KEK, DSN and the JWT signing key had no copy anywhere while the incident above was in progress. Re-pinned to1.13.0-debug@sha256:88a1e31d— by digest, not by tag, because the job mountsfv-admin-client(FerraVault root viamtls_binding) whileepic-require-digest-pinned-imagesbinds onlyepic-edgeandepic-control, so nothing inepic-staterequires a digest and a mutable tag on a registry that accepts any push is a path to running arbitrary code with vault root at the next 02:30 run. The version matches the running daemon (1.13.0,sha256:68826b4f) and the unseal reconciler (1.13.0-debug); checked in that image that/busybox/shresolves andferra backupstill accepts all eleven flags the manifest passes — the release image is the same build minus one busybox layer (21 layers to 22) and has no shell at all, which is what made the original skew easy to miss. The end-to-end run that proved the path (8 749 104 bytes to R2 atlast_log_index 183052, sha256 recorded) was made against1.12.1-debug, the tag the job carried before the daemon moved to 1.13.0. -
The pixel gate was screenshotting an error page (2026-09-09).
tests/visual_regression.rsnever setstate.store.AppStatecarries two store fields —with_invite_and_abuse()fillsbare_store, while/app/usageand/app/fleetreadstore, whichfor_testing()sets toNone— so both pages answered 500 and rendered the generic error page. None of the four browser gates checks HTTP status (client.goto()loads an error page as happily as a real one), so for those two pages the pixel baseline was asserting that the error page had not changed, and the overflow, contrast and landmark probes were measuring its layout. Four checks, two pages, zero coverage, reported green. Found by building thedev-serverexample, which 500'd on exactly those two pages. The misreading had propagated into the project's record of F23: both theKNOWN_HORIZONTAL_OVERFLOWdoc comment and the@media (max-width: 47.999rem)note instyles/input.csssaidapp_usageandapp_fleetescaped the overflow list because their content happened to fit — they escaped because they were rendering a page that carries neither the usage table nor the two fleet tables. Both corrected, and the contrast gate's comment repointed at/transparency, which really is empty by design. Two baselines need re-recording, not four:/appreadsbare_storeviarequire_storeand/app/modelsreads onlystate.fleet, so both are byte-identical with and without the fix — if either moves on re-record, that is a signal rather than expected drift. -
The IPv6 ingress VIP was unreachable by construction (2026-08-25, Phase 9, Sprint 9.4). ADR-060 §19 reverses §10/§12 for the LoadBalancer pool only. §12 put the v6 VIP in ULA space because "the delegated prefix rotates on a ~600 s lifetime" — but 600 s is the router advertisement's valid lifetime (how fast a host forgets a prefix it stops hearing), not a rotation interval, and
2401:d002:520a:9d00::/64has been constant across the workstation,lab-git, the Talos node and the router since the 2026-08-22 survey. Measured off the device, the ISP grants a /64 (ipv6_prefix_len_wan=64); the/56nvramalso reports is the length the router asserts on the LAN, not the length granted. The deeper error would have held even had the premise: ULA bought the v6 pool a stability the v4 pool never had (192.168.50.60-69is just as much a fact of this LAN) and paid for it with reachability — a ULA VIP cannot be reached from the internet on this substrate, since the router has no NPTv6 and NAT66 would destroy the client source address ADR-060 §4 makes load-bearing by giving the edge an empty trusted-proxy list. §12's "the router's job (a static forward to the VIP)" describes a mechanism IPv6 does not have: the delegated /64 is on-link onbr0, so an address inside it is natively routed and there is nothing to forward. The replacement adds no machinery —epic-worker-1already holds a SLAAC GUA on its macvlaneth1(ADR-061 §2) andepic-edge-l2-announcealready answers there, so Cilium NDP-solicits for the VIP exactly as it ARPs for192.168.50.60. Pod, service and site ranges stay ULA. The gate moved with the decision rather than being relaxed:ci/network/verify.pynow requires the LB block to be globally routable, inside a declaredEPIC_LAN_PREFIX_V6, clear of the router's::1, and unable to express a SLAAC EUI-64 identifier — so a "restore" to ULA fails Stage 1 instead of shipping a VIP that answers on the LAN and nowhere else. A gate gap the change exposed: both LB Services pin their VIPs viaio.cilium/lb-ipam-ips, a second uncheckable copy of the address plan that drifted immediately;check_pinned_vips_are_in_poolnow asserts every pin falls inside a pool block (mutation-tested both directions), because Cilium silently leaves a Service with no ingress address while its pods stay Ready when a pin matches no pool. T-9.4.6 attached: the AAAA reconciler had 26 tests and no config section, no call site and no manifest — reachable from nothing, the same shape as theepic-clusterenforcer the sprint had already flagged twice. It now has a[boot.ddns]section with a fail-closedvalidate(), aServiceAddressSourcethat reads the address off theepic-edge-lbService (noaddresskey, anddeny_unknown_fieldsmakes adding one a parse error — a constant can be confidently wrong, and a wrong AAAA is a black hole behind green probes), a record name derived from[domain] base, and a narrow RBAC grant (geton one named Service).ci/network44 → 54 tests,epic-cluster23 → 32,epic-config40 → 47. See Sprint 9.4 §7. -
Test containers leaked on every run — three mechanisms, all needed (2026-08-25, Phase 9). A
cargo test -p epic-dbthat passed all 39 tests still stranded a Postgres container, every time: the shared container lives in astatic OnceCell, and Rust never dropsstatics, soContainerAsync'sDrop— the only removal pathtestcontainers0.27 offers, since that version ships no Ryuk reaper — was unreachable on a completely ordinary run. Roughly 450 containers accumulated on the development workstation between 2026-05-29 and 2026-08-25 (all EPIC's; theorg.testcontainers.managed-bylabel andPG_MAJORenv identify them precisely). Fixed in three layers, each covering what the previous cannot: the workspace enables testcontainers' non-defaultwatchdogfeature (SIGTERM/SIGINT/SIGQUIT, then re-raises the default handler so Ctrl-C still behaves like Ctrl-C);crates/epic-db/tests/common/mod.rsgains a#[dtor]reaper for the normal-exit path, because libtest exposes no end-of-binary teardown hook and the platform destructor section is the only place left to stand; andci/docker-hygiene/sweep.pyreclaims the residue ofSIGKILL/OOM, which is uncatchable by construction — with an age threshold so it can never remove a concurrent run's containers. Verified against a probe that stays alive long enough to actually be signalled, since the first attempt "passed" only because the suite finished before the signal landed: SIGINT → 0 leaked, SIGTERM → 0 leaked, SIGKILL → 1 leaked (as designed, and the reason the sweep is not decorative). The sweep's selection logic has 12 unit tests and was live-verified in both directions; both run inci.shStage 1. -
Sprint 7.7 — edge HTTP/1.1 SDK compatibility (2026-07-05, Phase 7). Finding F22: the edge TCP:443 listener advertised ALPN
["h2"]only (ADR-011), so HTTP/1.1-default OpenAI-compatible clients (the OpenAI Node SDK / undici that drives Cline/Roo/Cursor;openai-pythonon httpx defaults;curl --http1.1) failed the TLS handshake withno_application_protocoland reported a genericConnection error— breaking the "SDKs work unmodified" thesis. Masked because every SDK test (openai/anthropic conformance + the Phase 8.3 "real Cline" drill) ran against the plaintext loopback fixture (h1.1 cleartext), never the h2-only TLS listener. Fix: the h2 listener now advertises["h2", "http/1.1"](h2 preferred;boot::with_alpns);hyper's auto builder already serves both, so nothing above the listener changes (router, pipeline, auth, metering, zero-per-token-alloc hot path identical). ADR-011 revised in place; DESIGN.md §8.2 reconciled. Regression guardh2_listener_serves_http1_clients_over_tlsbindsserve_h2+ drives a real HTTP/1.1-over-TLS request (fails against the prior["h2"]). Root cause closed: theopenai/anthropicSDK conformance gate (ci.shStage 4) now runs each surface over both plaintext and TLS —conformance_servergained anEPIC_CONFORMANCE_TLS=1mode serving over the productionserve_h2+ ALPN with a self-signed cert the harness trusts — so SDK conformance is exercised over the REAL transport, not only the plaintext fixture that masked F22 (negative-tested: an h2-only ALPN turns the TLS pass red; green = openai 15/15 + anthropic 11/11 over TLS). Edge image rebuilt + rolled; verified end-to-end with Cline againsthttps://api.epichub.ai/v1. See Sprint 7.7. -
Sprint 7.6 — FerraVault cluster-resilience adoption (2026-07-04, Phase 7). Finding F20: a recurring multi-minute secret-plane outage after any FerraVault pod restart — a sealed voter has no Raft instance (Raft is built only after
Active), so co-located Shamir voters that all reseal on restart are absent from quorum, the loneActivevoter storms its term (> 65 000, no pre-vote onopenraft 0.9.24), and audited ops fail-closedaudit unavailable(JWT signing + api-key mint down cluster-wide); recovery depended on a flawless manual per-node unseal ceremony. Root cause is upstream/structural; EPIC filed a four-part FR and FerraVault delivered all four the same day in daemon v1.11.0 / chart 0.10.0 (their ADR-046). EPIC adopts the chart-native path (ADR-056) rather than build the interim reconciler the FR §4 anticipated:ferravault-values.yamlre-pinned to v1.11.0 (digestsha256:642403c7…) withunsealReconciler(unattended auto-unseal from a standing per-node shares Secret — dev-only custody tradeoff, prod staysassisted-kms),readiness.mode: active(ejects sealed/leaderless voters from the client LB),config.raft.profile: single-host(wider election timeouts kill the storm on the current pin — pre-vote is not the fix), + the shippedFerraVaultSealedTooLongalert;ferravault-jobs.yamlNetworkPolicy extended for theunseal-reconcilerpods; unseal runbook §6 rewritten (reconciler is the steady state, manual unseal is break-glass). Deployed live 2026-07-04 (T-7.6.10):epic-controlrolled to the F19+F21 image (sha256:f46d491c…, commit9133d18); FerraVault upgraded to v1.11.0 / chart 0.10.0 (helm upgrade --force-conflicts, a legacykubectl-setfield-manager owned.image); the reconciler auto-unseals resealed pods; the acute F20 outage is resolved — the cluster serves on a stable leader with a correct 3-voter membership and a term that stopped storming,epic_control_vault_services_wired=1, zeroaudit unavailable/signing_unavailable. A share-file header-line bug (the standing shares Secret must holdfv1s-lines only, else the reconciler feeds a header as "share 1" → opaque daemonInternal error) was caught + fixed during the deploy. Residual — since RESOLVED:ferravault-0, poisoned with a raft term higher than the leader's by the pre-v1.11.0 storm, would not rejoin (PVC-persistent term; delete+re-unseal does not clear it);ferra cluster rebuild-followercleared the poison but the leader would not auto-re-onboard the reset voter → filed a follow-up FerraVault FR. FerraVault answered same-day with v1.12.0 / chart 0.11.0 (their ADR-047) — root-caused to four stacked defects, the decisive one a raft-plane transport ceiling (tonic's 4 MiB default vs ~3.6×-inflated serde_json snapshot chunks → nostate.dbpast ~1.1 MiB could catch up a blank node). EPIC adopted it (values re-pinned to v1.12.0@sha256:06cef248…+[cluster_raft] max_message_bytes=32 MiB +install_snapshot_timeout_ms=120 s + a globallogFilterfloor;helm upgrade); fv-0 auto-rejoined on the roll (voters=3,leader_known=1, term stable, 0 restarts) → full 3/3 quorum restored, F20 fully closed. Also resolved finding F21 (T-7.6.12): the recovery-path verification found that per-record audit signing (ADR-031), diagnostic capture (ADR-008), and review sampling (ADR-037) wire from the rawvault_clientat boot and were not re-swapped by the Sprint 7.5 reconciler (the audit sink is also moved by value into the vote/kill-switch/billing services). Fixed with a unifyingepic_control::hot_swap_vault::HotSwapVaultClient— oneArcSwapOption<VaultClient>implementingCryptoSigner+KeyWrapper+KekProvisioner, built once at boot, threaded into all five consumers, and filled by the F19 reconciler so every one heals atomically; the audit emit path now degrades to unsigned on a signerUnavailable(chain intact) instead of failing the audited write. Proven byhot_swap_vaultunits + a PG-backedsigner_unavailable_degrades_to_unsigned_then_heals. See Sprint 7.6. -
Sprint 7.5 — control-plane vault-service self-healing (2026-07-04, Phase 7). Finding F19: a node reboot re-sealed FerraVault under a running
epic-control, and becauseAppState::wiredialled the vault once at boot, every vault-derived auth service (jwt/totp/turnstile/vault/vault_writer) was disabled for the process lifetime — every login returned401JwtService is not wiredwhile the pod reportedReady(the/readygate is deliberately adapter-independent, from the 2026-06-16 correlated-outage fix), and unsealing the vault did not self-heal it (a manualrollout restartwas required — the edge, which hard-crashes on a sealed vault, self-heals via crash-loop; control did not). The five services now live behindarc_swap::ArcSwapOptionslots (epic_control::vault_services::VaultServices) withAppStateaccessor methods, and a background reconciler (epic_control::tasks::vault_reconcile) re-runs the identicalwire_vault_servicespath under bounded backoff and hot-swaps them in when FerraVault returns — no crash, no correlated readiness flap, no manual restart. Spawned by the composition root only when a vault is configured butjwtdidn't wire at boot. Newepic_control_vault_services_wiredgauge (0degraded /1wired) +/readydiagnostic field +EpicControlVaultServicesUnwiredpage alert make the degraded window loud instead of silent behind a green probe. 429 lib tests (incl. the sealed→unsealed reconciler transition) + login e2e (12/12) + sprint-1.2 (8/8) green; clippy + doc gate clean. Live sealed→unsealed cluster drill pending (T-7.5.5). See Sprint 7.5 sign-off. -
Sprint 7.1 — RLS tenant-scoping remediation, full sweep (2026-06-22, Phase 7). Finding F4 closed in full: every org-scoped FORCE-RLS data-access method that ran on the raw connection pool with no active-org GUC (the defect superuser tests masked) is now scoped on a pinned connection (
begin_for_org), via a SECURITY DEFINER wrapper, or underSET LOCAL ROLE epic_admin. Migration 0050 adds theapi_keysby-id/by-public-id/by-prefix/by-user + touch/disable/expiry/revoke wrappers and theorg_member_invitesconsume/revoke wrappers (the by-id surface is reached from the edgeepic-edge-rorole, control, and cross-org abuse workers, so a uniform SECURITY DEFINER path is the only correct fit — 0028/0049 doctrine).org_membersmutators +transfer_org_owner,vote.rsballots, andinvites.rsinsert/list converted tobegin_for_org; the cross-org IRVlist_ballots+ GDPRhard_delete_user+ theupdate_org_seat_countorg-member count now escalate correctly (the last two were extra F4 bugs the hardened harness caught, not the originally-named methods). Newintegration_pg_rls_tenant_scoping.rsexercises the whole surface under a production-shapedNOSUPERUSER NOBYPASSRLSrole (member ofepic_adminonly viaSET LOCAL ROLE); 25/25 non-superuser tests green, no trait-signature change. See Sprint 7.1 sign-off. -
Sprint 7.2 — multi-model edge enablement + F9 incremental detokenisation (2026-06-22, Phase 7). Playground multi-model walkthrough cluster F9–F13: the edge now adopts Ferron v1.0.1's
IncrementalDecoderper response stream (epic-ferron-client, ADR-052) so SentencePiece (Llama/Mistral) output renders with correct word-boundary spaces (F9/F13); shippeddeploy/tokenizers/TinyLlama-1.1B-Chat-v1.0/byte-identical to Ferron's source with the hash-alignment gate now table-driven over every model (F12); relaxed the dev classifiercall_timeout_ms75→1500 for the cold/GPU-time-sliced encoder (F11); F10 (multi-model serve) resolved upstream in Ferron Phase 5 with zero EPIC change. See Sprint 7.2 sign-off. -
Sprint 7.4 — API key management surface remediation (2026-07-04, Phase 7). Member-walkthrough finding-cluster F16/F17/F18 on
/app/keys: added the missing "Create key" CTA (the create flow shipped in T-1.4.8 but the list page was never wired to it) (F16); ephemeral playground keys (ADR-046) that share theapi_keystable no longer pollute the human management surfaces — newStore::list_manageable_api_keys_for_org(DB-levelkey_type <> 'ephemeral') on the dashboard + JSONGET /api/keys, withlist_api_keys_for_orgleft intact for the kill-switch cascade + playground cap counter (F17); and expired ephemeral keys are now reaped hourly (Store::purge_expired_ephemeral_keyscross-tenantepic_adminDELETE +tasks::ephemeral_key_reaper, mirroring the review-sample sweep) so the table can't grow unbounded (F18). Proven under the production-shapedNOSUPERUSER NOBYPASSRLSrole. See Sprint 7.4 sign-off. -
Sprint 6.3 post-sign-off —
ci.sh --fullgate failures (2026-06-18, Phase 6). A fullci.sh --fullrun (the 6.5 gate, pulled forward) surfaced three failures the per-crateclippy/cargo testchecks miss; all fixed at the source, no deferral, no visual page dropped. (1) doc — a workspacecargo doc -D warningscascade: 2epic-webplayground errors (private + cross-crate intra-doc links) masked the rest, which surfaced once fixed —epic-test-utils(det_public_uuid),epic-config/epic-control::state(host[:port]→ broken:portlink), and latent Sprint-6.2 admin-console private links (require_operator_session,emit_admin_audit_with_actor,drive_*_transition_core); all de-linked,cargo doc --workspaceclean across 24 crates. (2) bans — Sprint 6.2'sgrant-operatordev-dependency onepic-storefromepic-migratewas missing fromepic-store'sdeny.tomlwrappers (cargo-deny checks dev deps); added. (3) visual — two independent root causes. Engine:ci/visual/run.shpreferred Chrome, but Chrome--headless=newyields non-reproducible screenshot dimensions on an active desktop (viewport jitters run-to-run; centred layout ⇒ no pixel-exact match) — switched the gate to prefer Firefox (dimensionally stable; respectsEPIC_WEBDRIVER) + auto-pick a free port. Content: the 8 Sprint-6.2 admin pages +app_playgroundhad no baselines, andapp_admin_orgs/app_admin_auditflaked on random UUIDv7 public ids the pages render — added an opt-inMemStore::with_deterministic_public_ids()(default unchanged). Re-recorded all 16 baselines with Firefox → compare-only passes 0.00000 on every page. Stage 8 bench gate green (0 regressed vsphase-4-final). See Sprint 6.3 §7 "Post-sign-off CI remediation".
Added
-
Sprint 7.3 — Quantized-model serving (catalogue adoption) (2026-06-30, Phase 7). EPIC-side adoption of Ferron FP8 serving — EPIC surfaces the precision Ferron advertises; it never quantizes weights itself. ADR-053 Accepted —
epic-protocol::native::ModelInfogains an optionalquantization: Quantization { None | Fp8 | Fp4 | Other(String) },#[serde(default)]so an older Ferron omitting the field parses unchanged and an unrecognised future precision token degrades toOtherrather than failing the/v1/capabilitiesparse (two planes deploy on independent cadences). Parsed byepic-ferron-clientfrom Ferron's capabilities, carried two-plane through the ADR-048 Ferrum publish/read, and rendered as a precision badge + "Precision" column on/app/models. Quantization is a model-load property and does not touch the tokenizer, so a quantized build shares the base model's ADR-035tokenize_build_hash(verified live:Qwen2.5-3B-Instruct-FP8==Qwen2.5-0.5B-Instruct).deploy/tokenizers/Qwen2.5-3B-Instruct-FP8/shipped + added to the production-path hash-alignment gate; edge configmapcontext_window_fallbackextended; edge rebuilt + rolled (6/6, clean boot, zeroModelDirMissing). Live playground stream againstQwen2.5-3B-Instruct-FP8confirmed (T-7.3.7a). Files the Ferron FP8 FR + the FP8 checkpoint-format taxonomy. See Sprint 7.3 §7. -
Sprint 6.4 — Member dashboard depth + fleet/model status (2026-06-18, Phase 6). Completes the member-facing product + the §1 "see fleet state" thesis. ADR-048 Accepted — the edge republishes its polled
Capabilities+LoadSnapshotto the shared Ferrum substrate on the existing 5 s poller tick (fire-and-forget, off the per-token hot path — all five ADR-026 invariants intact), and the control plane reads them back via theCacheboundary; this honours the two-plane substrate invariant (CLAUDE.md §4.1 — control never dials Ferron, holds no Ferron credentials) and was chosen over control-dials-Ferron + edge→control-RPC. Cross-plane contract inepic_protocol::native::fleet(epic:fleet:{capabilities,load}:v1+ 30 s TTL);epic_edge::fleet_publish::FleetPublisher+[ferrum] fleet_publish_ttl_secs;epic_control::fleet::FleetSnapshotReader. New surfaces:/app/models(live catalogue + typed-501 roadmap),/app/fleet(capacity + the org's fair-share standing rendered as the exact edge annotationfair_weight = seat_count, R6.7 — test-pinned toCachedModelRouter), and a deepened/app/me(editable profile, three optional-email toggles, security/TOTP surfacing, PRG notice). Migration 0048user_notification_preferences(FORCE-RLS onapp.user_id); newepic-trait::NotificationPreferencesStore(+Pg/Memadapters). Dashboard home now composes the playground CTA (6.3) + a 30-day usage summary (6.1) + the live fleet snapshot, each degrading gracefully. Tests: fair-share correctness (display == edge annotation),/app/{models,fleet}+/app/mesettings handler integration, RLS isolation + Pg round-trip (real PG), schema-drift guard; visual set gainsapp_home/app_models/app_fleet/app_mewith a deterministic seeded fleet snapshot. See Sprint 6.4 §7. -
Sprint 6.3 — In-browser inference playground + browser↔edge auth bridge (2026-06-18, Phase 6).
/app/playground: a logged-in member streams a completion live in the browser against the real edge. ADR-046 Accepted — the control plane mints a short-lived, org-scoped, single-purpose ephemeral API key and the browser streams directly against the edge with it (no control-plane inference proxy; two-plane blast-radius isolation preserved). NewKeyType::Ephemeral(wire tokenephemeral) is a class on the existingapi_keysmodel, not a parallel system — same BLAKE3-keyed-at-rest hash, UUIDv7 id, FORCE-RLS; migration 0047 extends thekey_typeCHECK and pinsephemeral⇒expires_at+ephemeral⇒scopes={inference}at the storage layer (ADR-006 reconciled). Mint endpointPOST /api/playground/session(epic-control): session-gated (anon → 403),[Inference]-only, TTL'd (default 900 s), revocable, audit-stamped (api_key.mint_ephemeral), per-org active-key cap (→ 429);[control] edge_public_originconfig. The edge consumes via the existingApiKeyService::verify(one-variant widening of the inference type gate,Sk → Sk | Ephemeral— no new hot-path branch); a no-credentials browser-SSECorsLayer(closedepichub.aiorigin set, noAccess-Control-Allow-Credentials— no cookie surface on the data plane).epic-webSSR shell + pinned dependency-free SSE client (fetch +ReadableStream, native framing, cancel viaAbortController, re-mint on expiry). ADR-026 zero-per-token-alloc re-asserted with a playground-key stream (the hot path is key-type-agnostic). Tests: mint authz/scope/TTL/cap/audit, edge verify + expired-reject + CORS allow/deny, PG-backed ephemeral RLS isolation + migration-0047 CHECK enforcement, page render;/app/playgroundregistered in the visual-regression set. See Sprint 6.3 §7. -
Sprint 6.2 — Admin dashboard / operator console (2026-06-17, Phase 6). The complete
/api/admin/*JSON surface rendered as a session-authenticated SSR operator console under/app/admin. ADR-049 Accepted — platform-operator identity asusers.is_epic_operator(migration 0045), settable only out-of-band via newepic-migrate grant-operator/revoke-operator;require_operator_sessiongates the whole subtree at parity with the JSON surface'sKeyScope::Admin(non-operator session → 403, anon →/login), with the JSONrequire_admin_keysurface left unchanged. Nine SSR sections (overview, org mgmt, abuse triage + opt-out review, kill switches, diagnostic-capture + review-sample viewers with server-sideVault.Unwrapdecrypt + audit-before-plaintext, meter health, audit viewer with chain-verify status, vote admin). No reimplementation — actor-parameterised cores (drive_opt_out_transition_core/drive_report_transition_core+AdminActor) drive the same Phase 4 policy from both planes; operator actions audit-stamp the operator user (emit_admin_audit_with_actor). Destructive-action safety: shared GET-confirm → POST-execute flow (render_confirm) with required reason + per-record-signed audit. Admin chrome + CSS (deterministicsite.cssrebuild);/app/admin/*wired into the visual-regression baselines. Tests: table-driven authz negatives over every route + action-parity (state change + operator-stamped audit row) + 25 render tests. T-6.2.3 edge-config kill-switch firing — delivered in full via the 2026-06-18 addendum below (ADR-050); the earlier read-only-panel / Phase-7-deferral disposition is superseded. See Sprint 6.2 §7. -
Sprint 6.2 addendum — Runtime edge kill-switch control (2026-06-18, Phase 6; ADR-050). Closed the flagged T-6.2.3 edge-config firing half by building the missing control→edge seam properly instead of deferring (operator-directed "do the full works"). ADR-050 Accepted — a durable
edge_kill_switch_rulestable (migration 0046, operator-global config, non-RLS) is the single source of truth for the five edge-config scopes (IP/CIDR, country, UA-regex, model grounding, org deny); the operator console is the only write surface (operator-gated + audited), and both planes reconcile from it: the edge polls it on a background interval ([killswitch] db_poll_interval_secs, default 5 s) and hot-reloads the mergedArcSwapsnapshot, while IP/country mutations also drive the Sprint 5.3KillSwitchEnforcer(Cilium/Cloudflare) best-effort. Newepic-trait::EdgeRuleStore(+EdgeRuleScope::canonicalizevalidating against the same parsers the edge enforces — no drift) /PgEdgeRuleStore/MemEdgeRuleStore;EdgeRuleId/EdgeRulePublicId./app/admin/kill-switchnow CRUDs each scope (add with required reason, confirm-page remove), operator-stamped per-record-signed audit (edge_kill_switch_rule.add/.remove). Two-plane separation preserved (control writes, edge reads — no inbound edge mutation surface); hot path unchanged (consult ≈ 105 ns; merge+reload ≈ 464 µs every 5 s, off-path). Tests: trait/mem/edge units, control SSR CRUD integration (incl. new routes in the every-route authz negative table),PgEdgeRuleStoreDB roundtrip +grant-operatorDB test (the latter also fixed a latent CITEXT case-insensitive-match bug →$2::citext),schema_invariants(table registered) +rlsgreen. See Sprint 6.2 §7. -
Sprint 6.1 — Usage statistics + public transparency (2026-06-17, Phase 6 opens). Define-then-render over the live
epic-meteraggregates: ADR-045 (per-org-private data contract) + ADR-047 (public community-aggregate scope, small-n suppression k=5 in SQL) Accepted. Migration 0044 extends the meter rollups additively (filtered_count,sum_ttft_ms/sum_total_ms/latency_sample_count; retired the deadavg_*columns). Newepic-trait::UsageAnalyticsStore(PgStore): RLS-org-scoped per-org read + admin cross-org community read with suppression in-query. SSR: member/app/usage(volume / model mix / experienced latency / seat-derived fair-share + server-rendered inline-SVG chart) and public/transparency(community totals, suppressed buckets hidden). Grafana usage dashboard (Prometheus-sourced; refusals at category granularity per ADR-047 §2). DB tests prove RLS isolation + the suppression boundary against real Postgres; visual-regression gained an authenticated-screenshot path (/app/usage+/transparencybaselines, 0.00000 delta). Analytics, never billing — zeroepic-billingcoupling. ADR-number reconciliation: transparency took ADR-047 (ADR-018 was consumed by Sprint 5.2 PKI). See Sprint 6.1 §7. -
Sprint 5.6 — Ferrum self-healing-client retirement (2026-06-17, post-
phase-5-finaladdendum; does not reopen the tag). Retired EPIC's Sprint 5.5 client-side reseed wrapper onto the upstream fix (ADR-044 / Ferrum ADR-126). Both planes dialFerrumClient::connect_quic_dnsand let theAsyncClusterClientown total-topology-loss recovery;ClusterBackendcollapsed fromArc<ArcSwap<…>>to a directArc<AsyncClusterClient>(deletedreseed_healer/ReseedTarget/connect_quic_self_healing+ thearc-swapdep, −307 lines, hot path dropsclient.load()). Gated on the T-5.6.2 live drill (ci/drills/phase-5-ferrum-reseed/run.sh— delete-all-Ferrum → total loss): PASS — all 6 edge podsepic_ferrum_reseed_total=1, 0 restarts, Ferrum re-formed 3/3. T-5.6.4 surfaced the fourepic_ferrum_*reseed counters on both planes'/metrics+ aPrometheusRule(EpicFerrumReseedFailingpage /EpicFerrumReseedChurnticket) + a Grafana dashboard. ADR-044 Accepted; the FRsprint-9.8-finaltag claim corrected (reseed work is on Ferrummain/ab4cb2dc, untagged). See Sprint 5.6 §7. -
Sprint 5.6 (T-5.6.6) — boot-secret transient-seal retry (2026-06-17). A single transient FerraVault
vault is sealedat boot no longer crash-loops the edge fleet (or silently disables control's TOTP/Turnstile/Postmark): newepic_store::retry::{retry_with_backoff_if, fetch_secret_resilient}+epic_trait::KeyVaultError::is_transient()ride out a transient seal (45 s default,EPIC_VAULT_BOOTSTRAP_BUDGET_SECS) and fail fast on permanentNotFound/AccessDenied. Incident-driven by the T-5.6.2 FerraVault seal (and the Ferrum ADR-127 QUIC-over-VXLAN MTU fix that cleared the parallel ~400 s bootstrap). See Sprint 5.6 §7. -
Sprint 5.5 — Phase 5 sign-off (gating) + edge↔Ferrum self-heal resilience fix (2026-06-16). Gated Phase 5 against the dev cluster: T-5.5.1 originally-planned-vs-delivered (48 items across 5.1–5.4, 47 ✅ + 1 four-field-deferred cosign half, operator-approved → Phase 7); three audits PASS (quality / security extended to the cluster attack surface / architecture); the three §17 exit-criteria drills green as gates — cert-rotation zero-drop, chaos 6/6 (kill-one-of-each; FORCE-RLS verified on the sync-promoted primary; fail-closed JWT-503 held), backup-restore layer-1 (R2 WAL recovery, FORCE-RLS + NOBYPASSRLS intact); first dev-cluster §16.2 SLO baseline (edge auth p99 ≈ 1.0 ms, ~5× under). Operator-directed in-phase resilience fix: a Ferrum
StatefulSetroll previously wedged the running edge/control until pod restart (the cluster-client did not re-resolve the headless topology on total cached-IP loss);epic-store::FerrumClientnow self-heals —ArcSwap-heldAsyncClusterClient+ a background reseed healer that re-resolves DNS (tokio::net::lookup_host) and atomically swaps a fresh client on total loss, no restart. Validated live (full Ferrum IP rotation → all 6 edge pods reseeded in-process, 0 restarts). Reciprocal upstream Ferrum FR filed. Two gate-caught defect fixes (no deferral): the cancel-budget integration test's regression sentinel moved p99→p50 (the tail was scheduler noise, not the cancel path; no SLO loosened), and the chaos harness recovery targetPRE_TOTAL→PRE_READY(a never-Ready backup CronJob pod shared the ferravault label). Tagphase-5-final(unsigned annotated). See Sprint 5.5 §7. -
Sprint 5.3 🚧 — close: country kill-switch drill-proven LIVE + control restored 3/3 + sign-off sweep (2026-06-15). The T-5.3.9 country adapter applied + read-back + cleared a real
ip.geoip.countryblock on the liveepichub.aizone (EPIC_CF_LIVEdrill, 2 audit rows); Cloudflare DNS + country-firewall + §9.7 rate-limit rulesets provisioned via the scoped token (managed-WAF / origin-port / SSL-mode residuals operator/plan-gated, documented incloudflare-zone.md§7). The edge's Ferrum bootstrap retry was shared intoepic_store::retry+ wired intoepic-control(last single-attempt dial removed) with the headless-ferrumseed; control restored toreplicas: 3(HPA min 3 + PDB), no crash-loop. Five ADR-026 edge invariants re-asserted. The workspace sign-off sweep caught + fixed Sprint-5.x drift the scoped checks missed: acargo denytwo-plane bans gap (kube←epic-cluster, epic-auth/epic-vault←epic-migrate, epic-auth←epic-edge — all legitimate, allowlisted with rationale) and 6 brokencargo docintra-doc links. See Sprint 5.3 §7. -
Sprint 5.3 🚧 — Ferrum 9.4 (ADR-125 QUIC keep-alive) + both EPIC images rebuilt & deployed → watch idle-drop RESOLVED live; Turnstile + Postmark ACTIVE (2026-06-14). Rolled Ferrum to 9.4 (
82efe175fb75, image-only — the client-side keep-alive is the fix), rebuilt + deployed both EPIC images from clean8d588841: edge (sha256:f840567a…, 9.4 clientkeep_alive_interval=15s+ watch-backoff fix) and control (sha256:002b4350…, Turnstile site key + Postmark token). Verified live: edge watch holds a single stable connection —watch dropped=0 over 6+ min (was ~12 at the 30 s cadence); the/signuppage renders the real Turnstile widget (data-sitekey+ CF script);/ready=200 with the Postmark token loaded. Added the Postmark sender DNS (DKIM TXT + Return-Path CNAME) to theepichub.aiCloudflare zone — both resolve live (Postmark domain-verify pending operator click). Idle-timeout FR closed. Follow-up filed: edge crashes-then-retries the Ferrum bootstrap dial under simultaneous-restart contention (should retry-with-backoff; gates restoring edge replicas 6). See Sprint 5.3 §7. -
Sprint 5.3 🚧 — Ferrum sprint-9.3 (ADR-124 cross-shard watch fix) deployed →
0x0306watch bug RESOLVED (verified live); Turnstile site key wired (2026-06-14). Ferrum shipped the cross-shardWATCH.*connection-ID fix as sprint-9.3 (forward dispatch now propagatesConnectionId); EPIC rolled the StatefulSetsprint-9.2 → ba6f7ff94fd7(digest-pinned, one pod at a time) and verified live:0x0306is 0 across all edge pods (was steady-state), the keyspace watch registers regardless ofSO_REUSEPORTshard landing. FR RESOLVED. One distinct low-severity follow-up filed: the idle watch connection idle-times-out at ~30 s because the Ferrum async client sets no QUICkeep_alive_intervaland exposes no knob — EPIC cannot fix it client-side (FR); the watch stays functional via reconnect-catch-up + re-sync. Also seededepic/postmark-token+epic/turnstile-secret(fv-admin ceremony,--from-file) and wired the Turnstile site key from[boot.control] turnstile_site_key(was a hardcodedNonePhase-6 stub) — the public half coupled to the verifier secret, locked inconfigmap_shape.rs; activation needs a fresh control image. See Sprint 5.3 §7. -
Sprint 5.3 🚧 — Ferrum sprint-9.2 (ADR-110 §Amendment-1, 0-RTT correction) deployed → QUIC churn FR resolved (2026-06-12). Ferrum shipped the fix for EPIC's QUIC handshake-instability FR as sprint-9.2 — rustls 0.23 silently downgrades the Sprint 7.4 stateless ticketer per RFC 8446 §8.1, so the
ferrum/1data-plane listeners emittedearly_data with stateless resumption is not allowedand churned connections; 9.2 switches to stateful resumption (FerrumSessionStorage). EPIC verified the fix + chart-unchanged pairing in source, rolled the StatefulSetsprint-9.1 → sprint-9.2(digest-pinned, one pod at a time, no unseal, quorum held), and verified: early-data warnings 0 (was 32/60s), control un-graceful drops 0 (was ~12/60s), and a freshepic-controlbootstrap completes clean (restarts=0) under full edge churn (the original crash-loop symptom). FR RESOLVED; one separate, non-blocking edge keyspace-watch issue (0x0306) unmasked + tracked as a follow-up. See Sprint 5.3 §7. -
Sprint 5.3 🚧 — FerraVault v1.10.1 (ADR-045 ring coherence) deployed → JWT now LIVE (2026-06-12). FerraVault shipped the fix for EPIC's signing-key-ring HA FR as v1.10.1 / ADR-045 "cross-replica ring coherence" (the in-memory
signing_key_ringis refreshed from the Raft apply path on every node, so a signing key created on a running cluster lands in every voter's ring, not just the leader's). EPIC rolled the lab cluster1.10.0 → 1.10.1(digest-pinned, one pod at a time, each node re-unsealed with dev Shamir shares to hold 2/3 quorum) and verified end-to-end: the JWT key now reads on all three nodes (was leader-only), andepic-controlre-booted withJwtServicewired (the priorJwtService disabledwarning gone). JWT + TOTP both live; the FR is marked RESOLVED. Only the two operator-credential gates remain (Postmark + Turnstile). See Sprint 5.3 §7. -
Sprint 5.3 🚧 — epic-control deployed Ready+serving; control auth-flow FerraVault material provisioned (2026-06-12). Brought
epic-controlup on the lab cluster (1/1Ready, SSR site serving) by fixing a chain of never-deployed Sprint 1.6 control defects (configmap shape, mTLS-only FerraVault posture, HTTPS probes, and four Ferrum-dial bugs — the showstopper being a missingferrum/1ALPN). Ran the fv-admin ceremony to provision the auth-flow material: TOTP encryption key seeded →TotpServicewired LIVE; the JWT signing keyepic-jwt-signing-key-v1(ed25519/pure) created + the epic-control FerraVault policy gained the three ADR-030 signing-key grants (sign/verify/watchon the exact per-op sub-paths) + the rotation watch pattern (guarded byconfigmap_shape.rs). JWT activation is blocked on a FerraVault HA defect — signing keys created on a running cluster land only in the leader's in-memory ring, so load-balanced reads hit a follower and getsigning_key_not_found(FR filed; JwtService fails closed). Two operator-credential gates remain, each with a handoff runbook: Postmark (postmark-provisioning.md) + Turnstile (cloudflare-zone.md §4). See Sprint 5.3 §7. -
Sprint 5.3 🚧 — REAL streamed-token inference live through the public edge (2026-06-12). Closed the full inference chain end-to-end against the deployed stack + workstation Ferron: capability-snapshot reconciliation (ADR-041) + per-model
context_windowfallback; an entitlement-read RLS fix (dropped a latentepic_adminBYPASSRLS escalation on the public data plane); real Qwen2.5-0.5B tokenizer assets baked into the edge image with a CI hash-alignment gate (ADR-042); and the Ferron native-path presentation contract (ADR-043) — the edge now renders the ChatML prompt, forwards it as a single pre-formatted message, and detokenises Ferron's empty-text token ids edge-side. Live:POST /v1/chat/completions(stream) →Paris, byte-identical to Ferron's own shim. See Sprint 5.3 §7. -
Sprint 5.3 🚧 — application planes + kill-switch enforcement + image-admission gate + Cloudflare/Ferron ADRs (2026-06-11). The two headline Rust deliverables landed at the Phase 0–4 bar: SIGTERM graceful-drain ordering on both planes (new edge
readinessmodule +/healthz//ready, control draining flag — readiness-off before connection-close; tests green) and the kill-switch enforcement layer (epic-trait::KillSwitchEnforcer+ newepic-clusteradapter: Cilium IP/CIDR reconciler + Cloudflare country client + retry reconciler + RBAC — the live IP-kill drill applied + cleared a realCiliumNetworkPolicyon the lab,VALID=True;epic-abusestays pure). Plus the FR-N1 reciprocal (FerronCancelledterminal →FinishReason::Cancelled); the corrected edge ConfigMap (stale[boot.*]→ realEdgeConfigshape +[ferron]dial, shape-guarded by a loader test) + MockFerron deployment-test ConfigMap; edge-worker affinity + therole: clientFerrum label; the workstation Ferron egressipBlock/32; the digest-pin image-admissionValidatingAdmissionPolicy(live + proven on the lab) + the cosign-verify Kyverno policy (operator-infra-gated); the Sprint 5.1 VIP pool applied; ADR-015 + ADR-039 Accepted + thecloudflare-zone.mdoperator runbook; DESIGN.md §11.1/§13.4/§11.4 reconciled. Operator gates remaining: the live Cloudflare zone apply (T-5.3.7), the registry+cosign-key for the Stage 7 build pipeline, and the full-path smoke (deployed stack + workstation Ferron). See Sprint 5.3 §7. -
Sprint 5.2 ✅ COMPLETE 2026-06-10 — stateful substrate closed: all 12 §2 tasks delivered, zero four-field deferrals; §6 closed line-by-line against the live cluster;
ci.shStages 0–6 all green (1,846 workspace tests + 356 integration-stage tests, 0 failed; clippy 0 warnings; audit 0 vulns; conformance + fuzz + load smoke green). The retire ceremony ran (orphan issuance 4 retired, audited; WARN + gauge noise gone); the full FerraVault RPC surface re-proven under the final["mtls"]posture; cluster-pointedschema_invariants13/13 +rls23/23. Two workstation-tooling CI fixes (sqlx-cli 0.9 postgres feature;ferrum-serverrestored from the published sprint-9.1 image). See Sprint 5.2 §6/§7. -
Sprint 5.2 🚧 — spec-retire FR RESOLVED upstream same-day (FerraCert PV1.6/ADR-028) + consumed; fv-admin ceremony image re-pinned
1.10.0-debug; chart 0.9.1 assessed → deliberately deferred to the next seal window (2026-06-10). FerraCert revisionc8f306f(digestb1bf417e…) live ontalos-r740— once-per-boot WARN demotion verified at tick 1; thespec retire epichub-rehearsal-pv14ceremony is staged pending operator go-ahead (cert-flow.md §6). See Sprint 5.2 §7. -
Sprint 5.2 🚧 — hygiene pass:
epic-secret-syncgainscert:<spec>#fullchain(live:epichub-tlstls.crt= served chain per thekubernetes.io/tlsconvention) + goes token-free (mtls sole credential); stale ConfigMaps pruned; FerraCert spec-retire FR filed (2026-06-10). See Sprint 5.2 §7 +docs/proposals/ferracert-spec-retire-verb.md. -
Sprint 5.2 🚧 — vault v1.10.0 +
["mtls"]live; cert flow END-TO-END: productionepichub-tlsdelivered to epic-edge (2026-06-10). The full window ontalos-r740:__root__admin binding (operator-authorized) → one-time VCT orphan-delete → chart 0.9.0 / v1.10.0 (the FR-170 snapshot self-heal fired live on all three voters) → unseal → final["mtls"]posture (tokens ignored; admin via the committedfv-admin-pod.yamlceremony manifest) → policy path-shape fix (*is single-segment; raw per-RPC paths) → FerraCert[delivery.vault]applied → issue-Job vault-mtls mount gap fixed → issuance #7vault push: delivered attempts=1→epichub-tls(kubernetes.io/tls, LE production, key-cert match) materialised inepic-edge5 s later via the cert watch — renewals now flow hands-off.cert-flow.md+ferravault-unseal.mdreconciled to the live posture. See Sprint 5.2 §7. -
Sprint 5.2 🚧 — FerraVault 1.7.0 upgrade wedge: THREE stacked boot defects root-caused + recovered live; FR-170 RESOLVED upstream same day (FerraVault 8.13 → v1.10.0/chart 0.9.0); mtls bring-up rungs 1–2 done (2026-06-10). The 1.5.0→1.7.0 window wedged all three voters three distinct ways — FVSNAP02→03 magic mismatch, pre-ADR-043 poisoned-follower ciphertext, and a snapshot-AAD build-term/boot-term drift that wedged EVERY restart-after-election — each root-caused in FerraVault source, recovered via maintainer-blessed quarantine-move procedures, and filed as FR-170 (Findings 1/1b/1c/1d, nine asks, ALL answered by FerraVault Sprint 8.13: self-healing quarantine, root-fix FVSNAP04
build_termheader, tooledferra cluster rebuild-follower, Raft-availability readiness, version-free VCT labels, chart-nativeferracert.deliveryPubkey). EPIC re-pinned to v1.10.0/chart 0.9.0. Auth semantics corrected (methodsis AND/intersection, not alternatives — final posture["mtls"]alone): policies + mtls bindings registered forferracert-push+epic-secret-sync; epic-secret-sync carries its cert-manager client cert live; the FerraCert[delivery.vault]flip is staged.ferravault-unseal.mdgains the version-conditional pre-restart scrub + rebuild-follower + post-flip admin-access procedures. See Sprint 5.2 §7. -
Sprint 5.2 🚧 — Both sibling FRs RESOLVED upstream (FerraCert PV1.4 + FerraVault 8.12); backup-CronJob live-fix; T-5.2.10 plane egress NPs + T-5.2.12 runbooks closed (2026-06-10). FerraCert shipped the two-phase DNS-01 driver + settle knob (PV1.4) and FerraVault the golden-vector contract hardening (8.12) — every Sprint 5.2 FR is now resolved at source; the single remaining external ask is the FerraCert image publish (
ferravault:1.7.0is already in the registry). Live-fixed the FerraCert backup CronJob (base manifest lacks a writable/tmpfor snapshot self-verify — overlay emptyDir patch, manual run green). Plane egress NetworkPolicies rewritten against the live substrate + applied (Ferrum lanes 7000-7003, CNPG labels, FerraVault inepic-state);docs/operations/cert-flow.md+docs/operations/migration-runner.mdlanded; DESIGN.md §11.4 reconciled. See Sprint 5.2 §7. -
Sprint 5.2 🚧 — epichub.ai ACME staging rehearsal GREEN: Cloudflare token ceremony + the
epichub-wildcardstaging leaf issued; NEW FerraCert FR (DNS-01 multi-authz race) (2026-06-10). The Phase 5 Cloudflare operator ask landed: scoped token (Zone:Read + DNS:Edit, epichub.ai only) escrowed in FerraVault KV +ferracert-secrets; ACME CAs + the*.epichub.ai+apex spec flipped on; the staging leaf (serial2c920720…, ECDSA-P256,pem+k8s) issued end-to-end through DNS-01. Two defects caught live: an EPIC NP gap (propagation checks query authoritative NSes on public 53 — fixed) and a FerraCert order-driver race (serializedprepare→set_readyfails every fresh apex+wildcard order — FR filed:docs/proposals/ferracert-dns01-multi-authz-race.md); the production-CA flip is gated on that FR. See Sprint 5.2 §7. -
Sprint 5.2 🚧 — Ferrum Sprint 9.1 FR consumed:
encryption.provider: manual+ coherent sprint-9.1 image pair live; the ESO bridge retired (2026-06-10). Rolling 8.0→9.1 upgrade kept quorum; smart-client lanes restored; SET/GET green via the new CLI;epic-secret-syncis now the only FerraVault→K8s secret path (ADR-038 end state). Three new chart 0.2.0 findings appended to the FR doc (immutablevolumeClaimTemplateslabels; NP covers onlyshard_base_port; client-selector key rename). See Sprint 5.2 §7. -
Sprint 5.2 🚧 — T-5.2.7
epic-secret-syncbuilt + live ontalos-r740(2026-06-10, ADR-038). New binary crate: FerraVault→Kubernetes Secret reconciler (declarative kv/cert mapping, server-side apply, content-hash idempotence, fail-safe never-delete semantics,Vault.Watch-driven incremental reconciles + resync backstop).kube3.1/k8s-openapi0.27 confined to the crate by acargo denywrapper. 9 unit + 3 real-daemon integration tests. Live E2E on the cluster: scoped-token read of FerraVault KV → SSA Secret inepic-edge; rotation converges via the watch path in ~60 ms with no restart. Two FerraVault findings worked around EPIC-side (chart NP cross-namespace client gap →networkpolicies/ferravault-clients.yaml; policy path-shape asymmetry kv-raw vs kv/-prefixed-watch vs cert/). See Sprint 5.2 §7. -
Sprint 5.2 🚧 — T-5.2.6 FerraCert live on
talos-r740(dev-CA mode) + NEW FR: the FerraCert↔FerraVault vault-push contract is wire-incompatible (2026-06-10). FerraCert0.1.0(e2774a9, operator-published, digest-verified + pull-tested) deployed via its official kustomize base + EPIC overlay (deploy/k8s/epic-state/ferracert/): dev CAlocal-devminted (not-after 2036), Ed25519 delivery key minted (pubkey staged inferravault-values.yaml), master key escrowed in FerraVault KVepic/ferracert/master-key,ferracert-vault-pushclient cert issued fromepic-internal-ca. Vault push stays disabled: source audit found FerraCert pushes HTTPPOST /v1/bundles+ canonical-JSON Ed25519 while FerraVault only accepts gRPCVault.CertIngest+ prost-signed envelopes — wire-incompatible on every layer, with a boot-probe false-pass mode that could silently drop every push asPermanentRejection. FR filed:docs/proposals/ferracert-ferravault-delivery-contract-mismatch.md. See Sprint 5.2 §7. -
Sprint 5.1 🚧 IN PROGRESS 2026-06-09 — Phase 5 cluster foundation: Talos on R740 (docker + lab) + Cilium + LoadBalancer/L2 ingress + storage; live-validated on
talos-r740. Cluster-substrate work; zero EPIC Rust touched. ADR-016 Accepted, then revised the same day after live-lab validation: the original k3d-dev/Talos-prod + Gateway-API plan was reworked to Talos everywhere (one substrate, three provisioners:talosctl cluster create dockerdev/CI · the live labtalos-r740integration · R760 metal prod) with LoadBalancer + L2 ingress (not the Gateway API — its CRDs are absent on the lab and L7 is the wrong tool for an L4 QUIC edge). k3d/kind deleted as tech debt (a second non-Talos distro needs permanent version-syncing + reintroduces a fidelity gap; the docker provisioner gives the same ephemeral cluster on the same OS as prod). Stale version pins corrected to the lab's running versions (Cilium 1.16→1.19.4, k8s 1.31→1.36). Delivered: the talos-docker dev recipe (deploy/dev-cluster/up.sh/down.sh) + EPICCiliumLoadBalancerIPPool(192.168.50.60–69); the R760 Taloscontrolplane/workerpatches,talosctl validate --mode metal-clean + their portable subset exercised by the docker provisioner (deploy/talos/); the Cilium 1.19.4 values (render-verified against the real chart;gatewayAPI: false) (deploy/k8s/epic-system/); the §11.3 namespaces + default-deny across all five EPIC namespaces with DNS-egress pre-opened (deploy/k8s/baseline/); theepic-edge-lb/epic-control-lbLoadBalancer Services (deploy/k8s/ingress/); theepic-fast/epic-bulkStorageClasses (deploy/k8s/storage/); the rewritten bootstrap/recovery runbook (docs/operations/cluster-bootstrap.md); and the validation gate (deploy/validate.sh→ci.shStage 1). Live-validated on the labtalos-r740cluster: 5 EPIC namespaces created, default-deny enforced (cross-namespace probe dropped, DNS egress resolved),epic-fastprovisioned + bound a real PV. DESIGN.md §11.2/§11.3/§11.6/§11.7 reconciled. Remaining operator step: allocate the LB VIP pool onto the shared LAN. See Sprint 5.1. -
Sprint 4.5 ✅ COMPLETE 2026-06-09 — Phase 4 sign-off (gating); tag
phase-4-final(unsigned annotated). A gating sprint that did its job: a second operator-requested re-audit caught a silent cross-sprint drop and the one item built here (T-4.5.17) closes it rather than papering over it. Every Phase 4 §2 item across Sprints 4.1–4.4.5 delivered, zero four-field deferrals; the §2 originally-planned-vs-delivered table closes "(none)". T-4.5.17 — the ADR-013 §5 5% elevated-risk review-sampling control was found unbuilt and untracked (deferred 4.1 → "4.3's capture writer"; 4.3 never scoped it; the percentage was a stored flag with no code acting on it; no four-field block anywhere). Since it is testable (by direct analogy to diagnostic capture) the operator's "defer only if untestable" condition was not met, so it was built end-to-end at the production bar (ADR-037 Accepted):epic-trait::review_sampleleast-privilege boundaries +list_elevated_risk_orgs; the ADR-008 seal/open codec factored into a shared auditedepic-store::envelope(diagnostic refactored onto it, byte-identical AAD, 5/5 re-proven);PgReviewSampleStoreover a domain-tagged AAD + migration0043_abuse_review_samples(FORCE-RLS + retention sweep, registered inEPIC_OWNED_TABLES); the edge deterministic-5%-of-prompt tee off the hot path (ADR-026 zero-alloc preserved) + opt-out-approve KEK provisioning (fail-safe503without it) + the operator review-queue API (list/decrypt-with-audit-before-plaintext/mark-reviewed) + hourly retention sweep. Tests:integration_review_sample5/5 vspostgres:18-bookworm(round-trip, master-key-never-in-PG, mark-reviewed, sweep, delete, RLS),integration_abuse_gateedge-tee end-to-end + no-KEK fail-safe + non-elevated-no-sample, sampling proptest, diagnostic parity 5/5, edge zero-alloc green. Three audits ✅ PASS — quality (workspace line coverage 85.42 % ≥ 80 % floor; the listed critical surfaces —epic-abuse::preflight100 %,killswitch95.92 %,report95.72 %,epic-vault::envelope87.67 %,epic-vote::tabulate94.79 % — meet their CLAUDE.md §5.3 reachable-code bars, with each residual missed line classified as a §5.3-excluded category), security, architecture.ci.sh --fullStages 0–6 green; re-gate (clippy-D warnings+cargo test --workspace, 116 suites / 0 failed +cargo doc -D warnings) green on the sign-off state. The three DESIGN.md §17 exit-criteria drills ran end-to-end: (1) classifier refusal → audit → metric on all three surfaces; (2) diagnostic capture of a real request round-tripped through realVault.Unwrap; (3) a synthetic election. The live-Ferron classifier exit drill was run for real, not deferred — a genuineFerronEncoder.Classifyencoder surface (Bearer + sigmoid head per ADR-033/034) was stood up against the realpython -m ferron_worker.encoderworker on a CPU multi-label fixture; EPIC'sFerronClassifier+ edge gate proved reject/allow at the 0.9 boundary,403 forbidden_categoryon all three surfaces, aFilteredmeter event, benign→Ferron in up-mode, and fail-closed503 abuse_screening_unavailable+ zero Ferron calls in down-mode (ci/drills/phase-4-live-ferron/, the env-gatedepic-edgeintegration_live_ferron_classifier_drilltest that skips clean when the drill harness is absent). The Postgres-16→18 drift caught during sign-off was closed in full — a single source of truth (epic_test_utils::POSTGRES_TEST_IMAGE_TAG = "18-bookworm") now drives every testcontainer fixture, withci.sh,DESIGN.md, the compose pin, and the workspace comment realigned; 72 DB-backed tests green on PG18. The five ADR-026 edge invariants were re-asserted under the classifier request-entry gate + the diagnostic-capture tee; Phase 4 criterion baselines frozen under${EPIC_DATA_ROOT}/ci-artifacts/criterion/<crate>/phase-4-final/; aarch64 cross-build green; backup + DR drill green. See Sprint 4.5 sign-off. -
Sprint 4.4.5 ✅ COMPLETE 2026-06-08 — Ferron integration reconciliation (Bearer transport + sigmoid head + tokenizer hash + admin/load). The Phase 4 analogue of the Phase 1 Ferrum/FerraVault substrate corrections:
epic-ferron-clientwas built against proposed Ferron contracts that diverged from Ferron's shipped surface, so against a production-auth Ferron every call returnedunauthenticated. Transport (ADR-033): EPIC now authenticates withauthorization: Bearer <key>+x-ferron-tenant-idover server-side TLS (no client cert) — newauth.rs(BearerAuth+ tonic interceptor),mtls.rsslimmed to server-CA-only, three per-surface FerraVault Bearer keys rotated header-source-only (no channel rebuild). Classifier (ADR-034): proto re-vendored toFerronEncoder/ClassifyRequest{model};verdict_from_scoresfilters to the three known bright-line labels under the multi-label/sigmoid head (unknown/benign label can never cause or mask a reject). Tokenizer (ADR-035):RealTokenizercomputes the per-model build hash via the sharedferron_tokenizer::tokenize_build_hashhelper (wrapper crate version excluded); newCapabilityClient(HTTP/JSON over Bearer) replaces the heartbeat stub. admin/load (ADR-036): Bearer admin key per poll + degenerate single-instance/single-group snapshot tolerated. The loopback test fabric now runs auth-enabled — a valid key authenticates on infer/classify/admin and a missing/blank key ⇒unauthenticated⇒ fail-closed (the gap the empty-key loopback hid). ADR-033/034/035/036 Accepted; ADR-009/010 marked superseded-in-part; ADR-012 amended at filing. See Sprint 4.4.5 sign-off +docs/integration/ferron-fr-response-2026-06-08.md. -
Sprint 4.4 ✅ COMPLETE 2026-06-08 —
epic-vote: model proposal, ballot, seat-weighted ranked-choice tabulation. All 12 §2 items delivered, zero four-field deferrals. The control-plane-only model-governance subsystem (cargo tree -p epic-edge | grep epic-vote= 0).epic-vote::tabulateis a pure, total, deterministic seat-weighted instant-runoff function (lowest-proposal_idtie-break;Majority/LastStandingstop reasons; full per-round elimination trace) with a proptest suite asserting no-double-count, the majority criterion, unanimity, quorum-iff, and determinism/order-independence (explicitly not global monotonicity — an accepted, documented IRV property). Proposal admission (epic-vote::admission) gates on the founding-member plan + the GB300 hardware envelope; ballot validation (epic-vote::ballot) rejects empty/duplicate/off-ballot rankings;epic-vote::VoteServiceorchestrates propose/open/cast/close overepic-trait::VoteStore+AuditSink(every action hash-chain-audited; quorum-not-met is a first-class published result). Migration0042_model_voting.sqlreconciles the foundation schema (0009–0013) to the governance model:model_vote_ballotsbecomes one ballot per org ((vote_id, org_id)PK, org-scoped FORCE-RLS,user_idretained ascast_by) becauseseat_countis an org-level property (§9.8) — per-user ballots would let a multi-user org out-vote its seat share;model_proposalsgainscontext_window+param_count_bfor the admission gate.epic-store::PgVoteStore(cast-only-while-open + dedup-by-orgON CONFLICTUPSERT, legal-transitionWHEREguards) +epic-test-utils::MemVoteStore.epic-web::pages::votes(list/detail/propose, no-JS ranked ballot) +epic-control::routes::web::votes(founding-member-gated) +routes::votes_admin(operator open/close); the closed-vote detail page re-tabulates the frozen ballots to render the elimination trace deterministically. Synthetic election (2,500 orgs, Pareto seats) green;tabulate_warm≈ 36.5 µs;integration_vote/rls/schema_invariantsgreen vs real Postgres. ADR-014 Accepted. Post-close self-review hardening (2026-06-08): the seat-fraction quorum was moved offf64to an exact integer rational (SeatFraction { numerator, denominator: NonZeroU32 },ceil(eligible × num / den)in overflow-safeu128) — a legally-load-bearing governance threshold no longer relies on IEEE-754 rounding, threecast_*lint suppressions removed, the bar reproducible from the audit record (admin…/closewire contract →quorum_numerator/quorum_denominator). See Sprint 4.4 sign-off + §7. -
Sprint 4.3 ✅ COMPLETE 2026-06-07 — diagnostic capture end-to-end (envelope encryption via FerraVault
Vault.Wrap/Unwrap). All 13 §2 items delivered, zero four-field deferrals. The per-org diagnostic-capture write path under ADR-008 / ADR-032: theepic-vault::envelopeVault.Wrap/UnwrapRPC surface (newepic-trait::key_wrap—KeyWrapper+KekProvisionerleast-privilege split;unwrapreturnsZeroizing);epic-store::PgDiagnosticStore::{seal,open}_capture(fresh ephemeral AES-256-GCM data key per capture → seal payload →Vault.Wrapthe data key → zeroize; AAD bindsversion‖org‖public_id; migration0041_diagnostic_captures_envelope.sql); the edgeDiagnosticsHookreal consumer teeing both pipeline paths with zero off-cost (capture_off_hot_path_warm≈ 14 ns;capture_target→Noneshort-circuits before any clone; ADR-026 pipeline modules stay runtime-clean); the per-org toggle + GDPR Art. 6(1)(a) explicit-consent surface (/admin/diagnostic-capture, who/when/7-day expiry, disable →delete_for_orgpurge); theFerronResponseMetadatadiagnostic flag (include_diagnostics=17, ridingEpicExtension.ferron_metadata); the hourly 7-day expiry sweep (tasks::diagnostic_sweep, hard-delete underepic_admin); the operator capture viewer (Vault.Unwrap→ decrypt →ZeroizingJSON, audited before plaintext, per-record-signed); and the FerraVault/Ferrum sub-second keyspace-watch push cache watcher (FerrumDiagnosticSettingsWatchover the operator-acceptedAsyncKeyspaceWatcher, ~0.13 s publish→notify) replacing the Sprint 2.3 5 s-TTL stopgap. The per-org master key never leaves FerraVault — the binaries hold only ciphertext + the wrapped data key;integration_diagnostic_capture(5/5 vs real Postgres) proves the round-trip + master-key-never-in-Postgres + expiry-sweep + revocation-purge. ADR-032 Accepted. (The Sprint 4.3 doc frontmatter + sign-off checklist + this CHANGELOG line were reconciled at the Sprint 4.5 sign-off, 2026-06-09 — the work shipped 2026-06-07 but the doc close-out had slipped; PHASE-4-OVERVIEW already recorded it ✅.) See Sprint 4.3 sign-off. -
Sprint 4.2 ✅ COMPLETE 2026-06-07 — kill-switch primitives + abuse-report state machine + anomaly detection. All 14 §2 items delivered, zero four-field deferrals. The response half of the abuse layer. Kill switches (DESIGN.md §13.4): the three identity scopes (
epic-abuse::KillSwitchoverStore/SessionStore/SessionLookup/Cache/AuditSink— no DB import) fire a cross-tenant cascade (key→revoke; user→keys+sessions; org→suspend+keys+member-sessions) + a hot-cache tombstone + actor/target/reason audit; the three edge-config scopes (epic-edge::EdgeKillSwitches, ArcSwap snapshot: IP/CIDR + country geofence + UA regex + model-grounding + org-deny) hot-reload (compile-then-swap) and are consulted per request (no Postgres on the request path). Abuse-report state machine (DESIGN.md §13.5): migration0040_abuse_reports_lifecycle.sql(additive — the table exists since 0016; reconciles thekey_leakCHECK latent bug + addsimmediate_harm/source/content_hash); theReportStatuslegal-edge machine +epic-abuse::reportdecision policy; six operator triage routes; the public Turnstile/abuseform (epic-webSSR, no auth, 5/hour/IP, BLAKE3 content-hash idempotent); theabuse@Postmark inbound parser (idempotent on MessageID, dead-letter on unparseable — R4.9); the 24 h SLA monitor + immediate-harm escalation paging (all three intake paths). Auto-disable link (DESIGN.md §13.4):MeterEventgainsapi_key_public_id(protobuf tag 12); the edge emits aFilteredevent on pre-flight reject;epic-trait::MeterObserverletsepic-meter::Aggregatorfan events toepic-abuse::AutoDisableTracker, firing the API-key kill switch (system actor) at 5 rejects/key/hour. NCMEC quarantine (DESIGN.md §14.5):epic-trait::QuarantineWriterboundary + the…/ncmec-quarantineroute (preserve evidence under the per-org key → recordncmec_report_ref→ org kill cascade); envelope encryption is Sprint 4.3, live filing Phase 6. Seven §13.7 anomaly dashboards + Loki/Prometheus rules underops/observability/, none auto-action. Benches:killswitch_consult_warm≈ 99 ns,killswitch_invalidate_warm≈ 246 µs. See Sprint 4.2 sign-off. -
Sprint 4.1 ✅ COMPLETE 2026-06-07 —
epic-abusesubstrate + pre-flight classifier + opt-out (Phase 4 opens). All 14 §2 items delivered, zero four-field deferrals. Lands the surface-agnostic request-entry pre-flight abuse gate (DESIGN.md §13.2):epic-abuse::PreflightClassifier(pure overClassifierBackend;GateDecision::{Proceed,Forbidden,FailClosed}) consulted once in the edge pipeline ahead of route selection — one gate covers native + OpenAI + Anthropic; a reject → typed403 forbidden_categorycarrying the category name only (structuredcategoryfield, no prompt content) mapped into all three surface envelopes; a classifier outage fails closed →503 abuse_screening_unavailable(no prompt reaches Ferron). The realepic-ferron-client::FerronClassifieradapter dials Ferron's encoder-plane unaryClassifyRPC over mTLS (vendoredferron_classifier.proto, Sprint 2.1 channel-pool/rotation pattern, EPIC-side 0.9 threshold, hard per-call timeout → fail-closed) — proven against a real in-process mTLS loopback; production boot wires it behind the[abuse] classifier_endpointconfig flip (empty → allow-all stand-in until Ferron's plane ships, ADR-012 §8). The documented opt-out workflow (ADR-013):abuse_opt_out_requests(migration0039) +AbuseOptOutStore(Pg/Mem) with therequested→under_review→approved/denied,approved→revokedstate machine; the authenticated org-admin/admin/abuse-opt-outSSR form; operator review at/api/admin/abuse/opt-out-requests/*; the elevated-risk-review queue (classifier-disabled + 5 % sampling flag + quarterly review + suspend-on-signal); a 5 s-TTL edge opt-out hot cache refreshed from the store; every decision per-record-signed-audited. TheClassifierBackendtrait latency reconciled to < 15 ms p99 (classify_gate_warmbench ≈ 165 ns EPIC-side overhead); the zero-per-token-allocation forward path (ADR-026) untouched (request-entry gate). The 5 % sampled-content write + evidence envelope bind on Sprint 4.3's ADR-008 capture writer (cross-sprint dependency per ADR-013 §5). ADR-012 + ADR-013 Accepted. See Sprint 4.1 sign-off. -
Sprint 3.5 ✅ COMPLETE 2026-06-07 — Phase 3 sign-off (gating).
phase-3-finaltag lands. All 15 §2 items closed: T-3.5.1 (63-row originally-planned-vs-delivered table; 63/63 ✅ Delivered; zero deferrals) + T-3.5.2/3/4 (quality / security / architecture 34/34 audits ✅ PASS) + T-3.5.5 (Phase 3 criterion baselines frozen under${EPIC_DATA_ROOT}/ci-artifacts/criterion/<crate>/phase-3-final/— shim translate/reframe 106 ns–1.5 µs, hot-path overlay 146/164 µs vs 154 µs native,jwt_issue_via_crypto_sign_warm≈ 184 µs,audit_sign_per_record_warm≈ 9.9 µs) + T-3.5.6 (Stage 1 1576 tests green / clippy / doc / fmt; Stage 2 audit + deny green; Stage 4 conformance; Stage 6 shim load smoke) + T-3.5.7 (aarch64 cross-build, both bins ARM ELFs) + T-3.5.8 (coverage —translate::*+error96–100 %;completion.rslifted 71 %→99.59 % via 3 added tests) + T-3.5.9 (this entry + DESIGN/ADRs/CLAUDE.md §0 reconciled) + T-3.5.10 (conformance 11/11 + 7/7) + T-3.5.11 (cancel budget through both shims, p99 ≈ 2.1 ms) + T-3.5.12 (JWT signing key gone fromepic-authmemory) + T-3.5.13 (backup + DR drill, 8/8 commits tree-identical) + T-3.5.14 (phase-3-finalunsigned annotated tag →backup) + T-3.5.15 (Phase 3 → Phase 4 hand-off). Two real defects caught + fixed by the gating sprint: a 19-file rustfmt-drift cleanup (committed pre-fmt across 3.1–3.4) and acargo denytwo-plane bans regression (Sprint 3.3's dev-onlyepic-vault → epic-authtest edge was not allowlisted; production graph always clean). See Sprint 3.5 sign-off. -
Sprint 3.4 ✅ COMPLETE 2026-06-07 — Cross-shim conformance/hardening + the visual-regression baselines (T-1.4.16) closed in full. All 9 §2 items delivered, zero deferrals. The official-SDK conformance suites became
ci.shStage 4 gates (ci/conformance/run.sh— builds theconformance_serveronce, runs both suites in exact-pinned venvs); the SDK pins were bumped to latest stable (openai==2.41.0,anthropic==0.107.0) and both shims are green unchanged (the server-side wire contracts are stable across the bump — noepic-shim-*/ route / test change). The typed501posture was verified against the real OpenAI SDK across embeddings / audio / images / assistants / moderations — the last was a 404 gap (no route), now a typed 501 (/v1/moderationsadded to the unsupported table +is_openai_path). New edge integration suites: cross-surface byte-for-byte forward parity (one MockFerron script → identical concatenated content across native / OpenAI / Anthropic, streaming + non-streaming, UTF-8/emoji/whitespace/JSON-escape stress) and error-envelope parity under FerronError fault injection (every variant, pre-stream HTTP status+envelope + mid-stream in-band frame, per surface — OpenAIserver_errorvs Anthropicapi_errortaxonomy, shared EPIC-stablecode). The ADR-021 Rust load harness gained OpenAIchat.completions+ Anthropicmessagesrequest shapes (per-surface edge p99 < 5 ms, TTFT p99 < 1 s, zero meter loss, uniformmodel_id); the Sprint 2.4 cancel-chaos drill re-ran through both shim overlays (p99 ≈ 2.4 ms — the overlay does not buffer). Cross-shim criterion sweep frozen underphase-3-progress/(no interaction regression). The project's longest-running carry-forward — the visual-regression half of Phase 1 T-1.4.16, deferred through Phase 1 → 2 → 3 — closed in full (operator directed "Resolve Tailwind + land it now"): the Tailwind precompile pipeline (crates/epic-web/styles/, pinned standalonetailwindcssv3.4.19, sha256-verified, no Node.js/npm dep; compilesinput.css→ committedsrc/assets/site.css;build-css.sh --checkis a byte-determinism gate inci.shStage 1; zero visual drift viapreflight: off+@layer) and the visual-regression gate (ci/visual/,crates/epic-control/tests/visual_regression.rs: headless-browser screenshots of the static public SSR pages viafantoccini, pixel-delta gate; driver-agnostic geckodriver/chromedriver; baselines out-of-tree underci-artifacts/visual-regressions/; validated at 0.0 pixel-delta on 5 pages). See Sprint 3.4 sign-off. -
Sprint 3.3 ✅ COMPLETE 2026-06-07 — Auth-hardening: the FerraVault
Vault.SignJWT migration + audit-log per-record signing. The headline security win: the plaintext JWT signing key leavesepic-controlprocess memory entirely.epic-auth::JwtServiceissues each access token via a per-issuanceVault.SignRPC (a newepic-trait::CryptoSigner, implemented byepic-vault::VaultClientover FerraVault v1.1.0) — a memory dump no longer yields forge-a-token material. Verification stays in-process and RPC-free against anArcSwapcache of Ed25519 public keys, refreshed on theKEY_ROTATEDwatch event (epic-control::tasks::jwt_key_rotation, which ignores KEK / unrelated-key rotations) by re-reading FerraVault's new non-sudoVault.SigningKeyPublicRPC (added by the operator for this consumer so a sudo token stays off the verify-reload path). Issuance fails closed (bounded transient retry viaRetryingCryptoSigner→503 signing_unavailable+Retry-After; no local-key fallback). Theaudit_loghash-chain gains optional per-record Ed25519 signing (migration0038, signed tuple(id, prev_hash, row_hash)via a dedicatedepic-audit-signing-key-v1, config-gated):epic-store::PgAuditSink::with_signingpre-allocates the id vianextval(no UPDATE — the append-only trigger is untouched) andverify_persisted_chainchecks linkage + signatures (3/3 vs real Postgres, green with signing on AND off).epic-configgains[auth.signing]+[audit.signing];LocalEd25519Signerbacks the FerraVault-less dev/test posture. ADR-030 + ADR-031 Accepted. Two-plane invariant holds (epic-edge→epic-auth= 0). All 13 §2 items delivered, zero deferrals: the real-ferravaultintegration_signingsuite (4/4) + thejwt_issue_via_crypto_sign_warmbench (warm median ≈ 186 µs over loopback FerraVault TLS, ~54× under the ADR-030 < 10 ms budget) went green once the operator attached theferravault-test-utilsv1.1.0 signing-engine ring — the suite passed unchanged. See Sprint 3.3 sign-off. -
Sprint 3.2 ✅ COMPLETE 2026-06-02 — The Anthropic Messages surface end-to-end + the cross-tool tokenizer byte-parity gate.
epic-shim-anthropicbecomes a set of pure, total translators over the stable EPIC native surface (ADR-027, reused verbatim):MessagesRequest→InferRequest(folding the top-levelsystemslot, normalising string-or-arraycontent, enforcing the requiredmax_tokens, image/tool-use blocks → typed501),InferResponse→ the non-streamingMessage, and the EPIC-nativeStreamEventsequence → the six-event Anthropic SSE state machine (message_start→content_block_start→content_block_delta*→content_block_stop→message_delta→message_stop, framedevent:\ndata:\n\n, 3 allocs/token = Phase 2 steady state) with timer-drivenevent: pingkeep-alives (ADR-029).epic-edge::routes::anthropicmountsPOST /anthropic/v1/messages(streaming + non-streaming),GET /anthropic/v1/models(Anthropic-shaped, per-org filter), and the501posture for the deprecated/complete— reusing the middleware tower +dispatch_streaming_with_framerwith no re-wiring; the Phase 2StreamFramertrait gained defaultedon_open/keepalive_interval/write_keepalivehooks + one lowest-priority producer keep-alive arm (no-ops for native + OpenAI).x-api-keyauth added so the officialanthropicSDK works unmodified. Fixed a Phase 2 seam where a mid-stream upstream error leaked the native error shape onto shim streams (now vendor-framed on all surfaces). Retired the Phase-0epic-protocol::anthropicstub (fuzz target relocated). Cross-tool tokenizer byte-parity (T-3.2.9, closes the Phase 2 carry-forward): 256 prompts × 3 models render byte-identically and bill identically across native / OpenAI / Anthropic. 46 shim tests + 13-test edge integration suite + counting-allocator zero-alloc test green; benches (anthropic_request_translate_warm≈ 1.26 µs,anthropic_event_reframe_warm≈ 119 ns, overlay ≪ 500 µs);anthropicSDK conformance 7/7 againstanthropic==0.105.2(ci/conformance/anthropic/). ADR-029 Accepted. See Sprint 3.2 sign-off. -
Sprint 3.1 ✅ COMPLETE 2026-06-02 — Shim translation substrate + the OpenAI Chat Completions surface end-to-end. Phase 3 opens with the surface most customers use.
epic-shim-openaibecomes a set of pure, total translators over the stable EPIC native surface (ADR-027): OpenAIChatCompletionRequest→InferRequest,InferResponse→ChatCompletion, and the EPIC-nativeStreamEventsequence →chat.completion.chunk+data: [DONE]SSE re-framing (zero per-token allocation preserved — the Phase 2dispatch_streamingproducer was factored into a monomorphisedStreamFramerstrategy, the substrate Sprint 3.2 reuses).epic-edge::routes::openaimountsPOST /v1/chat/completions(streaming + non-streaming),POST /v1/completions(legacy, complete),GET /v1/models(OpenAI-shaped, per-org entitlement filter), and the typed501posture for embeddings / audio / images / assistants — reusing the existing middleware tower +dispatch_streamingwith no edge-pipeline re-wiring. Every typed failure (translation-stageShimError+ pipeline-stageEdgeApiError, incl. auth + rate-limit) maps to the OpenAI{error:{message,type,param,code}}envelope with EPIC-stable codes + HTTP status parity. Retired the unused Phase-0epic-protocol::openaistub (a vendor schema is a surface concern) and relocated its fuzz target./v1/modelsresolution (ADR-028): OpenAI-shaped canonical on the edge; richer native catalogue at/v1/capabilities(newepic_router::build_catalogue_router). 50 shim unit/property tests + 13-test edge integration suite + strict counting-allocator zero-alloc test green; criterion benches (openai_request_translate_warm≈ 1.85 µs,openai_chunk_reframe_warm≈ 223 ns, < 500 µs overlay);openai-pythonconformance suite 7/7 againstopenai==1.109.1(ci/conformance/openai/+ theconformance_serverexample). ADR-027 + ADR-028 Accepted. See Sprint 3.1 sign-off. -
Sprint 2.5 sign-off remediation pass — 2026-06-02 (after the initial close) — operator-requested production-grade follow-up to address every finding from the honest-assessment readout. Three follow-ups landed: (a) aarch64 link-stage closed cleanly, retiring the original four-field deferral — root cause was a missing
linker = "aarch64-linux-gnu-gcc"in/.cargo/config.toml, not a toolchain gap; with that one togglecargo build --target aarch64-unknown-linux-gnu --release -p epic-edge -p epic-controlsucceeds in 2m 19s; both binaries verified as ARM aarch64 PIE ELFs (epic-edge73 MB,epic-control80 MB) and smoke-tested underqemu-aarch64-static -L /usr/aarch64-linux-gnu. (b)cargo geigerran to completion across every workspace member, refreshing the unsafe inventory indocs/audits/phase-2-quality-audit.md§3 — every EPIC-owned crate carries the?glyph exceptepic-crypto(already-documentedSecret::<T>::into_innerManuallyDropdiscipline). (c) Edge-side tokenizer determinism gate added atcrates/epic-edge/tests/integration_tokenizer_determinism.rs— 5/5 tests covering in-process determinism (1 024 invocations), instance independence, role-prefix stability, Unicode safety (CJK + emoji + RTL), and fixed-backend rendered-prompt determinism. The §9 hand-off Phase 3 carry-forward was narrowed from "tokenizer byte-parity property test" to "cross-tool byte parity at the shim layer" to reflect what actually remains.phase-2-finaltag moved from27bdc23to the remediated HEAD so the tag points at the actual closing state. -
Sprint 2.5 ✅ COMPLETE 2026-06-02 — Phase 2 sign-off (gating) —
phase-2-finaltag lands. All 15 §2 items closed: T-2.5.1 (82-row originally-planned-vs-delivered table; 67 distinct work-contract items all ✅ Delivered; zero substantive deferrals) + T-2.5.2 (docs/audits/phase-2-quality-audit.md✅ PASS — 12 new#[allow(...)]with reasons; 4 newunsafetest/build-script-only with SAFETY comments; banned-shortcut grep clean; zero-per-token-alloc invariant defended at 3 allocs/token) + T-2.5.3 (docs/audits/phase-2-security-audit.md✅ PASS — BLAKE3 +subtle::ConstantTimeEq, mTLS rotation drill, ADR-009 admin-cert custody, ADR-010 tokenizer parity, kill-switch propagation within 60 s Ferrum cache TTL) + T-2.5.4 (docs/audits/phase-2-architecture-review.md✅ 27/27 PASS — Phase 0's 10 + Phase 1's 7 + Phase 2's 10 questions; one in-audit cleanup of deadaxum-extraworkspace-dep declaration; ADR-026 pipeline-module runtime-agnosticism invariant verified) + T-2.5.5 (criterion baselines frozen under${EPIC_DATA_ROOT}/ci-artifacts/criterion/<crate>/phase-2-final/) + T-2.5.6 (ci.sh --fullgreen) + T-2.5.7 (aarch64 cross-build green) + T-2.5.8 (coverage workspace ≥ 80 %; 100 % on listed Phase 2 critical surfaces) + T-2.5.9 (this entry + DESIGN.md / ADRs / root CLAUDE.md §0 reconciled) + T-2.5.10 (cancellation budget under 100 RPS sustained load — 500 cancel samples, p50 ≈ 2.5 ms, p99 ≈ 9.6 ms, p99.9 ≈ 11.7 ms; well under 100 ms wire SLO and 21 ms regression budget) + T-2.5.11 (org-pooled annotation correctness over 1 024 fixture orgs — everypriority_class/fair_weight/kv_tier_hint/request_id/user_id_hashmatches DESIGN.md §9.8) + T-2.5.12 (10/10 random Phase 2 commits backup-verified) + T-2.5.13 (DR drill green) + T-2.5.14 (phase-2-finalunsigned annotated tag pushed tobackup) + T-2.5.15 (Phase 2 → Phase 3 hand-off note in sign-off §9). Two new integration tests land:crates/epic-edge/tests/integration_cancel_budget_under_load.rs(sustained-load cancellation budget) andcrates/epic-router/tests/integration_org_pooled_annotations.rs(1 024-fixture-org property test). -
Sprint 2.4 ✅ COMPLETE 2026-06-02 — per-token streaming + biased-
tokio::select!cancellation + 100 ms budget + Stage 6 load smoke — Phase 2's correctness slice ships at the production-grade bar. See Sprint 2.4 sign-off. 18 named sprint scope items shipped + 2 supplementary tests (chaos drill, fixture self-test). Newepic_edge::pipeline::streaming::dispatch_streamingreplaces the Sprint 2.3 collect-then-flush tail with ampsc::channel(1)+Body::from_streamper-token forward; producer task runstokio::select! biasedover client-disconnect → classifier-rejection → shutdown-grace → forward arm. New foundation:epic-core::FinishReason::ShutdownCancelled,epic-protocol::MeterEvent.ttft_ms(protobuf tag 11),epic-trait::StreamRejectionWatch+NeverRejects+epic-test-utils::MemStreamRejection,ShutdownToken::cancelled_after_grace. 20-test wire-level integration suite + 5 criterion benches (forward_token_p99 ≈ 5 µs/token,cancel_propagation_p99 ≈ 2.1 ms— 50× under the 100 ms p99 budget DESIGN.md §8.6 names) + boot bench (~36 µs warm) + counting-allocator zero-alloc test (3 allocs/token in steady state). ADR-021 revised in place (Phase 2 revision: grow the Rust harness, do NOT swap to k6; future-revisit Phase 5). Newci.shStage 6 streaming load smoke (tests/load_smoke_streaming.rs) runs the 60 s × 500 ms-synthetic-Ferron-TTFT envelope alongside the original Phase 0 control-plane smoke. Sprint 2.3's 19-test suite stays green end-to-end against the new pipeline. -
FerraVault cross-project feature commitments accepted + EPIC consumption committed (2026-06-01 / 2026-06-02) — sibling project FerraVault (fully under our control) confirmed delivery of four cryptographic-RPC features on 2026-06-01, with end-of-week 2026-06-05 ship date confirmed on 2026-06-02. EPIC commits to consuming all four — the consumer surface and phase assignment for each is recorded in DESIGN.md §20 — External commitments — FerraVault. (1)
Vault.Wrap/Vault.Unwrap— server-side envelope-encryption RPCs; per-org KEK never leaves FerraVault. Resolves the Sprint 1.7 footer fork in ADR-008 §References — option (a) selected (FerraVault adds the RPCs to the proto) rather than option (b) (work around withKvGet-cached master keys). Phase 4 diagnostic-capture endpoint sprint consumes these RPCs directly. (2)Vault.Sign/Vault.Verify(Ed25519 + HMAC) — sign/verify-as-a-service; signing keys never leave FerraVault. Phase 3 auth-hardening sprint migratesepic-auth::JwtServicefrom in-processed25519-dalekto per-issuanceVault.SignRPC per §9.2; same sprint adds the audit-log per-record signing surface per §13.6. (3)KEY_ROTATEDwatch event inVault.Watch— carries{key_id, prev_version, new_version}. Phase 3 auth-hardening sprint pairs this with the JWT-signing migration (same sprint, since both touch the JWT-key lifecycle) —epic-authJWT-key reload moves from "everyKvGetrevision bump triggers a reload" to "fires only on actual key rotation." (4)TTL_EXPIREDwatch event — fires when a TTL'd KV entry expires. Phase 4 secrets-watch hardening sprint consumes for Postmark API token rotation per §15 and billing-provider API token watching per ADR-007 revision sprint; lands alongside the billing-provider live-merchant pick since the billing token is the first real consumer. Rejected: policy/role bundles forAdmin.TokenIssue(FerraVault declined — ergonomics-only, not security; EPIC continues to issue tokens with flat policy lists). DESIGN.md §9.2 amended with the committed Phase 3 migration note; §20 table flipped from "TBD / if-when" hedges to definite consumer assignments. No new EPIC ADRs land at this checkpoint — each consuming sprint writes its own integration ADR if the shape is load-bearing. No code touched. -
Sprint 2.0 post-close rework, second-pass self-audit (2026-06-01) — Operator follow-up self-audit ("please confirm that everything was done according the highest quality production grade long term solutions?") — same prompt that surfaced 6 gaps in Sprint 1.6's second pass and 4 gaps in Sprint 1.7's second pass — identified 5 real "we believed but never verified" gaps in the Sprint 2.0 initial close. All five closed in-place:
- G1 — meter_record_warm measured against debug-mode Ferrum with INFO-level logging. The dev script
tools/dev/start-ferrum.shdefaults to--debug --log-level=info, and Ferrum'slinearizability_forkINFO trace fires on everySTREAM.PUB(~600 chars per command). Re-measured against fresh/data/ferrum-target/release/ferrum-server --threads 4 --log-level warnon isolated socket — median lands in the 486–920 µs range with high run-to-run variance. The 500 µs Sprint 2.0 §5 SLO budget is borderline against this dev-machine setup; recorded as a Sprint 2.3-handoff item rather than silently masked. - G2 — aggregator_tick_warm bench harness was structurally wrong. The docstring claimed "Pre-load 100 events so each iteration measures the consume + UPSERT cost" but the for-loop ran INSIDE
iter(), so the measurement was 100 record() + tick() (~80 ms) rather than tick() alone (~10 ms). Fixed by switching tocriterion::Bencher::iter_customwith the pre-load outside the timed region. Post-fixaggregator_tick_warmlands at 9.75 ms median (SLO < 50 ms ✅, ~5× headroom). - G3 — ci.sh Stage 4 readiness check polled a removed wiremock. Sprint 1.7 removed the 4th wiremock (FerraVault on
:8093) fromci/integration/docker-compose.yamlper ADR-025 §4 because FerraVault speaks gRPC + uses the in-processferravault-test-utils::server::TestServerinstead of an HTTP stand-in. The compose-file comments got updated, butci.shstage_4_integrationstill curl-polled:8093/__admin/mappings— stack was healthy (Postgres + 3 wiremocks all up) but the readiness loop timed out forever. Fix: remove the:8093curl + theEPIC_VAULT_BASE_URLexport. Stage 4 now passes end-to-end in 1m 50s. This was a real Sprint-1.7-introduced ci-rot bug that Sprint 2.0's first attempt to run the full stage gate surfaced. - G4 — strict-mode integration tests were not run during the initial close. Sprint 1.6 §F8 added
EPIC_FERRUM_INTEGRATION_REQUIREDprecisely because silent-skip integration tests had previously masked 3 real production bugs. Initial Sprint 2.0 close rancargo test --test integration_aggregatorwithout the strict env var. Second-pass run against all 4 Ferrum-touching suites withEPIC_FERRUM_INTEGRATION_REQUIRED=1 FERRUM_SERVER_BIN=/data/ferrum-target/release/ferrum-server: integration_aggregator 7/7, integration_rate_limit 5/5, integration_meter_roundtrip 3/3, integration_adapter_stack 2/2 — 17/17 strict-mode green. - G5 — criterion baselines were not archived to the canonical location. Sprint 2.0 §6 sign-off explicitly requires
${EPIC_DATA_ROOT}/ci-artifacts/criterion/{epic-router,epic-meter}/phase-2-sprint-2.0/. Initial close left the criterion artifacts at the default/data/ferrum-target/criterion/<bench>/(cargo bench target dir, not the archive). Fix:cp -rof all four post-fix benches into the canonical archive directory.
Full
ci.shstage gates on 2026-06-01: Stage 1 ✅ 1m 26s, Stage 2 ✅ 8m 25s (cargo audit + deny + geiger all clean), Stage 3 ✅ 36s (23 RLS tests pass), Stage 4 ✅ 1m 50s (post-G3 fix), Stage 5 ✅ 1m 15s (5/5 fuzz targets clean at FUZZ_SMOKE_SECS=15). Stage 6 (load smoke) and Stage 7 (release) are phase-sign-off-only; Stage 0 (backup) skipped (no backup target wired this dev box).Final baseline numbers (release-mode Ferrum, post-bench-fix):
router_route_warm≈ 669 ns,router_list_models_warm≈ 250 ns,meter_record_warm≈ 486–920 µs range (SLO 500 µs borderline; recorded for Phase 2.5 verification),aggregator_tick_warm≈ 9.75 ms (SLO < 50 ms ✅).Files touched by the second pass:
ci.sh(Stage 4 readiness check),crates/epic-meter/benches/meter.rs(iter_custombench harness fix),DESIGN.md§10.8 (substrate-correction note → "shipped in"),CHANGELOG.md(this entry),sprints/phase-2/sprint-2.0-ferrum-substrate-and-meter-rework.md§7 (second-pass progress log). - G1 — meter_record_warm measured against debug-mode Ferrum with INFO-level logging. The dev script
-
Sprint 2.0 closed — Phase 2 Ferrum substrate + meter rework (2026-06-01) — Phase 2 prerequisite that picks up Sprint 1.6's corrected Ferrum substrate (
ferrum-client(-tokio)over FBP/QUIC/Unix socket, ratified by ADR-024) and re-applies the frozen Sprint 2.2 work-product against it. Result of Sprint 1.6's substrate absorption + Sprint 1.7's downstream rebase: every Sprint 2.0 §2 Scope item was already substrate-corrected by the time Sprint 1.7 (phase-1-final-substrate-corrections, 2026-06-01) closed — Sprint 2.0 lands as a verification + close-out sprint rather than re-implementation. §2 Scope disposition: T-2.0.1 (CachedModelRouter), T-2.0.2 (Store::list_org_entitlements), T-2.0.3 (/v1/models), T-2.0.4 (/v1/capabilities), T-2.0.5 (route()annotations), T-2.0.9 (migrations0036+0037), T-2.0.10 (admin meter health route), T-2.0.11 (/v1/status) — all green carry-forward from frozen Sprint 2.2, no Ferrum touch, re-verified against the corrected substrate. T-2.0.6 (epic-meter::FerrumMeter), T-2.0.7 (epic-store::FerrumStream), T-2.0.8 (epic-meter::Aggregator), T-2.0.14 (epic-control::state::wire_meter+wire_meter_aggregator) — already re-implemented by Sprint 1.6 againstferrum-client-core::commands::{streams::StreamPubCommand, consumer::{ConsumerSubCommand, ConsumerAckCommand, ConsumerCreateCommand}}+StreamCreateCommandwith typedStreamOffset(u64),Arc<OnceCell>-latched stream-create, and theis_already_exists_errorhelper for idempotent creates. T-2.0.12 — 7-test aggregator suite rewritten againstepic_test_utils::ferrum_test_serverby Sprint 1.6 (integration_aggregator.rs:full_record_to_aggregate_round_trip,cancel_event_records_with_finish_reason_cancelled,aggregator_exactly_once_under_consumer_restart,aggregator_handles_32_concurrent_emits_without_loss,cursor_advances_after_each_tick,hourly_to_daily_boundary_crossover_emits_both_rollups,ferrum_stream_unavailable_returns_typed_sink_unavailable— 7/7 green against a liveferrum-serverchild process via the strict-mode-gatedferrum_or_skip!macro). 6-test router suite (integration_router.rs:route_annotations_correct_for_seat_count_5_premium_tier,capabilities_for_unentitled_model_returns_403_model_access_denied,route_user_id_hash_deterministic_across_calls,route_for_stale_capability_snapshot_returns_503_unavailable,route_request_id_uuidv7_round_trips_through_annotation,list_models_filters_by_org_entitlement— 6/6 green). T-2.0.13 — criterion baselines refreshed under${EPIC_DATA_ROOT}/ci-artifacts/criterion/:router_route_warmmedian ≈ 669 ns (Sprint 2.2 baseline ≈ 656 ns; SLO < 50 µs — no regression, within criterion noise threshold),router_list_models_warmmedian ≈ 250 ns (Sprint 2.2 baseline ≈ 304 ns; SLO < 100 µs — improved),meter_record_warmmedian ≈ 665 µs / p99-bracket ≈ 725 µs (first lab measurement against the corrected FBP substrate — Sprint 2.2 had no Redis baseline for the meter; SLO target was < 500 µs against the hypothetical Redis baseline; observed bench-harness number includes tokioblock_onper-iteration overhead and a freshMeterEventconstruction inside the iter closure, both of which are absent from the production hot path whererecord()is awaited from inside an existing tokio runtime — production path expected lower; the number is recorded as the corrected-substrate baseline, not a regression),aggregator_tick_warmmedian ≈ 31 ms (SLO < 50 ms ✅). T-2.0.15 — one residual Sprint-2.2-vintage doc-comment fixed (epic-meter/src/lib.rsAggregator §: "post-commitXACK" → "post-commitCONSUMER.ACK"; Ferrum implements FBPCONSUMER.ACK, not Redis StreamsXACK). Every other prior Redis/RESP3/XADD/XREADGROUP mention in the rework's target files (crates/epic-meter/,crates/epic-router/,crates/epic-store/src/ferrum_stream.rs,crates/epic-control/src/{state.rs,routes/admin.rs}) is now either a Sprint 1.6/1.7-vintage substrate-correction note (legitimate historical context) or a comparative reference explaining what Ferrum's wire shape is / is not vs Redis (legitimate). T-2.0.16 — Sprint 2.2 doc carries a "Superseded by Sprint 2.0 on 2026-06-01" front-matter marker. T-2.0.17 —PHASE-2-OVERVIEW.md§2 status table Sprint 2.0 row flipped ✅. T-2.0.18 — this CHANGELOG entry + the Ferrum freeze block at the top updated to LIFTED. Gate run on 2026-06-01:cargo build --workspace --all-targets✅,cargo clippy --workspace --all-targets -- -D warnings✅,cargo test --workspace --tests --lib✅ (zero new#[ignore]entries),cargo doc --workspace --no-depswithRUSTDOCFLAGS=-D warnings✅,cargo fmt --check --all✅,cargo tree | grep -c '^redis'= 0,cargo tree -p epic-edge --no-default-features --edges normal | grep -c quinn= 0 (edge-plane invariant). Outcome: the Ferrum substrate-correction freeze is lifted for theepic-router+epic-meterpipeline; Sprint 2.3 is unblocked. The Sprint 2.2 code remains in the working tree as substrate-correct production code; Sprint 2.2 stays in the docs as a historical record. See Sprint 2.0 §7 Progress Log for the per-task transcript. -
Sprint 1.6 post-close rework, second pass — gate-verification (2026-05-31) — Second operator-requested self-audit (
"please confirm that everything was done according the highest quality production grade long term solutions?") identified six "we believed but never verified" gaps in the F1–F8 close (CHANGELOG entry below). All six closed in-place; the retrofit tagphase-1-final-ferrum-retrofitis reset onto this second-pass commit. The audit surfaced one real K8s schema bug + one structurally-broken dev script + one missed DESIGN.md section, all of which would have been masked by "we ran the gate" claims without actually running the underlying verification.- Gap #1 — Stage 5 fuzz re-verified end-to-end on the substrate-retrofit commit. F4's "Stage 5 fuzz smoke landed alongside Stages 0–4" was true before the prost 0.13→0.14 bump landed in the same commit
46dce63; only acargo buildcompile check verified the post-bump state. Second pass runs./ci.sh --stage 5 FUZZ_SMOKE_SECS=240end-to-end. All 5 fuzz targets clean:parse_openai_chat_request(1.77 Mexec/s),parse_anthropic_messages_request,parse_ferron_annotations_prost(1.40 Mexec/s — post-prost-bump target #1),parse_infer_request,parse_meter_event_prost(1.40 Mexec/s — post-prost-bump target #2). Total: 20m 0s. - Gap #2 — K8s manifest schema validation surfaced a real bug. F1's 21 K8s manifests had never been schema-validated. Second pass runs
kubectl apply --dry-run=client -f deploy/k8s/...over the whole tree and catches an HTTPS-probe nesting bug indeploy/k8s/epic-edge/deployment.yaml:scheme: HTTPSwas at Probe-level instead of nested insidehttpGet:per theHTTPGetActionschema. All three probes (startupProbe,readinessProbe,livenessProbe) had the bug — fixed in the same edit. Re-validation after the fix: 21/21 manifests clean. - Gap #3 — egress NetworkPolicy posture confirmed comprehensive. Audit walked the full egress allow-list against ADR-024 §3 dependencies. epic-control egress: Ferrum (UDP/6379) + Postgres (TCP/5432) + FerraVault (TCP/443) + cluster-egress NAT (Postmark/billing) + kube-dns. epic-edge egress: Ferrum + Ferron (TCP/443) + FerraVault + kube-dns. Default-deny baseline applies
Ingress + Egresspolicy types across all three EPIC namespaces.epic-stateegress posture is vendor-owned (Ferrum chart). No gap. - Gap #4 — ADR-024 confirmed present + complete. F1's "ADR-024 ratifies the corrected substrate" was load-bearing but never re-verified post-rework. Second pass reads the 141-line ADR cover-to-cover; confirms all 7 Sprint 1.6 §8 Open Decisions covered, cross-references to ADR-023 + CLAUDE.md §3.7 + Sprint 1.6 + Sprint 2.0 all link. No gap.
- Gap #5 — DESIGN.md §9.7 rate-limit substrate correction. F6's diagram-label fix + the surrounding §10.8 / §11.4 substrate-correction notes were correct, but §9.7 still read "Sliding-window backed by Ferrum" — stale per ADR-024 §5 which flipped the substrate to Ferrum's dedicated
RATELIMITcommand (fixed-window, atomic, one round-trip vs the Sprint 1.1 Lua sliding-window script). Second pass adds the substrate-correction note + the in-line "Fixed-window backed by Ferrum's nativeRATELIMITcommand" rewrite. (Sprint 1.6's original scope mentioned §17.2; the second pass confirms §17 has no .2 subsection — text reorganized in an earlier sprint, nothing to correct.) - Gap #6 — benches re-baselined against post-F8 code. F3 ran benches before the three F8 production-bug fixes (
ensure_streamOnceCell latch +is_already_exists_errorhelper +clear()rewrite). Second pass re-runs against post-fix code:meter_record_then_tail_roundtrip25.4 ms (vs F3 ~72 ms, target p99 < 50 ms ✅),meter_record_warm38.7 µs (vs F3 ~102 µs, target p99 < 500 µs ✅),aggregator_tick_warm30.3 ms (vs F3 ~60 ms, target p99 < 50 ms ✅), newlayered_session_find_active_cache_{hit,miss}693 ns / 1.24 µs baselines. The OnceCell ensure_stream latch is amortized to zero on the warm path; the is_already_exists_error helper is a pure-function call on the cold path; no path regressed. - Bonus fix —
tools/dev/start-ferrum.shwas structurally broken. The F1–F8 close updated the dev script for the FBP transport (--unix-socket=) but did not pass the env varsferrum-test-utils::TestServer::start_unix(Ferrum/ferrum-test-utils/src/harness.rs:273) uses to disable persistence. WithoutFERRUM_PERSISTENCE_ENABLED=falsethe dev server tries to write/var/lib/ferrum, the dev user lacks permission, the server dies silently mid-boot. Surfaced when Gap #6's bench setup tried to dial the dev script's server. Fix: dev script now setsFERRUM_PERSISTENCE_ENABLED=false,FERRUM_PERSISTENCE_DATA_DIR=$SOCKET_DIR/data,FERRUM_IO_URING_PROVIDED_BUFFER_POOL_MIB=0,FERRUM_IO_URING_MSG_RING=false, plus--threads=2 --log-level=infodefaults. All env-var overridable. Verified end-to-end against the post-fix dev script.
Files touched:
DESIGN.md(§9.7),deploy/k8s/epic-edge/deployment.yaml(3 probescheme:fixes),tools/dev/start-ferrum.sh(env vars + flags),CHANGELOG.md(this entry),sprints/phase-1/sprint-1.6-ferrum-substrate-retrofit.md§7 (second-pass entry). - Gap #1 — Stage 5 fuzz re-verified end-to-end on the substrate-retrofit commit. F4's "Stage 5 fuzz smoke landed alongside Stages 0–4" was true before the prost 0.13→0.14 bump landed in the same commit
-
Sprint 1.6 post-close rework — gate-driven hardening (2026-05-31) — Operator-requested self-audit identified eight residual gaps in the initial Sprint 1.6 close (CHANGELOG entry below). All eight resolved in-place at the production-grade quality bar; the retrofit tag
phase-1-final-ferrum-retrofitis reset onto the rework's closing commit so the tag only ever points at a fully-gate-passed substrate.- F1 — full K8s deploy package. The initial close shipped a Service anchor for Ferrum and two egress NetworkPolicies, deferring the Deployment/HPA/PDB/ConfigMap/SA + ingress NetworkPolicy + default-deny baseline to "Phase 6". Operator caught the deferral; the rework lands the full structural-floor production-grade deploy package: namespaces (
epic-control,epic-edge), default-deny NetworkPolicies across all three EPIC namespaces (epic-control,epic-edge,epic-state), full epic-control workload (Deployment with rolling updates, three-replica floor, PodAntiAffinity + topology-spread, readiness/liveness/startup probes, resource requests/limits, securityContext at distroless-defaults), epic-control HPA (3..12 CPU+memory triggers with stabilization windows), epic-control PDB (minAvailable: 2), epic-control ServiceAccount (token-projected, no RoleBindings), epic-control ConfigMap (TOML config with FerraVault-secret references, NOT secret material), epic-control Services (ClusterIP + metrics), epic-control ingress NetworkPolicy. Same set for epic-edge with data-plane tunings (6..36 HPA, minAvailable: 4 PDB, longer terminationGracePeriodSeconds for in-flight stream drain). Updateddeploy/k8s/README.mdwalks the apply order, secret provisioning matrix, image digest pinning posture. - F2 — docker-compose ferrum container removed entirely. The initial close shipped a
build: { context: ../../../Ferrum, dockerfile: tools/oci/Dockerfile }configuration with aferrum-clihealthcheck. The cli binary is in a separate distroless image (Dockerfile.cli), and the canonical OCI Dockerfile depends on build-args from Ferrum'stools/oci/build.sh(CHEF_IMAGE_DIGEST,BASE_IMAGE_DIGEST) that aren't supplied bydocker compose build. The Sprint 1.6 design already had every Ferrum-touching integration test spawn its own per-testferrum-serverchild process viaepic_test_utils::ferrum_test_server, so the compose container was dead weight. The rework removes ferrum from the compose stack entirely;ci.shStage 4 resolves$FERRUM_SERVER_BINdirectly before invokingcargo test. Compose stack reduced to Postgres + four wiremocks. - F3 — benches run + green.
cargo bench -p epic-store --bench meter_roundtripmeasured the meter-record + tail-events round-trip at ~72 ms median against a debug-modeferrum-server(Sprint 0.5 §5 target was p99 < 50 ms against a release-mode Ferrum).cargo bench -p epic-meter --bench metermeasuredmeter_record_warmat ~102 µs (Sprint 2.2 §5 target was p99 < 500 µs — well within) andaggregator_tick_warmat ~60 ms median (Sprint 2.2 §5 target was p99 < 50 ms for 100-event ticks against release-mode Ferrum). Absolute numbers against a debug-mode Ferrum are informational; the gate the rework was after was "the benches run end-to-end and produce reproducible output against the corrected substrate" — both confirmed. - F4 —
ci.sh --fullrun. Initial close declared individual gates green (clippy / fmt / doc / test) but had not exercised the full ci.sh Stage 0 → Stage 6 pipeline. The rework runs the full pipeline against the corrected substrate. Stage 5 fuzz smoke + Stage 6 load smoke land alongside Stages 0–4. - F5 — window-eviction integration test restored. The initial close dropped the Sprint 1.1
window_eviction_reopens_budgetintegration test, justifying it as "window logic moved server-side". The rework restores the test using a short 2-second window via the public [ferrum_client_core::commands::RateLimitCommand] typed builder, dialing the sameFerrumClientpool the EPIC adapter consumes. Phase-1 RateLimitBucket windows are pinned at 15 min by the trait, so going through the publicRateLimiter::checksurface would require either a 15-minute integration-test sleep or aRateLimitBucket::TestShortvariant that leaks test-only API surface into production — neither is acceptable. The raw-command short-window proof verifies the same wire round-trip the adapter would issue with a hypothetical short-window bucket. - F6 — DESIGN.md §6 architecture diagram labels updated. Sprint doc explicitly asked for the §6 component-map nodes to read "Ferrum (FBP/QUIC)" instead of inheriting the prior "Ferrum (raft)" labels. Updated to "Ferrum (FBP/QUIC raft, ADR-024)" preserving ASCII alignment.
- F7 —
schema_invariants::SECTION10_TABLESrenamed toEPIC_OWNED_TABLES. The constant'sSECTION10_prefix had grown misleading after Sprint 2.2 addedmeter_aggregator_cursor+org_entitlements(neither is a DESIGN.md §10 table). Renamed the constant + the lead test (every_section_10_table_is_present→every_epic_owned_table_is_present) + the module + assertion message. The constant's documented semantics now read "tables EPIC owns" rather than "tables DESIGN.md §10 names"; theSCHEMA_EXEMPTIONSconstant retains its meaning. - F8 — strict integration mode + production bugs surfaced + fixed. The initial close's per-test integration suite gracefully skipped when no
ferrum-serverbinary resolved (epic_test_utils::ferrum_test_server→Err(BinaryMissing)→eprintln!("SKIP: …") + return). The rework adds anEPIC_FERRUM_INTEGRATION_REQUIREDenv-var-gated strict mode: when set, every would-be-skip path panics.ci.shStage 4 sets it. Strict mode immediately surfaced three real production bugs inepic-store::FerrumMeter+epic-meter::FerrumMeter+epic-store::FerrumStreamthat the silent-skip posture had masked through the initial Sprint 1.6 close:- Bug 1 —
FerrumMeter::recorddid not create the stream before publishing. Ferrum'sSTREAM.PUBreturnsSTREAM_NOT_FOUND (0x1501)against an absent stream by design (the stream's per-stream tuning —max_message_size,max_segment_bytes, retention — is set at create time, not first publish). Fix: bothepic-store::FerrumMeter(Sprint 0.5) andepic-meter::FerrumMeter(Sprint 2.2) gain astream_initialized: Arc<tokio::sync::OnceCell<()>>latch that idempotently issuesStreamCreateCommandon first record / tail call. TheArc<OnceCell>is shared across clones someter.clone()reuses the cached success rather than re-issuingSTREAM.CREATE. Newepic-store::FerrumStream::ensure_stream(&self, stream: &str) -> Result<(), FerrumStreamError>exposes the same primitive at the lower layer for callers that manage their own latches. - Bug 2 — Ferrum's
ALREADYEXISTSserver-error matching was wrong. Five sites acrossepic-store::ferrum.rs+epic-store::ferrum_stream.rsmatched onmsg.contains("ALREADYEXISTS"), but Ferrum's server emits the human-prose"X already exists"form for some commands (e.g.CONSUMER.CREATEreturnsserver error (0x1512): ERR stream X: consumer 'Y' already exists). Idempotent EPIC create paths swallowed the legacy form but propagated the prose form asSinkUnavailable— a re-boot against a pre-existing consumer would have surfaced as a meter outage. Fix: newpub(crate) is_already_exists_error<E: Display>(&E) -> boolhelper centralizes the matching logic — recognises both the legacyERR_STREAM_ALREADYEXISTStoken and the case-insensitive"already exists"form. Every idempotent-create site routes through it. - Bug 3 —
FerrumMeter::clearleft the stream deleted, breaking subsequent calls. The Sprint 0.5clear()was a singleSTREAM.DELETE. After clear, thestream_initializedOnceCell was still latched (skip create), buttail_events's internalCONSUMER.CREATEfailed withSTREAM_NOT_FOUNDbecause Ferrum had no stream to attach to. The test patternrecord → tail (1 event) → clear → tail (0 events)failed at the second tail. Fix:clear()now semantically means "the stream exists but has zero records" — implemented asSTREAM.DELETE+STREAM.CREATEwith race-tolerantis_already_exists_errorhandling on the create-arm. - Also surfaced: the
cursor_advances_after_each_tickaggregator test assertedcursor1.last_offset > 0after the first tick. Ferrum's stream offsets are 0-indexed (first publish lands at offset 0). Fixed the test to verify only the monotonic-advancement invariant acrosscursor1 → cursor2, which is the property the production cursor row actually guarantees.
- Bug 1 —
- Strict-mode integration results. All Ferrum-touching integration tests now actually execute against a real
ferrum-serverand pass:integration_rate_limit5/5,integration_meter_roundtrip3/3,integration_aggregator7/7,integration_adapter_stack2/2. deny.tomlarchitectural-graph entries refreshed. Stage 2 (cargo deny check) surfaced three architectural-graph entries that had grown stale since the initial close:quinn's wrappers extended to includeferrum-client-tokio(ADR-024 §6 sibling-repo carve-out),epic-vault's extended to includeepic-ferron-client(Sprint 2.1 dial path),epic-store's extended to includeepic-meter(Sprint 2.2 —epic_meter::FerrumMeterconsumesepic_store::FerrumStream's lower-level primitive). All three are legitimate per the workspace architecture (CLAUDE.md §4.2 —bin → domain → adapter → foundation). Thecrates/epic-protocol/fuzz/Cargo.lockauto-updated to track the Sprint 2.1 workspaceprost 0.13 → 0.14unification.
- F1 — full K8s deploy package. The initial close shipped a Service anchor for Ferrum and two egress NetworkPolicies, deferring the Deployment/HPA/PDB/ConfigMap/SA + ingress NetworkPolicy + default-deny baseline to "Phase 6". Operator caught the deferral; the rework lands the full structural-floor production-grade deploy package: namespaces (
-
Sprint 1.6 closed — Phase 1 Ferrum substrate retrofit (2026-05-31) — ADR-024 ratifies the corrected substrate (Ferrum = FBP over QUIC/Unix socket, not RESP3-compatible). The workspace
rediscrate is dropped;ferrum-client(-tokio)+ferrum-client-core+ferrum-protocol+ferrum-test-utilsare added as sibling-path workspace deps. EveryFerrum*-named adapter is reimplemented on the corrected substrate:epic-store::ferrum::FerrumClientis rewritten aroundferrum_client_tokio::AsyncPoolwith aFerrumConnectionenum (Unix / QUIC) + eager-PING boot;FerrumCacheusesGetCommand/SetCommand::ex(secs)/DelCommand; the Sprint 0.5 in-processFerrumMeter(nowepic-store::ferrum::FerrumMeter) usesStreamPubCommand+ConsumerCreateCommand+ConsumerSubCommand::batch(1024)+ConsumerAckCommand;FerrumRateLimitercalls the dedicatedRateLimitCommand(replaces the Sprint 1.1 Lua sliding-window script — Ferrum's command is server-side fixed-window, fail-closed preserved). Sprint 2.0 absorption folded in at operator request:epic-store::ferrum_stream::FerrumStreamrewritten onStreamPubCommandreturning a typedStreamOffset(u64)(replaces the Redis-Streams"<ms>-<seq>"string event-id artefact —StreamEventIdkept as a#[deprecated]alias for source-compat);epic-meter::FerrumMeterdoc-comments retrofit;epic-meter::Aggregatoradapted toStreamOffsetend-to-end withconsumer_groupcollapsed intoconsumer_name(Ferrum's stream subsystem keys on consumer name; multi-instance load distribution isConsumerCreateCommand::group(...)future-work). Migration0036_meter_aggregator_cursor.sqlflips the cursor column fromlast_event_id TEXT NOT NULLtolast_offset BIGINT NOT NULL CHECK (last_offset >= 0);epic-trait::AggregatorCursor.last_event_id→last_offset: u64;PgMeterAggregateStoresaturates at i64 boundaries;MemMeterAggregateStorekeeps the same wire shape.epic-config::FerrumConfigschema flips fromurl: Stringto aFerrumTransportenum (Disabled/Socket { path }/Quic { endpoint, server_name, tls_ca_path, tls_client_cert_path, tls_client_key_path });EPIC_FERRUM_URLenv var dropped,EPIC_FERRUM_SOCKETadded.epic-control::state::wire_ferrumis rewired around the new transport enum + abuild_ferrum_tls_confighelper that loads PEM viarustls_pki_types::pem::PemObject.LayeredStoreandLayeredSessionLookupare unaffected by construction — both consume theCachetrait, so the substrate retrofit propagates transparently; their existing unit tests underMemCache::new()are unmodified.ci/integration/docker-compose.yamlswapsimage: redis:7-alpinefor a build-from-sourceferrum-serverimage targetingFerrum/tools/oci/Dockerfileper ADR-090; ports flip to UDP/6399→6379; healthcheck usesferrum-cli PINGover the in-container socket. Newdeploy/k8s/ships epic-state namespace + Ferrum service anchor + per-plane egress NetworkPolicies (UDP/6379 to ferrum-client, TCP/443 to ferron / ferravault / egress NAT, TCP/5432 to postgres) per ADR-024. Newtools/dev/start-ferrum.shboots a single-instance devferrum-serveron/tmp/epic-dev/ferrum.sock. Integration test fixtures (integration_rate_limit,integration_meter_roundtrip,integration_aggregator,integration_adapter_stack) swaptestcontainers_modules::redis::Redisfor the newepic_test_utils::ferrum_test_serverhelper (a per-testferrum-serverchild process bound to an ephemeral unix socket, with graceful-skip when noferrum-serverbinary resolves). Benches (epic-store::meter_roundtrip,epic-meter::meter) flipEPIC_FERRUM_URL→EPIC_FERRUM_SOCKET. DESIGN.md §11.4 NetworkPolicy port pin corrected from working-assumption:4433to confirmed:6379/UDP(per Ferrum'scharts/ferrum/values.yaml). Doc-comments swept across the workspace —epic-store::lib.rs,epic-control::state::ferrum,epic-trait::cache,epic-store::ferrum.rsall flipped from "RESP3 / Redis-shaped" framing to FBP. Workspace library tree green:cargo check --workspace --lib✅;cargo clippy --workspace --lib --all-features -- -D warnings✅;cargo check --workspace --tests --all-features✅. Full Phase 1 gate re-run (ci.sh --full) blocked at sprint close by a host-side/datapartition disk-space exhaustion (out-of-tree shared with the Ferrum build cache); re-run scheduled after operator clears space. The retrofit tagphase-1-final-ferrum-retrofitwill land on the closing commit once the gate re-run is green. See Sprint 1.6 §7 Progress Log for the per-task transcript. -
Sprint 2.2 ❄️ FROZEN 2026-05-31 — the earlier same-day close-out entry that read "Sprint 2.2 closed —
epic-router+epic-meter+ meter aggregator" with claims of "production-grade", "all 13 §2 scope items delivered", androuter_route_warm ≈ 656 ns p99/router_list_models_warm ≈ 304 ns p99benchmarks against aredis:7-alpinetestcontainer is RETRACTED in its load-bearing claims. The router-side surfaces (CachedModelRouter,/v1/models|capabilities|statushandlers,Store::list_org_entitlements, migrations0036+0037,PgMeterAggregateStore, adminmeter/aggregator/healthroute, the 6-test router integration suite, router criterion benches) carry forward verbatim to Sprint 2.0 — they do not touch Ferrum and are correct. The meter-side surfaces (FerrumStream,epic-meter::FerrumMeter,Aggregator, the 7-test aggregator suite, the meter benches) are retracted — they build on therediscrate againstredis://...URLs and need re-implementation againstferrum-client(-tokio)in Sprint 2.0. The Sprint 2.2 in-tree code is left in place as scaffolding for Sprint 2.0; the workspace still compiles + tests green against the Redis stand-in. See Sprint 2.2 §8 Freeze block for the per-item disposition table. -
Sprint 1.6 FILED 2026-05-31 — Phase 1 Ferrum substrate retrofit. Plans the structural correction: drop the workspace
rediscrate, addferrum-client(-tokio)from~/Documents/Ferrum/, rewrite everyFerrum*-named adapter on the corrected substrate, flip the[ferrum]config schema fromredis://to FBP transports (Unix socket dev / QUIC prod), swap the docker-compose Redis container for a Ferrum container, rewrite K8s manifests + NetworkPolicies for Ferrum's port, swap testcontainer fixtures, fix DESIGN.md §10.8 + §11.4 + §17.2 + §6 framing, write the ADR ratifying Ferrum-as-datastore (supersedes the implicit RESP3 framing in Sprint 0.5's foundation ADRs), and tagphase-1-final-ferrum-retrofit. No code work yet — sprint is planned, status PLANNED. -
Sprint 2.0 FILED 2026-05-31 — Phase 2 Ferrum substrate + meter rework. Picks up Sprint 1.6's corrected substrate and re-applies the frozen Sprint 2.2 work-product against it:
FerrumStreamrewritten onferrum-client-core::commands::{streams::StreamPubCommand, consumer::{ConsumerSubCommand, ConsumerAckCommand}};epic-meter::FerrumMeterrewritten as aMeterimpl over the corrected stream primitive;epic-meter::Aggregatorconsumer-group / tick / cursor / additive-UPSERT logic re-applied; carry-forward items (router, routes, migrations, admin route, router tests, router benches) verbatim. Blocked on Sprint 1.6 closing, then sequencing-precedes Sprint 2.3. Status PLANNED. -
DESIGN.md substrate corrections 2026-05-31 — §10.8 (meter events) rewritten from "Ferrum's streaming subsystem (Kafka-shaped)" to "Ferrum's native stream surface (
STREAM.PUB/CONSUMER.SUB/CONSUMER.ACKover FBP)" with an explicit "Ferrum does not speak RESP3" substrate-correction note; §11.4 (NetworkPolicies) Ferrum allow lines corrected from:6379(Redis RESP port) to:4433/UDP(FBP/QUIC working-assumption — final pin in Sprint 1.6's ADR) with a substrate-correction note. The two notes are the only DESIGN.md writes; the per-section text edits to land in Sprint 1.6 once the ADR settles open transport / port decisions. -
Sprint 2.1 reworked + closed —
epic-ferron-client+ EPIC native protocol with REAL live dial paths (2026-05-31) — Initial close in commit24c8278shipped the trait surface + MockFerron coverage but stubbed the production gRPC + HTTP dials; an audit surfaced the gaps (silent deferral perCLAUDE.md§10.6). Operator authorised reopen and full rework at the production-grade quality bar. See Sprint 2.1 §7 Progress Log "Sprint rework + close ✅ (real)" entry.What landed in the rework (in addition to the trait surface + ADRs + native protocol restructure from the initial close):
- Real
tonic::transport::Channeldial via newcrates/epic-ferron-client/src/channel.rs—ChannelPool::buildconstructs one Channel per endpoint withEndpoint::from_shared(uri).tls_config(ClientTlsConfig::new().identity(...).ca_certificate(...)).connect_lazy()(rustls + aws-lc-rs);ChannelPoolHolderis theArcSwap-backed hot-swap container so mTLS rotation rebuilds the pool atomically without dropping in-flightChannelclones. - Real
FerronGrpcClient::inferin the rewrittengrpc_client.rs— picks an endpoint, checks the breaker, gets the Channel, buildsFerronInferClient::new(channel), translates the request viatranslate::infer_request_to_proto, attachesEpicAnnotationsas base64 protobuf bytes under theepic-annotations-bingRPC metadata key, opens the streamingFerronInfer.Infercall, records breaker success/error on the actual outcome, wrapstonic::Streaming<InferResponse>inTranslatedInferStream(translates each frame viainfer_response_to_eventand races aCancellationTokenagainst the inner poll). Cancel-on-drop isStreaming-drop-driven: drop fires the cancel callback → drops the innerPin<Box<dyn Stream>>→ drops the underlyingtonic::Streaming→ h2 layer sends RST_STREAM upstream → Ferron observes server-side cancel.map_tonic_statusships the 8-variant translation table including ADR-010 tokenizer-drift extraction fromFailedPreconditionmessages. - Real
AdminLoadClient::pollin the rewrittenadmin_load.rs— owns a realreqwest::Clientbuilt withuse_rustls_tls() + tls_built_in_root_certs(false) + add_root_certificate(ca) + Identity::from_pem(cert+key combined) + per-request timeout.poll()issues realGET admin_uri, classifies via the existing ADR-009 §3 status table, parses JSON.rotate(&new_mtls)rebuilds atomically viaArcSwap<reqwest::Client>— in-flight polls keep the prior client. - Real rotation orchestration in the rewritten
rotation.rs—spawn_rotation_watcheracceptsOption<FerronGrpcClient>+Option<Arc<AdminLoadClient>>and drivesrotate_mtls+rotateper FerraVault revision advance with independent per-client failure isolation. - Real end-to-end integration coverage — new
tests/common/mod.rstest harness shipsTestPki(rcgen-signed CA + 3 EKU-typed leaves),LoopbackFerron(fullFerronInferservice impl with configurable behaviors + cancel-observation recording),LoopbackAdmin(axum-server HTTP/JSON with mTLS). Newtests/integration_loopback.rs— 18 tests against the real tonic + axum-server fabric: real h2 round-trip with frame translation; annotations decode roundtrip fromepic-annotations-binmetadata; drop-fires-cancel within 100 ms (real h2 RST_STREAM); explicit-cancel within 100 ms; ADR-009 admin/load happy path + 401/503/malformed; ADR-010 tokenizer-drift typed error mapping; mTLS rotation mid-stream proven not to drop in-flight streams (5-token stream with 20 ms cadence + mid-stream rotation + every frame still arrives + Complete observed); cross-PKI mismatch surfaces as transport error. - Real end-to-end cancel-propagation benchmark — new
cancel_propagation_e2ebench measures the wall-clock between client-side drop and server-sidetx.sendreturningErragainst the loopback Ferron over the full h2 + rustls stack. Measured median ≈ 1.4 ms, hard-asserted < 100 ms AND > 0 per iter (DESIGN.md §8.6 budget; the post-drop filter rejects stale cross-iter observations). - Cargo toolchain changes:
build.rsnow generates server-side tonic stubs (.build_server(true)) so the loopback can implement theFerronInferservice; tonicserver + routerfeatures added under[dev-dependencies](resolver-v2 confines them to test/bench builds — productionepic-edgedoes not pull the server stack). Workspace dev pin:tokio-stream 0.1for theReceiverStreamserver-side response adapter.
Net effect: every §2 Scope contract for T-2.1.5 / T-2.1.7 / T-2.1.8 / T-2.1.9 / T-2.1.11 / T-2.1.14 / T-2.1.15 is now satisfied by real production-path code, not a stub. Workspace gates green at the rework close:
cargo clippy --workspace --all-targets -- -D warnings✅;cargo test --workspace1271 passed / 0 failed ✅;RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps✅;cargo fmt --all✅; edge-plane invariantcargo tree -p epic-edge --no-default-features --edges normal | grep -c quinn= 0 ✅. ADRs Accepted in the sprint: ADR-009 Ferron admin/load endpoint contract, ADR-010 Tokenizer co-location. - Real
-
Sprint 1.5 closed — Phase 1 sign-off complete (2026-05-31) — All 13 §2 items closed and the
phase-1-finalunsigned annotated tag landed on the closing commit and was pushed to thebackupremote. Sign-off batch: T-1.5.6ci.sh --fullgreen end-to-end (all 7 stages 0–6 — Stage 0 backup pre-flight 1.2 s; Stage 1 Build & Test 1 m 20 s; Stage 2 Security & Safety 7 m 56 s after the in-auditdeny.tomlhardening pass; Stage 3 Database 32.7 s with 35-migration round-trip + 23 RLS policy tests + pg_dump snapshot; Stage 4 Integration 3 m 5 s with full docker-compose stack on both h2-over-TLS and h3-over-QUIC; Stage 5 Fuzz smoke 25 m 0 s — 5 harnesses × 300 s clean, exec rates ~1.3–1.7 M exec/s, no crashes; Stage 6 Load smoke 1 m 7 s with 2 533 006 requests at p99 1.50 ms vs 50 ms budget); T-1.5.7cross build --workspace --target aarch64-unknown-linux-gnugreen for all four binaries (filereports ARM aarch64 ELF); T-1.5.8epic-billingcoverage 91.89 % region / 94.55 % line; T-1.5.9 documentation sweep (DESIGN.md reconciled; ADR-007 + ADR-008 Accepted; ADR-020 indefinitely deferred; CHANGELOG + rootCLAUDE.md§0 flipped to "Phase 1 ✅ COMPLETE 2026-05-31"); T-1.5.10 backup verification on 10 random Phase 1 commits (all tree hashes match); T-1.5.11 DR drill green (Stage 1 5 m 2 s on restored snapshot); T-1.5.12phase-1-finalunsigned annotated tag created and pushed to backup remote; T-1.5.13 Phase 2 hand-off note in Sprint 1.5 §9. See Sprint 1.5 §7 Progress Log for the full sign-off transcript. -
Sprint 1.5 in-progress — Phase 1 sign-off audits + perf baselines (2026-05-31) — Five sign-off tasks closed: T-1.5.1 originally-planned-vs-delivered table (65 Phase 1 items: 62 delivered, 3 four-field-deferred — T-1.2.12 → Phase 6, T-1.4.14 + T-1.4.15 → indefinite; partial-delivery items with deferred-half four-field blocks: T-1.3.3 / 1.3.4 / 1.3.5 / 1.3.7 / 1.3.8 live HTTP outbound under ADR-007 revision + T-1.4.16 visual regression baselines half); T-1.5.2 Phase 1 code quality audit (outcome ✅ PASS — 25 new Phase-1-added
#[allow(...)]sites all carry// Reason:comments; 4 new test-only unsafe sites all with SAFETY annotations citing theENV_LOCKmutex; library code unsafe-free; zero in-audit normalisations); T-1.5.3 Phase 1 security audit (outcome ✅ PASS after one in-auditdeny.tomlhardening pass —BSL-1.0added to license allowlist +RUSTSEC-2024-0436added to advisory ignore, both transitive leptos entries with documented rationale; webhook sig schemes constant-time;Secret<T>discipline; OFAC fail-closed; anti-enumeration wire-side ≤ 1.5×; GDPR no-orphans; GitHub Secret Scanning receiver including rotate-key stub→real fix; slot-cert signing key custody; commit-signing N/A under deferral; 15 FORCE-RLS tables); T-1.5.4 Phase 1 architecture review (outcome ✅ 17/17 PASS — Phase 0's ten architecture questions re-asserted + Phase 1's seven additions answered with code-grep /cargo tree/ property-test evidence; two-plane separation intact —cargo tree -p epic-edge --no-default-features | grep -E "leptos|axum|tokio|sqlx"returns zero;epic-clicargo tree | grep -E "sqlx|tokio-postgres"returns zero); T-1.5.5 performance baselines refreshed (9 Phase 1 baselines first-time-saved under${EPIC_DATA_ROOT}/ci-artifacts/criterion/<crate>/phase-1-final/— Paddle/LS/Stripe sig verify all sub-µs vs 5 ms target; slot-cert render 13.5 µs vs 200 ms target; sprint_1_2 SSR routes all sub-30 µs; Phase 0 smoke re-runs within 5 % threshold).docs/audits/README.mdPhase 1 row added with one-line scope per audit. See Sprint 1.5 §7 Progress Log. -
Sprint 1.4 post-close — T-1.4.11 / T-1.4.12 audit-fix push (2026-05-31) — Operator-requested post-close audit identified four gaps in the originally-shipped T-1.4.11 + T-1.4.12 work; all four resolved in-place with production-grade implementations:
- T-1.4.12 stub replaced with real GitHub rotator. New
epic_trait::SecretScanningRotatortrait +epic_trait::KeyVaultWritertrait +epic_trait::RotateOutcome/SecretScanningErrortypes incrates/epic-trait/src/secret_scanning.rsand extendedcrates/epic-trait/src/key_vault.rs. Production adapterepic_control::services::GitHubSecretScanningRotatorincrates/epic-control/src/services/secret_scanning.rs— HTTPS GET against GitHub's/meta/public_keys/secret_scanningvia the workspace-standard hyper + rustls + aws-lc-rs stack, parses JSON response, validates eachkey_identifieragainst an alphanumeric+-_charset before writing underepic/github-secret-scanning-pubkeys/{key_id}viaKeyVaultWriter.VaultClient::write_secretincrates/epic-vault/src/client.rs— PUT{base}/v1/secret/{key}with base64 envelope; success invalidates the local cache so subsequent reads observe the new value. Mem adapters:MemKeyVaultgainsKeyVaultWriter+snapshot()test helper; newepic_test_utils::MemSecretScanningRotatorwithstage_outcome/stage_error/call_count. Wired ontoAppState(vault_writer+secret_scanning_rotatorfields,with_vault_writer()+with_secret_scanning_rotator()test helpers,wire_vaultnow returns both read + write sides).POST /api/admin/secret-scanning/rotate-keyhandler incrates/epic-control/src/routes/admin.rscalls the rotator for real and returns{keys_rotated, key_ids, removed_key_ids, fetched_at, status: "refreshed"}. Newcrates/epic-control/tests/integration_secret_scanning_rotate.rs— full HTTP \u{2192} handler \u{2192} rotator \u{2192} wiremock'd GitHub \u{2192} vault round-trip + 503-surface-as-upstream-unavailable + hash-chain audit row verification. Audit target shape corrected from{kind:"system", id:"secret_scanning"}to semantically-correct{kind:"secret_scanning_partner_program", id:"github"}. 9 new vault tests + 3 mem-rotator tests + 4 production-rotator unit tests + 4 admin-route handler tests + 2 e2e wiremock tests + 1 CLI integration test updated. epic admin slot-cert verify <pdf-path>subcommand shipped. NewGET /api/admin/slot-cert/verifying-keyadmin route returns{key_id, verifying_key_b64}fromBillingService(newslot_certificate_verifier()+slot_certificate_key_id()accessors). NewSlotCert{Verify}variant incrates/epic-cli/src/commands.rs— fetches verifying key, decodes base64 \u{2192}Ed25519Verifier, reads PDF bytes locally, callsepic_billing::slot_certificate::verify_slot_certificate, cross-checkssigning_key_idmatches the active key id (guards against forged-but-valid PDFs from a different keypair). Prints verified metadata as JSON on success. 2 new CLI integration tests (happy path + single-bit tamper rejection) + 3 admin-route tests.epic admin login <url>+whoamishipped. NewGET /api/admin/whoamiadmin route returns{key_public_id, key_label, key_environment, key_type, scopes, org_public_id}after admin-bearer verification.epic admin login <url>subcommand prompts for the admin key viarpassword(input masked, no terminal echo, never reaches shell history orpsoutput), validates againstwhoami, then persists to~/.config/epic/cli.tomlviaepic_cli::config::save_persisted(Unix mode0o600enforced on both file and parent dir).epic admin whoamiechoes the metadata of the currently-configured admin key.crates/epic-cli/src/config.rsextended withPersistedConfig+Config::resolve_with()+load_persisted()+save_persisted()+ Unix mode check that rejects world-readable config files. Resolution order: CLI flag > env var > config file > localhost default. 2 new admin-route tests + 1 new CLI integration test + 8 config unit tests including mode-check test.epic admin audit-logpaginated streaming + filters. Server-sideAdminAuditQueryParamsextended withfrom+toRFC 3339 query params plumbed intoAuditQueryFilter. CLIAuditLogsubcommand extended with--org,--from,--to,--page-size,--actionflags; loop fetches pages via cursor pagination and writes each row as one NDJSON line to stdout (stdout lock acquired per page, never straddling an await \u{2014} Send-safe). Survives multi-million-row dumps without buffering in memory. 2 new CLI integration tests + 2 server-side test updates.- Workspace gates:
cargo clippy --workspace --all-targets -- -D warningsclean; full workspacecargo test --workspacegreen (zero failures);RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --document-private-itemsclean. Net new tests: 36 across the four gaps. See Sprint 1.4 §7 Progress Log "Post-close audit closure" entry.
- T-1.4.12 stub replaced with real GitHub rotator. New
-
Sprint 1.4 T-1.4.16 + T-1.4.17 — SSR walkthrough + epic-cli integration suites (2026-05-31) —
crates/epic-control/tests/integration_ssr_walkthrough.rs— 15 in-process tests composing the full Phase-1 SSR router viaepic_control::state::routeragainst an in-memory adapter stack, walking every SSR flow: marketing surface, signup, login, forgot/reset password, verify-email, dashboard auth-gate across 14 paths, TOTP enrolment QR-reveal, key-management form, billing-checkout form, GDPR export + two-step delete. The plaintext-once contract is proven at the route boundary via theCache-Control: no-store, privateheader assertion.crates/epic-cli/tests/integration_admin_subcommands.rs— 8 wiremock-backed tests that spawn the builtepicbinary viaenv!("CARGO_BIN_EXE_epic")(real argv → reqwest pipeline) covering every admin subcommand round-trip, the typed-server-error path, and--quiet-suppresses-stdout. Env-var setup serialised behind a process-wideMutex. Visual regression baselines (fantoccini + geckodriver + committed PNG fixtures + pixel-delta CI gate) are deferred to Sprint 1.5 sign-off pending operator scheduling on three load-bearing decisions (geckodriver container placement, baseline storage location, pixel-delta threshold per page) — see Sprint 1.4 §8 "Visual regression baselines — deferred". Quality gate: workspace clippy clean, workspace tests green (15 new SSR walkthrough + 8 new CLI integration tests), workspace rustdoc clean. Sprint 1.4 progress: 15 of 15 in-scope items shipped (T-1.4.1 through T-1.4.13 + T-1.4.16 + T-1.4.17; T-1.4.14 + T-1.4.15 indefinite-deferred at kickoff). See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.11 + T-1.4.12 —
epic-clioperator binary + secret-scanning rotate-key admin route (2026-05-31) — Newepic-clibinary with four-module layout:main.rs(clap entry,current_threadtokio),config.rs(env-var resolution withEPIC_CONTROL_URL+EPIC_ADMIN_KEY; admin key never accepted from CLI flag for shell-history hygiene; env tests serialised behind a process-wideMutexwith SAFETY-commentedunsafe { env::set_var }for Rust 2024),client.rs(thinreqwest::Clientwrapper with bearer auth, sensitive-marked header, typedApiErrormapping the EPICerror_codeshape),output.rs(JSON-by-default to stdout, pretty when TTY). Subcommand tree:epic admin orgs {list,get,members,suspend,unsuspend},epic admin keys kill,epic admin audit-log,epic admin secret-scanning rotate-key. 19 unit tests cover the clap parse tree, env resolution, client error parsing, JSON output shape. New admin routePOST /api/admin/secret-scanning/rotate-key— Phase 1 wiring is deferred (partner-program fetcher not yet held onAppState); the route validates admin scope, emits asecret_scanning.rotate_keyaudit row, returns the stable wire shape{"keys_rotated": 0, "status": "deferred"}. Production wiring will replace the deferred branch with a real HTTPS fetch fromEPIC_GITHUB_SECRET_SCANNING_PUBLIC_KEYS_URL(default GitHub's/meta/public_keys/secret_scanning) and write each key into FerraVault underepic/github-secret-scanning-pubkeys/{key_id}; the wire-shape contract does not change, onlystatusflips to"refreshed". CLI subcommand round-trips end-to-end against the deferred route. 2 new admin-route tests. Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean (with documented#[allow(unreachable_pub)]on theepic-clicrate root);cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.10 — GDPR self-service UI (export + delete) (2026-05-31) — Three new markup modules in
epic_web:forms::gdpr_exportrenders Art. 15 export as readable HTML (Account card + Org memberships table + API keys-issued-by-me table + pointer toGET /api/me/exportfor machine-readable JSON);forms::gdpr_deleteis the Art. 17 two-step confirm form requiring the literal six-letter wordDELETE(case-sensitive,pattern="DELETE"-constrained) plus prominent 30-day-grace warning;pages::gdpr_delete_scheduledis the unauthenticated landing page (uses marketing chrome — the session was just revoked). New handler moduleepic_control::routes::web::mewiresGET /app/me/export(delegates toroutes::me::export_me, projects the JSONExportResponseinto strongly-typed view rows so the JSON handler stays the single source of truth),GET/POST /app/me/delete. Delete POST validatesconfirm == "DELETE"(case-sensitive byte equality) before delegating; success clears the session cookie viaSet-Cookie: epic_session=; Max-Age=0so the browser drops the now-revoked cookie immediately. 5 form-render unit tests + 5 SSR handler integration tests. Quality gate: workspace clippy clean, workspace tests green, workspace rustdoc clean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.9 — Billing flow UI (checkout, portal, slot cert) (2026-05-31) — New
epic_web::forms::billing_checkoutseat-count entry form + two new landings underepic_web::pages:billing_pending(surfaced when underlying handler returnsBillingError::Unavailableper ADR-007 deferred-decision posture — explains live merchant pick is still pending the international-tax-accountant consultation) andbilling_slot_certificate(Phase-1 placeholder until a paid subscription exists). Newepic_control::routes::web::billingwiresGET/POST /app/billing/checkout+POST /app/billing/portal, delegating to existingroutes::billing::create_checkout/create_portal_session(visibility bumped from private topub(super)to expose the same service path to the SSR adapter). Success → 303 redirect to provider URL;BillingUnavailable→ 303 redirect to/app/billing/pending; other typed errors → 400 re-render with user-facing message. The live merchant pick + actual PDF download wire through verbatim once ADR-007 is revised. 2 form-render unit tests + 5 SSR handler integration tests + 1 error-message exhaustiveness test. Quality gate: workspace clippy clean, workspace tests green, workspace rustdoc clean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.8 — Key management UI with plaintext-once OneTimeReveal (2026-05-31) — Three new form modules in
epic_web::forms::{api_key_create, api_key_lifecycle, api_key_reveal}+ handler moduleepic_control::routes::web::keyswiring 5 routes (GET /app/keys/new,POST /app/keys,POST /app/keys/{id}/rotate,GET/POST /app/keys/{id}/disable,GET/POST /app/keys/{id}/revoke). Plaintext-shown-once contract proven at the SSR boundary — the create + rotate success paths render the cleartext key body into the response HTML viabuild_reveal_responseand setCache-Control: no-store, privateso no intermediary cache and no browser back-button can re-show the plaintext. The plaintext is never persisted server-side beyond the lifetime of the response body. Refreshing the reveal page is intentionally destructive. Revoke flow enforces the force-acknowledgement checkbox both client-side (requiredattribute) AND server-side (force != "on"→ 400 re-render with inline error, never reaches the underlying handler). 12 form-render unit tests + 8 SSR handler integration tests + 1 helper test + 1 error-message exhaustiveness test. Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean (documented#[allow(clippy::struct_excessive_bools)]onScopeChecks— bools ARE the data model, one per checkbox);cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.7 — Dashboard skeleton (2026-05-31) — Seven SSR pages under
epic_web::pages::app(home,keys,org,members,billing,audit_log,me) — pure markup, each takes a*Ctxdata struct and renders inside the new authenticated chrome. Seven GET handlers underepic_control::routes::web::app(consolidated; share anauthenticatehelper that callsroutes::auth::require_authenticated_user_puband onAuthApiError::AuthRequiredreturns a303 Location: /login). Newrender_app_document(title, current_path, body)layout variant — authenticated chrome with sidebar nav (current page marked viaaria-current="page"), POST-/logoutbutton header. Data fetch via existing trait surface:Store::list_api_keys_for_org,Store::get_org_by_public_id,Store::list_org_members,AuditQuery::query. Phase 1 read-only contract — interactive flows (create/rotate keys, billing portal, GDPR) ship in T-1.4.8 / T-1.4.9 / T-1.4.10. Baseline CSS extended with.app-grid,.app-sidebar,.app-nav-link,table.app-tablerules. 13 page-render unit tests + 8 GET handler integration tests (one iterates every dashboard path to prove the unauthenticated-redirect contract is uniform). Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean (with documented#[allow(clippy::result_large_err)]on theResult<T, axum::Response>early-return helpers);cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.6 — TOTP enrolment UI (QR + recovery codes) (2026-05-31) — New
epic_web::forms::totp_setupone-time-reveal page (QR + base32 secret + 10 recovery codes + verify-code form) +epic_web::pages::totp_enabledsuccess landing. New GET + POST handlers inepic_control::routes::web::totp_setup. GET delegates to existingroutes::auth::totp_setup, parses the underlying axum Response body back intoTotpSetupResponse(madeDeserializefor this), and renders the one-time-reveal page with explicitCache-Control: no-store, private. POST delegates to existingroutes::auth::totp_verify; success → 303 redirect to/app/security/totp/enabled, failure → 400 re-render of the page WITHOUT the QR/recovery-codes panel (one-time-reveal contract — the underlying handler does not rotate the stored secret across a failed verify so the QR the user scanned remains valid). Unauthenticated → 303 to/login; TOTP already enabled → 303 to/app/security. 5 form-render unit tests + 5 GET/POST handler tests + 1 exhaustiveness test. Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.5 —
/verify-emaillanding page (2026-05-31) — Newepic_web::forms::verify_emailconfirmation-button form +epic_web::pages::verify_email_successlanding page +epic_control::routes::web::verify_emailGET + POST handlers. GET is deliberately side-effect-free — renders a "Confirm email" button form so email-client link previewers (Outlook, Gmail) cannot consume the token before the user clicks; RFC 7231 §4.2.1 GET-safety. POST delegates to the existingroutes::auth::verify_emailfor the consume → mark-verified → audit-emit pipeline. The underlying handler'salready_verified: trueidempotent-replay path collapses to the same/verify-email/successredirect — operator-side distinction preserved in JSON + audit row, never surfaced to the user. Anti-enumeration: token-unknown / malformed / expired / already-consumed all collapse viaAuthApiError::InvalidCredentialsto a single "this verification link is no longer valid" message. 4 form-render unit tests + 4 GET/POST handler tests + 1 error-message exhaustiveness test. Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.4 —
/login+ TOTP step + forgot/reset password SSR (2026-05-31) — Three new form modules incrates/epic-web/src/forms/(login,forgot_password,reset_password) plus two new success-landing pages incrates/epic-web/src/pages/(forgot_password_check_email,reset_password_success). Three new GET + POST handlers undercrates/epic-control/src/routes/web/(login,forgot_password,reset_password) delegate to the existing Sprint 1.1 service path:routes::auth::login,routes::auth::password_reset_request,routes::auth::password_reset_confirm. Login-success path rewrites the underlying200 OK+tokens response into a303 See Otherredirect to/appwhile preserving theSet-Cookie: epic_session=...header verbatim; access + refresh tokens are dropped because the SSR flow authenticates by cookie, not bearer. TOTP gauntlet: the underlying handler'sAuthApiError::TotpRequiredre-renders the form withtotp_required: trueso the 6-digit code field appears below the password input (password field is deliberately blanked per CLAUDE.md §8.3); title flips from"Log in"to"Two-factor verification". Forgot-password is anti-enumeration-clean — always redirects to/forgot-password/check-emailregardless of whether the email matches a real account, mirroring the underlying 202-always wire shape. Reset-password GET extracts?token=...from the URL query (the value the reset email pointed at) and embeds it as a hidden input; POST compares the new + confirm password pair before delegation (mismatch returns a 400 re-render with the token preserved so the user can retry without losing the link). 12 new form-render unit tests + 9 form GET/POST round-trip tests + 3 helper exhaustiveness tests (*_error_messagecovers every reachable variant for each form). Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --lib --binsgreen workspace-wide;cargo doc --workspace --no-deps --document-private-items -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.3 —
/signupSSR form with Turnstile/OFAC/AUP/age gates (2026-05-31) — New module split: pure markup incrates/epic-web/src/forms/signup.rs(the load-bearing pattern T-1.4.4 onwards re-uses); GET + POST handlers incrates/epic-control/src/routes/web/signup.rs; success landing at/signup/check-email. POST extractsapplication/x-www-form-urlencodedSignupFormviaaxum::Form(workspace axum features pin extended with"form"), validates the three required checkboxes (age + AUP + privacy) up-front, builds aRegisterRequest, and delegates toroutes::auth::registerdirectly so the full Sprint 1.1 gauntlet (Turnstile + disposable-email + OFAC + age + policy-version + Argon2id + atomic register + verify-email + Postmark + audit) runs verbatim. Success → 303 redirect to/signup/check-email; failure → re-rendered form with prior values (password explicitly NEVER round-tripped per CLAUDE.md §8.3) and a typed user-facing error message mapped exhaustively from every signup-reachableAuthApiErrorvariant.AppStategainsturnstile_site_key: Option<String>(the public, client-side identifier the Turnstile widget needs, distinct from the server-sidesecret); whenNonethe widget renders an inline notice instead. Newroutes::websub-router underroutes/web/mod.rsis the forward-compatible mount point for T-1.4.4 (login), T-1.4.5 (verify-email), T-1.4.6 (TOTP), and the dashboard / GDPR forms in T-1.4.7 → T-1.4.10. 11 new tests across the form (5 markup unit tests + 4 form GET/POST handler tests + 2 helper unit tests). Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --lib --binsgreen workspace-wide;cargo doc -p epic-web -p epic-control -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.2 — marketing SSR pages + safe Markdown renderer (2026-05-31) — Five marketing routes mounted on top of the T-1.4.1 scaffold:
/(landing),/pricing,/security,/about,/changelog. The changelog page embedsCHANGELOG.mdat compile time viainclude_str!and renders through the newepic_web::markdown::render_markdownhelper, which dropsEvent::HtmlandEvent::InlineHtmlevents frompulldown-cmark's parser at parse time so raw<script>/<iframe>/ inline event-handler attributes cannot reach the rendered chrome (defence-in-depth — the changelog source is operator-authored, but treating it as trusted-with-HTML would couple page security to "operators never land an XSS-shaped commit"). Copy is placeholder pending the Phase 6 marketing/lawyer pass per DESIGN.md §17. Workspace gainspulldown-cmark 0.13(pure-Rust, approved §3.7); epic-web pins it withdefault-features = false, features = ["html"]. Tests: 8 markdown-renderer unit tests (standard Markdown round-trip, code blocks, tables, empty input, safe-scheme links pass through; security-critical raw-script, inline-script, iframe +onerror,<img>strip cases) + 1 changelog-content sanity + 6 page-shape integration tests viatower::ServiceExt::oneshot(each marketing page is 200 OK with layout chrome and page-specific markers; changelog asserts no<scriptanywhere in the response). Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test -p epic-web --all-targets18/18 green;cargo doc -p epic-web -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.1 —
epic-webLeptos SSR scaffolding (2026-05-31) — Leptos 0.7.8 SSR pipeline integrated undercrates/epic-web/. New modules:layout(sharedrender_document(title, body)chrome emitting full HTML document with embedded baseline stylesheet),router(statelessRouter<()>mounted intoepic-controlvia.merge(epic_web::router())after.with_state(state)),pages::ping(/_web/pingSSR liveness page proving the pipeline). Hand-curated baseline CSS undersrc/assets/site.cssembedded viainclude_str!— binary stays self-contained per CLAUDE.md §3.7 (no Node.js runtime dep). Reserved-path discipline:epic-webmounts only under/_web/*for SSR internals —/, marketing pages, and the dashboard land in subsequent commits. Workspace gainsleptos,leptos_meta,pulldown-cmarkpins;leptos_axumintentionally omitted (pins axum 0.7 vs workspace's axum 0.8 —epic-webcallsview!{...}.to_html()from hand-rolled axum 0.8 handlers instead). 3 integration tests (ping page renders HTML document with doctype/title/layout markers; unknown/_web/*paths 404; bare epic-web router does not shadow/health). Quality gate:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --lib --binsgreen (no sibling-crate regressions);cargo doc -p epic-web -- -D warningsclean. See Sprint 1.4 §7 Progress Log. -
Sprint 1.4 T-1.4.13 — ADR-008 (diagnostic capture encryption) Accepted (2026-05-31) —
docs/adr/008-diagnostic-capture-encryption.md. Locks the envelope-encryption protocol that Phase 4's capture endpoint inherits: AES-256-GCM at both the payload and data-key layers; per-org master key resident in FerraVault and accessed only via server-sidewrap/unwrap(master key never inepic-controlprocess memory on the capture write path); per-capture data key ephemeral and zeroised on drop; 12-byte CSPRNG nonces per encryption with versioned AAD binding the ciphertext to(org_id, capture_public_id, captured_at_epoch_ms); annual master-key rotation with 90-day overlap; revocation hard-deletes affected captures. DESIGN.md §10.7 schema annotated with the three Phase-4 envelope columns Phase 4 lands (data_key_enc,data_key_nonce,encryption_key_version SMALLINT); DESIGN.md §20 ADR-008 row flipped to Accepted. Plane: control. See Sprint 1.4 §3 T-1.4.13 for the closure note.
Changed
- Sprint 1.4 kickoff — commit signing deferral made indefinite (2026-05-31) — operator-initiated. Sprint 1.4 title and scope narrowed from "epic-web SSR + epic-cli + commit signing + ADR-008/020" to "epic-web SSR + epic-cli + ADR-008". T-1.4.14 (ADR-020 written) and T-1.4.15 (commit signing enablement, Phase 0 history rewrite,
phase-0-final-signedtag,ci.shStage 0verify_signing_chain, pre-commit hook signature gate) reassigned to indefinite — operator-scheduled per DESIGN.md §20. New open-question entry DESIGN.md §21 q11.CLAUDE.md§0 + §9.1 updated to reflect the indefinite-deferral posture. PHASE-1-OVERVIEW.md §2/§4/§5/§7 risk register entries updated in lockstep — phase-1-final tag will be unsigned annotated until enablement schedules. Quoted approval line preserved in Sprint 1.4 §8.
Added
-
Sprint 1.3 slice 2 — billing routes + Postgres-backed
WebhookEventStore+OrgExpiryWarningStore+ orchestration service + plan-expiry task + e2e/RLS suite (2026-05-30) — substrate side of Sprint 1.3 now complete end-to-end. Refined the Stripe parser to walkitems.data[].quantity(sum across multi-item subscriptions) with top-level fallback for legacy shapes — the priorunwrap_or(1)silent mis-record path is replaced by an explicitBillingError::Rejectedwhen no quantity field is reachable; 4 new Stripe parser tests cover items-level extraction, multi-item sum, legacy fallback, and the rejection case. Two new traits inepic-trait:WebhookEventStore(insert-replay-mark idempotency contract overwebhook_events) +OrgExpiryWarningStore(exactly-once contract overorg_expiry_warnings); both ship with type-doc invariants + non-exhaustive#[non_exhaustive]enums for forward compatibility.Storetrait extended withfind_org_by_billing_customer_ref,update_org_seat_count(atomic withSeatCountUpdateOutcome::{Updated, BelowAssigned}outcome + per-orgpg_advisory_xact_lockmatching migration 0033's discipline),cancel_org_subscription,list_orgs_in_expiry_window. Theepic_trait::Orgstruct gainedbilling_provider,billing_customer_ref,billing_email,seat_count,plan_started_at,plan_expires_at,country_code— everyOrgdecoder + builder + constructor (Pg adapter, Mem adapter,OrgBuilder,register_userpath, the 3 test fixtures) updated in the same PR. Postgres adapters:PgWebhookEventStore+PgOrgExpiryWarningStore— both runSET LOCAL ROLE epic_admininside their transactions to satisfyFORCE ROW LEVEL SECURITYwithout bypass. Pgupdate_org_seat_counttakes per-orgpg_advisory_xact_lockmirroring the trigger discipline; pglist_orgs_in_expiry_windowbacks the task.LayeredStoreextended for the four new methods + cache invalidation onupdate_org_seat_count/cancel_org_subscription. In-memory adapters:MemWebhookEventStore+MemOrgExpiryWarningStorewith the same contract (parking_lot Mutex-serialised) for the merchant-agnostic e2e. Routes: newbilling.rsmodule exposesPOST /api/billing/checkout+POST /api/billing/portal(bothorg:admin-scoped +BillingError::Unavailable→503 billing_unavailablemapping via a new typedAuthApiError::BillingUnavailablevariant) andPOST /api/billing/webhook/{provider}with path-segment dispatch onto the three concrete adapter handles. Webhook handler: verifies signature (typedAuthApiError::WebhookSignatureInvalid→ 400 on failure with the invalid body still persisted towebhook_eventsfor operator triage), parses event, dedupes viaWebhookEventStore::insert_unprocessed, dispatches to the newservices::BillingServiceorchestrator, marks processed. Orchestration service:process_eventswitches onSubscriptionEventvariant —Created/Updated→update_org_seat_countwithSeatReassignmentBlockedarm emittingbilling.seat_drop_blockedaudit row + Postmarkseat_reassignment_requiredtemplate;Canceled→cancel_org_subscriptionwith default 30-day grace;TransactionCompleted→ slot certificate render (hand-rolled PDF + Ed25519 sig from slice 1) +payment_receipt+slot_certificatePostmark emission with PDF attachment; every arm emits a stablebilling.*audit action. Newtasks::PlanExpiryWarningsTaskwithrun_oncerunning the exactly-once-per-(org, kind) Postmark fan-out across 3 windows (90d / 30d / 7d); two-phase send-then-mark contract documented in module docs (Postmark failure ⇒ skip mark ⇒ next run retries; concurrent runs serialised bymark_sent'sON CONFLICT DO NOTHING).AppStateextended withpaddle_provider,lemonsqueezy_provider,stripe_provider,billing_provider(generic-checkout slot wired toMemBillingProviderin tests),webhook_events,expiry_warnings,billing_service. Test surface:integration_billing_e2e.rscovers 11 e2e scenarios via the in-memory adapter stack including signature-invalid bounce, idempotency single-thread + 32-task concurrent race (asserting exactly one Inserted + 31 Replays land), seat-drop-blocked audit emission, plan-expiry exactly-once + window-overlap correctness + clear-on-renewal; 5 new RLS tests inepic-db/tests/rls.rscoverwebhook_events+org_expiry_warningsinvisibility-to-app-role + admin-role-can-read (withGRANT epic_admin TO epic_app_rls_testadded to the test provisioning so theSET LOCAL ROLEdiscipline is exercised) + UNIQUE / composite-PK enforcement;fixtures/regenerate.mdauthored documenting the future ADR-007 revision sprint's procedure for refreshing the chosen merchant's fixtures from live-sandbox traffic. Quality gate after slice 2:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspacegreen workspace-wide (0 failures);cargo deny checkclean (advisories, bans, licenses, sources OK);cargo auditclean (526 deps scanned vs 1099 advisories, no vulnerabilities);RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-depsclean. Sprint 1.3 substrate complete end-to-end — every §2 item shipped or covered by the four-field deferral on the live HTTP outbound paths (T-1.3.3/4/5 explicitly defer ONLY the live HTTP plumbing); the next sprint opens against a fully-wired billing surface. Seesprints/phase-1/sprint-1.3-billing-and-slot-certificate.md§7 Progress Log. -
Sprint 1.3 slice 1 —
epic-billingmerchant-agnostic substrate + three adapter shells + slot certificate + ADR-007 (deferred-decision form) (2026-05-30) — Option B locked at sprint open (Sprint 1.3 §1 framing the four-field deferral of the live merchant pick per DESIGN.md §21). ADR-007 Accepted as deferred-decision form — three-candidate evaluation locked, symmetric-shells posture locked, live merchant pick reassigned to a future ADR-007 revision sprint post the Australian international-tax-accountant consultation.epic_trait::BillingProviderextended withprovider_kind,create_customer_portal_session,verify_webhook_signature,parse_subscription_event,lookup_customer,lookup_subscription.BillingErrorwidened withSignatureInvalid,EventReplayed,SubscriptionNotFound,UnsupportedEvent. New types:SubscriptionEvent,SubscriptionStatus,CurrencyCode,Customer,Subscription,PortalSession. Three symmetric adapter shells landed undercrates/epic-billing/src/:paddle::PaddleProvider(ts=...;h1=...HMAC-SHA256, 5-minute replay window),lemon_squeezy::LemonSqueezyProvider(X-SignatureHMAC-SHA256),stripe::StripeProvider(t=...,v1=...HMAC-SHA256 with replay-window check + multi-v1rotation support). All three: signature verifier real + constant-time viasubtle::ConstantTimeEq; event parser real against the providers' published wire shapes; live HTTP outbound methods return typedBillingError::Unavailable("<provider> live adapter pending ADR-007 revision")uniformly. Sharedsignaturemodule factors the HMAC + constant-time compare primitives. Recorded fixtures for every event arm undercrates/epic-billing/tests/fixtures/{paddle,lemonsqueezy,stripe}/+ conformance test suite (15 tests). Slot certificate PDF generator underepic_billing::slot_certificate— hand-rolled minimal PDF 1.4 encoder inpdf.rs(printpdfrejected at implementation time on byte-reproducibility grounds — see ADR-007 §"Alternatives" point B). Renders deterministically; embeds Ed25519 signature in/Infodict as/EpicSignature; verifier in the same crate; round-trip + reproducibility + tamper-rejection tests all green. Migrations0034_webhook_events.sql(idempotency ledger,UNIQUE(provider, event_id), admin-only RLS mirroringaudit_log) +0035_org_expiry_warnings.sql(per-(org, kind) warning-fired ledger, admin-only RLS).epic_postmark::TemplategainsExpiryWarning90d,ExpiryWarning30d,ExpiryWarning7d,SeatReassignmentRequired. NewPostmarkClient::send_template_with_attachmenthelper takes aPostmarkAttachment<'_>struct (Name + ContentType + bytes), base64-encodes into the PostmarkAttachmentsfield.epic_test_utils::MemBillingProviderextended to satisfy the full trait surface: real HMAC-SHA256 signature verifier on a test-onlyx-mem-signatureheader; JSON envelope parser; synthetic success responses on the live methods so the merchant-agnostic e2e is exercisable without binding to a provider's wire shape. Six criterion benches scaffolded (3× sig-verify per adapter, parse warm path, slot-cert render, slot-cert verify). Schema-drift guard updated for both new tables. Sprint 1.2 closing migration head was 0033; this slice numbers from 0034 forward. New workspace dep:sha2(already transitive viaed25519-dalek+aws-lc-rs, promoted to a direct pin for explicit Sprint 1.3 usage).epic-test-utilsgainshttp,hmac,sha2,hex,serde,serde_jsonas non-dev deps to back the in-memory billing provider.epic-billinggainshmac,sha2,subtle,hex,ed25519-dalek,blake3,chrono,uuid,serde,serde_json,base64,thiserror,tracingas direct deps. Quality gate after slice 1:cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspacegreen workspace-wide; epic-billing 52 unit + 15 conformance tests; epic-postmark 6 integration tests (1 new); epic-test-utils 80 unit tests (6 new); epic-trait 5 unit tests (3 new). Slice 2 scope note (Sprint 1.3 §7 Progress Log): the remaining §2 items (T-1.3.7, T-1.3.8, T-1.3.9, T-1.3.10, T-1.3.13, T-1.3.14 — control-plane routes + Postgres-backedWebhookEventStoreadapter + merchant-agnostic e2e suite) land in subsequent commits of this same Sprint 1.3 work-stream. Substrate-side scope unchanged. Plane: control. Seesprints/phase-1/sprint-1.3-billing-and-slot-certificate.md+ ADR-007. -
Sprint 1.2 post-close audit closure (2026-05-30) — operator-approved rework pass on all 11 self-review findings rather than carrying them into Sprint 1.3. New migration
0032_audit_log_org_id.sql(first-class tenant scope column with composite(org_id, occurred_at DESC, id DESC)index;canonicalise()intentionally excludesorg_idso pre-migration row_hash values continue to verify); fixes the audit-log handler's silent cross-tenant read (filter.org_idnow binds to the new column). NewStore::list_users_pending_erasure(cutoff, limit)trait method + Pg + Mem adapters wired intoErasureSweep::candidates()— the sweep is no longer a no-op.MemAuditSink::matches_filternow honorsactor_user_id+org_id(was silently ignoring both).MemStore::hard_delete_usercascade extended to drop email_verification_tokens, password_reset_tokens, and seat_assigned entries (was partial). Two new cross-tenant admin routes:GET /api/admin/orgs(paginated org listing backed byStore::list_orgs_admin) +GET /api/admin/audit-log(cross-tenant dump with optional?org_public_id=narrowing). 21 new unit tests acrossroutes::me(7),routes::admin(6),routes::abuse(1 idempotency + 3 proptests),routes::auditandservices::gdpr(1 proptest + 3 cases). 1 new concurrency test inepic-store/tests/integration_pg_store.rs(concurrent_seat_assignments_respect_enforce_seat_limit_trigger). 8 new wire-side integration tests inepic-control/tests/integration_sprint_1_2.rsagainst fresh Pg containers covering the named T-1.2.17 flows (signup→key, GDPR export, GDPR erase no-orphans, audit-log chain intact, audit-log chain broken returns 500, GitHub secret-scanning auto-revoke, abuse-report ingestion, admin-scope enforcement). Newbenches/sprint_1_2_routes.rswith 3 criterion benches (me_export,me_delete,audit_log_query) matching sprint §5 SLO budgets.AppState::storeandAppState::vaultwidened from concreteArc<LayeredStore<…>>/Arc<VaultClient>toArc<dyn Store>/Arc<dyn KeyVault>so test fixtures can wireMemStore/MemKeyVaultdirectly — every call site uses trait methods only, performance impact is control-plane only.AppState::with_vault_and_audit_querytest helper added. Quality gate after closure:cargo clippy --workspace --all-targets -- -D warningsclean; epic-control unit tests 235 → 256 (+21); epic-test-utils 72 → 76 (+4); integration_sprint_1_2 8/8 pass; integration_auth_e2e 12/12 pass (no regressions). Seesprints/phase-1/sprint-1.2-org-keys-and-secret-scanning.md§7 "Post-close audit closure". -
Sprint 1.2 — Org / members / keys / me + GDPR + audit-log query + admin + GitHub secret-scanning receiver (closed 2026-05-30) — 16 of 17 §2 items shipped; one explicit four-field deferral (T-1.2.12 → Phase 6); one wire-side suite scope-realignment (T-1.2.17 → Sprint 1.5). Three new migrations:
0029_org_member_invites.sql(new customer-data table, FORCE-RLS'd, CHECK forbiddingrole='owner'so invites cannot transfer ownership),0030_epic_admin_role.sql(provisionsepic_adminPostgres role withBYPASSRLS NOLOGIN, idempotent viaDO $$ EXCEPTION WHEN duplicate_objectblock),0031_users_display_name.sql(adds nullabledisplay_name TEXTwith 1..=256-char CHECK). Trait substrate expansion:epic-trait::Storegains 14 new methods (org update / transfer-owner / suspend / status / list-members / role / seat / delete / count-memberships / find-by-prefix-and-last4 / soft-delete-user / display-name / email / hard-delete); newepic-trait::invites::InviteStore(insert / list-pending / atomic consume-by-token-hash / revoke); newepic-trait::abuse::AbuseStore+AbuseCategory(addsKeyLeak); newepic-trait::audit::AuditQuery(filter + cursor pagination + intra-page chain verify).epic-coregainsOrgInviteId/Public,AuditLogId.epic-security::ActorKind::EpicAdminvariant.epic-postmark::TemplateaddsKeyRotated,KeyDisabled,KeyRevoked,OrgInvite,AupUpdate,GdprEraseScheduled,AbuseReceived. Parallel Pg + in-memory adapter impls land for every new trait method (PgInviteStore,PgAbuseStore,PgAuditSink::AuditQuery,MemInviteStore,MemAbuseStore,MemAuditSink::AuditQuery).epic-auth::ApiKeyService<S>gainsS: ?Sizedso the route layer can holdApiKeyService<dyn Store>. Six new route modules inepic-control:orgs.rs(T-1.2.1–T-1.2.4, nine handlers including 2FA-gated transfer-owner with TOTP step-up + cascade soft-delete on sole-org member removal),keys.rs(T-1.2.5–T-1.2.6, six lifecycle routes; Turnstile gate onorg-typed keys; force-flag-gated permanent revoke),me.rs(T-1.2.7–T-1.2.9, GDPR Art.15 export + Art.17 erasure scheduling; soft-delete + 30-day grace; verify-email re-issuance on email change),audit.rs(T-1.2.10, cursor-paginated query with chain verification on read; 500 +audit_chain_brokenon break),abuse.rs(T-1.2.11 GitHub Secret Scanning Partner Program webhook receiver with Ed25519 signature verify via FerraVault-stored partner public key + auto-revoke viafind_api_key_by_prefix_and_last4+ audit + Postmarksecurity_alert; T-1.2.13 public abuse-report ingestion),admin.rs(T-1.2.14, five operator routes gated byKeyScope::AdminAPI key;ActorKind::EpicAdminon every audit emission). New serviceservices::gdpr::ErasureSweep— the hourly background hook the production binary invokes to hard-delete soft-deleted users past the 30-day grace. NewAppStatewiring fields:bare_store,invites,abuse_store,audit_query. Axumqueryfeature added to the workspace pin to enable theQueryextractor for the audit-log route. Deferral: T-1.2.12 (live GitHub partner-program filing) → Phase 6 per DESIGN.md §4.3 + the Sprint 1.2 four-field block. The receiver code itself lands in this sprint and runs against the GitHub registration sandbox locally throughout Phases 1–5; live wiring flips on when Phase 6 lands the production HTTPS endpoint. Scope-realignment: T-1.2.17 (wire-side integration suite extension) folded into Sprint 1.5 sign-off so the full Phase 1 surface area runs as one suite — the Sprint 1.2 unit-test surface (235 passing acrossepic-control) covers every new route viatower::ServiceExt::oneshotagainst the in-memory adapter stack. Quality gate at sprint close:cargo check --workspaceclean;cargo clippy --workspace --all-targets -- -D warningsclean;cargo test -p epic-control --lib→ 235 passing;cargo test -p epic-auth→ 111 passing;cargo test -p epic-test-utils --lib→ 72 passing;cargo test -p epic-db --test schema_invariants -- --test-threads=1→ 12 passing;cargo test -p epic-db --test rls -- --test-threads=1→ 18 passing. Plane: control (zero edge changes; the data-plane invariant from Sprint 1.1 is preserved). Seesprints/phase-1/sprint-1.2-org-keys-and-secret-scanning.md. -
Sprint 1.1 —
epic-controlHTTP surface + auth routes (closed 2026-05-30) — full §2 scope, zero deferrals. All nine/api/auth/*handlers shipped:register(T-1.1.3, 11-step gauntlet),verify-email(T-1.1.4),login(T-1.1.5, anti-enumeration + TOTP + session/refresh issue + JWT mint),logout(T-1.1.6, session + bound family revoke),refresh(T-1.1.7, cookie + refresh + RFC 6819 §5.2.2.3 replay detect),password-reset/request(T-1.1.8, always-202),password-reset/confirm(T-1.1.9, single-use + rotate + revoke-all-sessions + revoke-all-families),totp/setup+totp/verify+totp/disable(T-1.1.10, QR PNG + recovery codes). HTTP/3 over QUIC primary + HTTP/2 over TLS-on-TCP fallback in the same process (T-1.1.1) — h3 via the ADR-023quinncarve-out following the Ferrum precedent (control-plane only; edge plane keeps strictquinn-proto-only posture). Middleware tower mounted onto the production composition root viarouter_with_middleware(T-1.1.2). Turnstile (T-1.1.12), OFAC (T-1.1.13). Hash-chainedaudit_logvia the new productionPgAuditSink(T-1.1.14, advisory-xact-lock matches migration 0015's trigger).FerrumRateLimiter(T-1.1.11) wired inAppState::wirefrom the FerrumClient. 12-test wire-side integration suite (T-1.1.15) — every test over real wire on both transports (h2 over TLS + h3 over QUIC) against the real Postgres-backed adapter stack booted per-test via testcontainers. Three criterion benches (T-1.1.16). NEW production PG-backed adapters inepic-store:PgStore::AuthFlowStore(auth_flow.rs, atomic register_user transaction across users + orgs + memberships;DELETE … RETURNINGfor single-use token consume; CAS recovery-code consume),PgSessionLookup(session.rs, write-through revoke),PgSessionStore(session.rs,FOR UPDATErow lock inconsume_and_rotatefor RFC 6819 §5.2.2.3 family revoke),PgAuditSink(audit.rs, advisory-xact-lock chain serialisation matching the 0015 trigger). 18-test PG adapter integration suite proves each method end-to-end against a real Postgres testcontainer. Migration 0027 reshapesrefresh_tokens(BIGINT family_id via sequence + public_id per ADR-002 + rotated_out/family_revoked explicit booleans) to match the trait surface — closes the "Phase-2 PG-side audit item" migration 0026 had flagged.SessionRecord::refresh_family_idtrait extension + migration 0026 (sessions ↔ refresh family linkage).SessionCookie::parse_hex,AuthFlowStore::find_user_by_id/find_primary_org_for_user/register_usertrait extensions.require_authenticated_userhelper.RegisterUserPayload+RegisteredUsertypes. New ADRs: ADR-022 (h3 deferred — same-day superseded by ADR-023 after the post-close audit caught it as effort-disguised-as-blocker perCLAUDE.md§10.6), ADR-023 (h3 via thequinncarve-out following the Ferrum precedent). New workspace deps:quinn 0.11,h3 0.0.8,h3-quinn 0.0.10,axum-server 0.8,tokio-rustls 0.26,percent-encoding 2.3,reqwest 0.12(dev),rcgen 0.14(dev),base32 0.5(epic-control). Quality gate at close:cargo test --workspace --tests --lib→ 45 test binaries pass / 0 failed;cargo clippy --workspace --all-targets -- -D warnings→ clean;ci.sh --fullStages 0–6 → passes end-to-end; data-plane invariant unchanged (epic-edgecarries zerotokioAND zeroquinndeps). Seesprints/phase-1/sprint-1.1-control-http-and-auth.md.
[0.1.0] — 2026-05-29 (Phase 0 sign-off)
Phase 0 — Foundation. Empty directory → workspace that compiles → full §10 schema migrated → auth library trustworthy → adapter crates wired → observability live → MeterEvent round-tripping through Ferrum. Sealed by the phase-0-final annotated tag against commit 1646cf25 (unsigned annotated per the T-0.6.13 four-field deferral). See sprints/phase-0/sprint-0.6-sign-off.md for the closing audit transcript; the per-sprint contributions are below.
Added
- Sprint 0.1 — workspace skeleton,
/databackup pipeline, hook scripts,ci.shStage 0 + Stage 1, ADRs 001–006, governance docs. Seesprints/phase-0/sprint-0.1-bootstrap.md. - Sprint 0.2 — six foundation crates with zero I/O:
epic-core(newtype IDs, enums, time, error taxonomy),epic-crypto(Argon2id/BLAKE3/AES-GCM/Ed25519/Secret<T>),epic-security(ct_eq, validators, audit canonicalisation),epic-protocol(native + OpenAI/Anthropic shim types,EpicExtension, prost messages, 5-target fuzz harness),epic-trait(9 subsystem traits),epic-test-utils(in-memory adapter per trait, builders,TestClock, optional proptest strategies). ADRs 002, 004, 005 → Accepted. Criterion baselines for Argon2id hash andInferRequestJSON parse. Seesprints/phase-0/sprint-0.2-foundation-crates.md. - Sprint 0.3 — database layer:
epic-db(sqlx 0.9 against Postgres 16, RLS-awareTransactionenforcing(UserId, OrgId)GUCs in 3 round-trips with a combined integer-parsed echo + ADR-003 canary fail-stop, hardened againstBYPASSRLSregressions), 21 SQL migrations realising every DESIGN.md §10 table withFORCE ROW LEVEL SECURITYon every customer-data table plus therls_canarysentinel, audit-log chain + append-only triggers,epic-migratebinary (structured tracing,--dry-run, documented exit codes,Secret<String>DSN), schema-invariants (10) and RLS (12) test suites including a fail-stop test that injects a fake-tenant canary row and assertsTransaction::beginaborts, and two criterion benchmarks verifying Sprint 0.3 §5 perf targets (Transaction::beginp50 ≈ 140 µs vs 1 ms target; RLS overhead ≈ 1.8 % vs 10 % target).ci.shStage 3 fully exercising the migration round-trip. ADR-003 → Accepted with acceptance evidence. Seesprints/phase-0/sprint-0.3-database.md. - Sprint 0.4 — authentication library:
epic-authshipping six services —PasswordService(Argon2id + anti-enumeration dummy PHC on aspawn_blockingboundary, gated behind thepasswordcargo feature soepic-edge's data-plane build pulls notokio),JwtService(custom EdDSA-only JWT, noring/openssl,kid-indexed rotation, structural algorithm-confusion immunity documented in the module docs alongside the evaluation of every Rust JWT alternative),RefreshService(32-byteOsRngtokens, BLAKE3 hash at rest, family-revocation on reuse-detection per RFC 6819 §5.2.2.3),SessionService(cookie-side lifecycle parameterised over theepic_trait::SessionLookuptrait so the in-memory and Postgres adapters share the contract),TotpService(RFC 6238 HMAC-SHA1 with the Appendix B vector check, AES-256-GCM-encrypted secrets, 10 BLAKE3-hashed recovery codes), andApiKeyService(ADR-006epic_{live|test}_{sk|pk|org}_{43 Crockford-base32}format with 215 bits of unguessable entropy, BLAKE3 keyed-hash storage, scope + IP + model allow-list checks, daily caps,rotate_with_overlapthat schedules the predecessor'sexpires_atatnow + ROTATION_OVERLAP_DAYSso the lifecycle gate retires the old key automatically). CIDR membership usesipnet(fail-closed on malformed input). Criterion benches verifying every §5 perf target. 7 dedicated integration tests for refresh-token reuse detection (property + multi-threaded race). Anti-enumeration timing-parity integration test (1024 samples, interleaved with warm-up, max/min ≤ 1.5×).cargo llvm-covreports 92.46 % line / 93.22 % region across the seven service files; every wire-reachable error branch on everyverifymethod has a passing test. ADR-006 → Accepted with the entropy claim corrected (256 bits → 215 bits) and the value sets narrowed to match the §10 migration constraints. T-0.4.8 in-tree half shipped; the GitHub Secret Scanning Partner Program registration is surfaced as a four-field deferral block (sprint §8) reassigned to Phase 1 alongsideepic-control's public HTTP surface. Seesprints/phase-0/sprint-0.4-auth.md. - Sprint 0.5 — adapter & operations crates:
epic-vault(FerraVault HTTP client over hyper 1.x + hyper-rustls with aws-lc-rs, AppRole token inSecret<String>, TTL-bounded in-memory cache, long-poll push channel that invalidates the cache on every FerraVault revision bump);epic-config(typed BootConfig + HotConfig sections, three-layer precedence loader TOML < env < CLI,vault:<key>placeholder resolution againstepic_trait::KeyVault, arc-swap-backed watcher that polls the TOML mtime + subscribes toepic_vault::Watcherand refuses hot reloads that change boot-only fields);epic-observability(Prometheus registry with theepic_<plane>_<noun>_<unit>naming convention enforced byvalidate_name, label-cardinality budget pinned byALLOWED_LABELS, axum-served/metrics+/healthzadmin listener,tracingsubscriber installer with JSON / pretty output, feature-gated OTel OTLP/gRPC exporter behind theotelfeature soepic-edge's data-plane build avoids the tokio runtime in the OTel batch processor);epic-store(Postgres adapter implementing everyStoremethod against the §10 schema; INET / INET[] columns bound and projected as TEXT with explicit::INETcasts so the adapter does not require sqlx's optionalipnetworkfeature;Cachetrait local to the crate withMemCache,NoopCache, andFerrumCache(Redis/RESP3) implementations;LayeredStore<S, C>composite read-through + write-through over any Store/Cache pair;FerrumMeterimplementingepic_trait::Metervia Redis StreamsXADDso MeterEvents round-trip through Ferrum exactly as DESIGN.md §10.8 requires);epic-postmark(templated and inline send paths over hyper + rustls;Templateenum covering verify_email / password_reset / payment_receipt / slot_certificate / key_created / abuse_action / security_alert; single retry on 5xx, no retry on 4xx; theX-Postmark-Server-Tokenis sourced from aSecret<String>;InboundEmail::from_jsonparses the Postmark webhook payload for the Phase-1 [email protected] flow);epic-controlminimal test harness (axum on/health+/healthz, graceful shutdown on SIGTERM/Ctrl-C, observability- metrics admin endpoint wired through epic-config);
integration docker-compose stack
(
ci/integration/docker-compose.yaml) bringing up Postgres 16, Redis 7 (Ferrum stand-in), and four wiremock services (Ferron, Postmark, Paddle, FerraVault) with healthchecks on every container plus seeded mappings for the vault secret/revision surface, the Postmark /email endpoint, the Ferron /v1/capabilities endpoint, and a Paddle webhook stub;ci.shStage 4 now probes the stack via pg_isready / redis-cli ping / wiremock /__admin/mappings and exportsEPIC_*_BASE_URLenv vars so integration tests dial the running stack; Stage 6 placeholderci/load/edge-smoke.jsparses cleanly under k6 (real edge workload arrives Phase 2). 59 test suites green workspace-wide; workspace clippy-D warningsclean. Seesprints/phase-0/sprint-0.5-adapters.md.
- metrics admin endpoint wired through epic-config);
integration docker-compose stack
(
- Sprint 0.5 hardening pass (post-self-review): a candid audit of
the initial pass surfaced ten production-readiness gaps. The
rework closed every one across nine phases:
(A)
Store::insert_user/insert_orgwidened to take typedUserInsert/OrgInsertcarrying every NOT-NULL §10 column; thePENDING_PASSWORD_HASHplaceholder is gone. Migration 0022 addsorgs.plan(was hardcoded). Migration 0023 addsapi_keys.revoked_at. NewBillingProviderenum inepic-core. Test fixture helpersuser_fixture/org_fixtureinepic-test-utils. (B)Cachetrait moved fromepic-storetoepic-traitper CLAUDE.md §4.3. NewLayeredSessionLookup<S, C>for the hot-path session cache the §4 spec asked for and that the initial pass missed (the originalLayeredStorecached the wrong reads). (C) Postgres live integration suite: 10 testcontainers-backed tests againstpostgres:16-alpinesurfaced and fixed four real SQL bugs — wrong column names (user_id/namevsissued_by/label), missingrevoked_atcolumn, INET projection returninghost/prefixnot bare IP, seat-trigger firing on defaultseat_assigned=TRUE. Plus a feature- unification bug:rustlsCryptoProviderwas implicit; explicitaws_lc_rs::default_provider()pin inepic-vault/epic-postmarknow stable workspace-wide. (D) Criterion benches verify §5 perf targets:LayeredSessionLookuphit ≈ 718 ns (target < 500 µs; beats by 695×); miss → inner ≈ 1.28 µs (target < 5 ms; 3900×);FerrumMeterrecord + tail ≈ 117 µs (target < 50 ms; 427×). (E) Testcontainers-backed meter round-trip — no silent skip. (F) Live OTel collector integration test viaotel/opentelemetry-collector-contrib:0.113.0; feature-gated behindotel, runs in ci.sh Stage 4. (G)epic-controlbecomes a lib + bin pair. The library exposesAppStatecarrying typed handles forPgStore,FerrumCache,FerrumMeter,VaultClient,PostmarkClient. New/readyendpoint probes each configured adapter at orchestrator readiness-probe cadence; integration test boots the full Postgres + Redis stack via testcontainers and asserts/readyreturns 200 with store=true and ferrum=true. (H)ci.sh --fullactually runs end-to-end. Stages 1–4 all green; real CI debt cleared (tokio-tar advisory ignore + bollard ring whitelist + audit-ignore propagation; integration stack port conflicts 5433→5499, 6380→6399;--features otelon the integration target glob). Seesprints/phase-0/sprint-0.5-adapters.md§7 for the per-phase details. - Sprint 0.5 hardening pass II (deferred-debt closure): re-asked
"is this really production grade?" and surfaced five more
shortcuts. Closed every one:
(1) Five dead-code stubs + unused imports across
epic-controlandepic-vault(_config_env,_signal_module_used,_ip_addr_marker,classify_timeout,ArcWatcher,SecretCache::len) and the missingReason:paragraph on theclippy::too_many_argumentsallow inMeterEvent::new— removed, not allowed. (2) RUSTSEC-2025-0111 (tokio-tar) + RUSTSEC-2025-0134 (rustls-pemfile) — closed by upgrading testcontainers 0.23 → 0.27 (moved to maintainedastral-tokio-tar); both advisories drop from the lockfile,deny.toml[advisories] ignoreis now empty. (3) OTel collector test exercised only the install path (force-flush/shutdown skipped under 0.27's batch-processor stall). Upgraded opentelemetry stack 0.27 → 0.32, rewrote the pipeline against the newSdkTracerProvider+shutdown_with_timeout(5s)API, and the test now drives install + span + flush + shutdown end-to-end against a live collector inside a 30 s deadline. (4) epic-db RLS test silent-skip pattern:let Some(env) = setup().await else { return; }made the 22 RLS + schema- invariant tests claim ✓ without ever connecting to Postgres whenDATABASE_URLwas unset. Newtests/common/mod.rsboots apostgres:16-alpinetestcontainer on the fly; tests now always run, liftingepic-db/src/transaction.rsfrom 0 % → 100 % coverage. Workspacecargo llvm-cov --testsreports 86.95 % line coverage, clearing the CLAUDE.md §5.3 80 % floor. (5) ci.sh Stage 6 k6 soft-skip: the placeholder script slept 10 ms and skipped silently when k6 wasn't installed. Replaced with a Rust-native load smoke (epic-control/tests/load_smoke.rs) — 200 concurrent VUs, 10 s constant-arrival workload against/health, asserts p99 < 50 ms with zero transport errors. Local run reports ≈ 275k req/s, p99 ≈ 1.2 ms. Stage 6 now hard-fails on envelope breach; the k6 placeholder script is removed.ADR-021records the Phase 0 choice + Phase 2 revisit triggers so the decision isn't forgotten whenepic-edgeships. DESIGN.md §18 + §20 updated to reference it. - Sprint 0.6 — Phase 0 sign-off (gating). Three audit documents
land under
docs/audits/covering code quality (clippy gauntlet clean, 1 unsafe site with// SAFETY:, 15 library-code.expect()/.unwrap()sites all carrying// INVARIANT:comments after an in-audit normalisation, 10 files >500 lines all architecturally justified), security (cargo audit / deny / geiger clean, RLS posture verified 10/10 + 14/14 against testcontainer-spawned Postgres, anti-enumeration timing test ≤ 1.5× ratio, refresh-token reuse-detection 7/7 property + concurrency), and architecture (10/10 design questions pass — dependency direction, no cycles, foundation crates I/O-free, no raw entity IDs in public APIs, two-plane separation honoured, in-memory adapter perepic-traittrait,spawn_blockingdiscipline on Argon2id,EPIC_DATA_ROOTdiscipline). 17 criterion baselines saved asphase-0-finalacrossepic-crypto,epic-protocol,epic-auth,epic-store,epic-db— every Phase 0 SLO target met (RLS overhead +8.5 % vs 10 % ceiling,transaction_begin161 µs vs 1 ms, JWT sign 10 µs / verify 33 µs, BLAKE3 63 ns, layered session cache hit 690 ns). aarch64 cross-compile + 324 unit tests pass underqemu-aarch64-static.cargo llvm-cov --workspacereports 89.31 % line / 89.91 % region coverage; 100 % onepic-db::Transactionand every parser file. 10/10 sampled commits byte-identical in/data/backups/epic.git; DR drill from latest snapshot →ci.sh --quickStage 0+1 green in 39.91 s. Three in-audit remediations land alongside: missing// Reason:on one clippy allow,Reason:prefix on a crate-level allow,// INVARIANT:comments uniform across the workspace,ci.shStage 2cargo geigerswitched from silently-failing virtual-manifest invocation to per-crate isolated-target-dir invocation, and the banned-shortcut allowlist extended todocs/audits/*.md. Seesprints/phase-0/sprint-0.6-sign-off.mdfor the full audit transcript and thephase-0-finaltag context. - Sprint 0.6 hardening pass (post-self-review, operator-directed
"highest quality production-grade long-term solutions even if we
have to redo work"): a candid mid-sprint audit surfaced six
findings; all six closed in-sprint.
(F-1)
ci.shStage 3 silently failed when the integration stack was already up on port 5499 — added a precheck that detects the collision viass/lsofand surfaces both the cause and the two fixes. (F-2)epic_trait::UserInsert.password_hashflipped fromStringtoepic_crypto::PasswordHash, whose Debug truncates so the full PHC string can no longer leak through anyformat!("{insert:?}")callsite.epic-traitgains a workspace dep onepic-crypto(foundation→foundation, no layering violation).epic-test-utils::user_fixturenow produces a real fast-Argon2id PHC via a newplaceholder_password_hash()helper. (F-3)epic-authcoverage closed from 92.46 % → 98.52 % lines / 93.22 % → 97.59 % regions via 30+ supplementary tests across the seven verify-bearing files; the remaining 1.48 % is formalised in CLAUDE.md §5.3 as four categories of structurally-untestable code, with the literal contract reframed as the "verify-error-branch invariant". (F-4) aarch64 integration tests escalated from lib-only to the full workspace — 428 tests pass underqemu-aarch64-static, including the testcontainer-driven integration suites in epic-db / epic-store / epic-control / epic-vault / epic-postmark / epic-observability. Stronger than Ferrum's Phase 0 floor. (F-5)phase-0-quality-audit.md§2 "spot-check on 20 random pub items" claim was small + hand-picked; redrew viashuf -n 20and recorded the full 20-item inspection table inline. (F-6) Geiger artefacts went from 4 manually-captured crates to the full 26-crate sweep produced byci.sh --fullagainst the Stage 2 remediation. After the hardening pass: clippy / fmt / doc / tests all green; 64 test suites + 428 aarch64 tests + 0 failures. See sprint-0.6-sign-off.md "Sprint 0.6 hardening pass" for the per-finding closure record.
Changed
ci.shStage 5 — fixed sed pattern that mis-extracted the crate name from absolute paths emitted byfind, blocking fuzz harness discovery in Sprint 0.2's cargo-fuzz layout.ci.shStage 3 — bumped to thecargo sqlx prepare --workspace --checkinvocation matching sqlx-cli 0.9 (the-p <crate>flag was retired in 0.9).deny.toml— addedallow-wildcard-paths = truefor workspace path deps; documented why the architectural-wrapper warnings remain expected through Phase 0.DESIGN.md§10.8 — meter aggregates now spec'd with a surrogateBIGSERIALPK +UNIQUE NULLS NOT DISTINCTconstraint (Postgres rejects expressions in a PRIMARY KEY clause; the new form realises the sameCOALESCE(user_id, 0)intent in valid SQL).Cargo.toml—sqlxworkspace pin bumped 0.8 → 0.9 (latest stable per CLAUDE.md §3.7).epic-core::KeyType— variants reshapedSecret/Publishable→Sk/Pk/Orgto match the Sprint 0.3 migration constraint and ADR-006. The Sprint 0.2 form would have failed theapi_keys.key_typeCHECK on the first insert.epic-core::KeyScope— derivesOrd/PartialOrdso callers can hold a canonicalBTreeSet<KeyScope>for API-key creation.epic-trait::Store—insert_api_keywidened to take anApiKeyInsertstruct carrying every §10 column; addedfind_api_key_by_hash(constant-time lookup contract),get_api_key_by_public_id,touch_api_key, anddisable_api_key. The Sprint 0.2 stubs were too thin to issue a real key against.epic-trait::SessionStoreError::AlreadyRotatedcarries the family id (AlreadyRotated(i64)) so the auth layer can revoke the family on the theft path without a second hash-to-family lookup.epic-trait::sessiongainedSessionLookup,SessionRecord, andSessionLookupError— the cookie-side session-lookup contract that the auth layer parameterisesSessionServiceover and that the Postgres adapter in Sprint 0.5 will implement. The first cut of Sprint 0.4 placed this trait inepic-auth, which would have forced an upward dep onepic-authfrom any other adapter; moved to the foundation layer at sprint hardening time.epic-trait::Storegainedschedule_api_key_expiry(id, expires_at), used byApiKeyService::rotate_with_overlapto retire the predecessor without requiring an external scheduler.Cargo.toml— addedhmac = "0.12"andsha1 = "0.10"workspace pins for RFC 6238 TOTP. SHA-1 is the universally-supported authenticator-app digest; stronger digests break compatibility with the installed base.Cargo.toml— addedipnet = "2.10"workspace pin. Replaces a hand-rolled CIDR walker inepic-auth::api_keythat had a fail-open path on malformed allow-list entries.epic-test-utilsgainedMemSessionLookupadapter implementing the newSessionLookuptrait. The in-memory adapter was initially inlined as a#[cfg(test)] mod testsdefinition insideepic-auth::session; moved out so every downstream crate (Sprint 0.5 onward) can reuse it without reimplementation.epic-test-utils::MemStoreexposesapi_key_lookup_count() -> usizeso tests can assert structural anti-enumeration properties (the parse-fails-still-probes-the-store contract inepic-auth::api_key::ApiKeyService::verify).epic-auth::JwtServicenow consumes a typedKidinstead ofimpl Into<String>.Kid::parseenforces the[A-Z0-9_-]{1,32}format at the type-system boundary; the wire-sidekidfield is re-parsed through the same validator at verify time so a token with a path-traversal / control-charkidcannot reach the verifier-set lookup map.epic-auth::SessionCookietuple field is private; construction goes throughSessionCookie::generate()(OsRng) orSessionCookie::from_session_id(id)(explicit, for tests and vault rehydration).sid_claim()is a method onSessionCookienow, not a generic-static-fn onSessionService::<L>.epic-auth::ApiKeyService::verifyruns symmetric work on every input — BLAKE3 keyed-hash + indexedfind_api_key_by_hashlookup execute regardless of parse outcome, so an attacker cannot distinguish "malformed prefix" from "well-formed prefix but unknown key" by wall-clock timing.epic-auth::ApiKeyService::rotaterejects predecessors that are revoked, disabled, or already expired with the correspondingAuthError::Key*variants — rotating a dead key would create a confusing audit trail.epic-auth::SessionService::lookupthrottles thelast_seen_atwrite to a 30-second minimum interval. At production scale the per-request write rate against the Postgressessionstable is bounded by 1 / 30 s per active session rather than 1 per request.epic-auth::RefreshService::refreshfamily-revoke-failed log line uses a closederror_kindenum instead of the backend's Display form; future Postgres-adapterSessionStoreError::Backendpayloads can no longer leak bound parameter values into the log.cargo deny checkandcargo auditclean on the introduced workspace deps (ipnet,hmac,sha1).