ops(0455): trustworthy deploy diff, dashboard coverage, guard fails closed - #427
Open
karolko9 wants to merge 14 commits into
Open
ops(0455): trustworthy deploy diff, dashboard coverage, guard fails closed#427karolko9 wants to merge 14 commits into
karolko9 wants to merge 14 commits into
Conversation
…ricFilter Second slice, first item. The processor fails in two disjoint modes and each is invisible in the other's metric: a crash raises Lambda Errors but a failed CH write does not (the handler reports batch-item failure so SQS redelivers — Errors stayed 0 through the whole 0454 outage), while ChWriteFailures, the filter-minted metric the zero-tolerance alarm reads, counts exactly those quiet failures. The errors widget now carries both series, so a ch-write-failures page has a dashboard answer instead of a widget showing zero. The declared-vs-emitted guard gains a second legitimate publisher kind: a name read via namespace: passes if Rust publishes it OR a logs.MetricFilter in the same stack mints it (the filter's PATTERN is already guarded by the filter-literal suite, so the chain stays covered end to end with no link checked twice). Without this the widget's read of ChWriteFailures — which never occurs in Rust by construction — failed CI. Sabotage-verified on this branch: a fabricated BogusNeverPublished read goes red with the new message; restored, 5/5.
The relaxation shipped in 9d23043 inferred filter-minted metric names by proximity — every metricName within 400 characters of a metricNamespace:. Smell-tested on the operator's hunch and it failed: a fabricated metric read inserted next to the filter block was accepted silently (6/6 green). Proximity is not provenance, and the failure was in the one direction a guard must never take. Replaced with an explicit one-entry list naming the filter that mints the name. Adding an entry is now a deliberate act; forgetting one fails CI loudly, which is the safe direction. Same smell test now goes red with a message that says what to do. Noted in the comment: the exact form of this check asserts against the synthesized template (Template.fromStack), where filter transformations and dashboard bodies are real resources instead of source text — worth building if the list ever grows past a couple of entries.
…y blank line H1: the CH-write matrix cell said "partial" — the errors widget now carries the ChWriteFailures series, so the cell names it. H2: the escape hatch gains the cold-start query. The widget removed in the first slice graphed InitDuration, which is not a CloudWatch metric at all; Logs Insights does parse it off every REPORT line, so the data was never lost — only the way to reach it was undocumented. Also strips a trailing blank line that 9d23043 carried in: it was left by that commit's sabotage test (append comment, delete comment, blank line survives), and the pre-commit hook did not catch it because it runs a single `nx format:write` pass and that command is not idempotent — the same trap recorded in 6e37fb4. `format:check --all` was red on the branch tip until now.
D6 — the last alarm without a widget (enrichment worker error rate) gets one. Plots the raw error COUNT, not the alarm's errors/invocations ratio: at this worker's traffic the ratio is unreadable (a 1-of-1 window is 100%), while the count is what an operator compares against the enrichment DLQ depth sitting beside it — errors climbing with a flat DLQ means the retries are absorbing them. D8 — Cost section, the dashboard answer 0449 asked for. Cost Anomaly Detection publishes no CloudWatch metric, so the only graphable spend signal is AWS/Billing EstimatedCharges: cumulative month-to-date (the slope is the daily burn, the line resets on the 1st), ~6 h refresh, us-east-1 only, account-wide — the same scope the anomaly monitor watches, so alert and graph cannot disagree. It exists for the case the monitor structurally cannot see: slow creep never looks like a step change, and budgets were dropped 2026-08-10. Layout: the failure row was at five widgets (30 of 24 units), so concurrency moves to its own context row. The failure row now reads as 'every one of these has an alarm behind it'. Also records, in the guard's module doc, the three-way comparison behind its shape (grep test / synthesized-template assertions / SSOT codegen) with the trigger that should promote it — so the next reader inherits the decision instead of re-arguing it.
…d slice The branch predated the alarm-mute hotfix and still carried the single-statement topic policy that caused it. Merging first means the second slice builds on the fix rather than colliding with it. One conflict, docs/runbooks/health.md coverage matrix, resolved as a union: develop's canary row and delivery-chain incident record kept, this branch's CH-write dashboard cell grafted over develop's placeholder. No cell dropped from either side.
…as permanently dirty cdk diff reads the deployed template back through a path that mangles non-ASCII. Byte-checked 2026-08-19: the live alarm carries e2 80 94 (an em dash) while the read returns 3f (?). Nine alarm descriptions and one dashboard title therefore showed as changes that survive being deployed - ten entries that never go away. That matters because the diff is the only gate before a production deploy. A gate that always shows ten false entries teaches the reader to scroll, and the topic-policy change that muted every alarm for 19 hours on 2026-08-18 was in a diff that was read and approved. Em dash to hyphen, arrow to ->. Comments untouched; they never reach the template. A header comment records why, so the typography is not restored as an improvement later.
…ionale reach the slice One conflict, the same coverage-matrix table as the previous merge, resolved the same way: develop's side kept (the delivery-chain cell now reads DECIDED rather than OPEN DECISION, and the note naming the config as the source of every threshold quoted in the table), with this branch's CH-write dashboard cell grafted over develop's 'partial' placeholder. Verified no cell was lost from either side: 17 rows before and after.
ADR 0054 rule 4 — one knowing page per planned pause is cheaper than suppression logic — was argued from scratch in three separate alarm comments, and rule 3's paging-twice-mutes-alarms in a fourth. Each restatement was correct and none was needed: the rule has a home, and a citation is shorter than a re-derivation and cannot drift from it. Cut: the backlog-age discriminator paragraph, the >10-threshold rejection rationale, the 5xx paragraph restating rule 2 verbatim, and one clause in the lag alarm pointing at coverage the alarm below already claims. Kept, deliberately: every measured number, every incident that explains why an alarm has the shape it has, the ASCII and topic-policy DANGER blocks, and the tombstone recording two removed widgets. Those are the point of the file. Net 11 lines. The review that raised this described four deletable essays; the file does not contain them — it contains one argument repeated four times, around measurement records that earn their space. Comment ratio 0.68 -> 0.66. Build green, tests 4/4.
The processor and enrichment-worker error-rate alarms were 19 code lines each and differed in five: construct id, alarm name, description, which function feeds the expression, and the series label. The expression, both periods, the comparison and evaluationPeriods were duplicated, so the shape had to be changed twice or the two diverged unnoticed. threshold and treatMissingData stay at the call site on purpose. The number must not become a fourth place a threshold lives, and 'no datapoint' means something different for each Lambda — 0 invocations is a pause or a dead input for the processor, and the normal state for the gated worker. ADR 0054 rule 3 wants that reason beside the alarm; a shared default would erase it, which is how an alarm slept through a 16 h stall on 2026-07-08. Measured, because the number matters more than the intent: code 650 -> 644, comments 430 -> 438. Six lines of code saved, eight spent explaining the seam. Extraction at two call sites does not pay for itself in volume — what it buys is that the shape exists once. Recorded here so the trade is visible rather than assumed; the same accounting is why the whole-file alarm factory (finding 18) was rejected. Construct ids unchanged, so logical ids and the deployed resources are untouched. Build green, tests 4/4. Template equality is not machine-checked here — cdk diff before deploy is that gate, and it is now trustworthy.
…o copies" This reverts commit 3f00706.
Two gaps were raised: this test checks neither the metric dimensions an alarm reads nor which log group a filter reads against which crate emits the literal. Both are real. Neither is fixed, and that is the decision, not an omission: closing them inside a source-text scraper means adding two more heuristics, which the header already ruled to mean 'go to the synthesized template instead'. The template version buys exactness for one filter, one namespace, two metric names and one log group, and costs a fixture for a constructor taking two Lambdas, three queues, an ECS cluster and a REST API. On the same arithmetic that rejected the alarm factory and the metric constants, it does not pay yet. Three things were measured before deciding, and are recorded so nobody re-derives them: - the filter-literal half is EXACT — a renamed emit site fails, and so does a rename to a superstring of the declared literal, because the pair regex requires the closing quote; - the metric-name half is the inexact one, and is already fail-closed through an explicit list; - Nx does track the Rust sources this test reads — breaking an emit site invalidates the cache and the test re-runs, which matters because a cached pass here would be a silent skip of the guard against silent skips. One trigger added: an emit site moving out of crates/indexer. That is the one live scenario the log-group gap would hide.
…lity-second-slice
…erge The CH-write dashboard cell was grafted over develop's placeholder when the branches were reconciled, which left that row's column padding off. Formatter only; no cell content changed. Caught by running the CI format gate locally rather than after a red run.
karolko9
added a commit
that referenced
this pull request
Aug 20, 2026
Sixty-eight findings came out of the post-incident review and every one now has a disposition; the task README had no record that any of it happened. Added: where the 68 went, what shipped from them, and the lesson that cost most of the effort. Every finding naming a file and a line held up. Every aggregate characterisation did not — 'about 600 lines of copy-paste' was 226 in total, '~30 findings on other people's files' was nine files all belonging to this operator, 'four justification essays' was one argument repeated four times. Ten findings died on arithmetic that was never done when they were written. That is why the alarm factory, the error-rate helper and the shared metric constants were rejected or reverted instead of shipped on three reviewers agreeing. Three acceptance criteria corrected rather than ticked: - dashboard/alarm coverage advanced with PR #427, and records that sharing a metric constant between alarm and widget was skipped despite favourable arithmetic, so their equality stays an assertion; - latch-proofing now states that production contradicts it knowingly — the enrichment DLQ alarm has been in ALARM for 47 days and is mute by construction, skipped on the operator's call; - the child-task criterion names the five tasks spawned (0507-0511) and the four extended, and records 0449 moving backlog to active because its detection half has been live since the last release. Still ten criteria open. None was ticked today, because nothing was verified today that would tick one.
karolko9
added a commit
that referenced
this pull request
Aug 20, 2026
…d as written Two close honestly. Cost: the monitor and its IMMEDIATE subscription to the alarm topic were confirmed by read-only checks after the deploy, and the boundary is written down rather than glossed — the last hop to the channel is not proven, which is true of every alarm here and is what ADR 0054 rule 5 now gates. API types: a stated N/A is answered, not pending. The eight left are sorted into three shapes so the next session does not re-derive them. Four need a production window — a deliberate pause, a simulated stall, a real per-stack release tag. One is a measurement that the deploy has now unblocked. One is in flight with PR #427. One is an operator decision on re-scoping child tasks. One is gated on another. The eighth cannot be ticked at all: 'no alarm can sit latched and mute' is contradicted by an alarm that has been in ALARM for 47 days and was skipped knowingly. Both honest ways out are recorded — drain the queue and let the criterion mean what it says, or narrow it and write the reason. Leaving it open and unexplained is the only option that makes the task lie. Four of seven remaining need production, not code. That is the real state of this umbrella: the building is done, the proving is not, and the proving is deliberately not something that can be faked from here.
…lity-second-slice
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cdk diffstops lying. Reading a deployed template back mangles non-ASCII (byte-checked: the live alarm carriese2 80 94, the read returns3f), so nine alarm descriptions and one widget title showed as changes that survive being deployed. Synthesized strings are ASCII-only now, with a header comment recording why — the diff is the only gate before a production deploy, and one that always shows ten false entries teaches the reader to scroll.Refs #422 predecessor. Task 0455.