fix(0222): record the deploy and the FILL induction - #250
Merged
Conversation
Prices-production-Observability deployed alone at 17:32 UTC; all six no-invocations alarms now evaluate FILL(invocations, 0), and the ledger-processor alarm moved to 2/2. AC 4 induced the same hour: a temp alarm with no actions went OK to ALARM at 17:46:28 on a trailing gap, which is the first evidence that an alarm built on FILL transitions rather than only that FILL returns zeros to get-metric-data. Run at Period=300 rather than the 900 the AC specified. At 900 the third empty bucket completes exactly at :30, racing the next sweep run, so it is a less reliable test rather than a stricter one. AC restated, not silently substituted. Two findings from stateReasonData outrank the tick: evaluation windows are query-anchored and slide rather than aligning to the clock, which falsifies the bucket arithmetic used for the earlier control run; and on the corrected baseline FILL detects silence roughly an order of magnitude faster than the raw single-metric form. A third observation - the newest evaluated bucket had closed only 28 seconds before being read as zero - is the first supporting evidence for the flap risk that motivated 2/2. AC 7 remains open: the ledger-processor flap watch is a single describe-alarm-history read, deferred to 2026-08-26 because history persists and no attended watch is needed.
24 consecutive hourly datapoints, no gaps: Duration max 6,057-10,807 ms against a 240,000 ms threshold, Invocations 1/hour, Errors 0/hour. The alarm's StateUpdatedTimestamp is unchanged since the hand-off, so it has not flapped in between. The baseline recorded at hand-off is stale and would mislead a later check. It quotes ~27,000 ms with a stage split including a ~21 s coarse sweep; task 0218 moved that sweep into its own Lambda on 2026-08-24, so the remaining 6-11 s matches the 1m-pass and historical-sweep figures. The drop is the split showing up in the metric, not a change to chase. Notes the demonstrated-write-load criterion as still outstanding: it needs to land in only one of the six remaining checks, but a quiet week does not count as a soak.
The 2/2 ledger-processor flap did not materialise. Zero state transitions in the 13.5 h since the 17:32 deploy; StateUpdatedTimestamp is still 2026-07-09 and the only retained history item is the ConfigurationUpdate at 17:31:40, so the metric-math conversion did not even reset the state. The OK was checked for substance rather than taken at face value: 56 of 56 expected 900 s buckets over 17:00-07:00, 155-161 invocations each, and FILL output byte-identical to raw with no filled buckets. Adds Finding 4 — a hand-run get-metric-data FILL query whose window ends in the future fabricates a trailing gap. A 12:00Z end-time against a 07:02Z clock returned 19 consecutive zero buckets that read exactly like a 4h45m ingestion halt. Bound the window at now. It also strengthens the 2/2 judgement call: if future buckets fill as zeros, so does an incomplete current one. 7 of 7 criteria resolved — 5 ticked as written, AC 4's Period and AC 5 restated with the reasoning recorded. Follow-up 0223 already spawned for the NOT_BREACHING siblings.
adamkoot
added a commit
that referenced
this pull request
Aug 27, 2026
0222 was already taken on develop by the no-invocations alarm bug (PR #250) before this branch merged it in; two backlog files shared the id and [[0222]] resolved to both. 0226 is the next free id; the two code comments that cited it are re-pointed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FILL(invocations, 0)fix (Prices-production-Observabilityalone, 2026-08-25 17:32 UTC) — all six-no-invocationsalarms verified on metric math,ledger-processorat2/2, cleanup ruleDISABLEDbefore and afterOK → ALARMat 17:46:28, the first evidence that an alarm built on FILL transitions rather than only that FILL returns zeros toget-metric-dataPeriod=900to the300actually run: at 900 the third empty bucket completes exactly at:30, racing the next sweep run — a less reliable test, not a stricter onestateReasonData: evaluation windows are query-anchored and slide (falsifying the earlier control run's bucket arithmetic), FILL detects silence ~8× faster than the raw form, and a bucket closed 28 s before being read as zero — the first support for the flap risk behind2/2ledger-processorflap watch is a singledescribe-alarm-historyread on 2026-08-26. Keeping this PR open until it lands, then the task moves tocompletedand archives on this branch.