feat(history): add fenced rollup and retention maintenance by ammarheidari · Pull Request #236 · araditc/Kafdeck

ammarheidari · 2026-09-28T12:45:03Z

Authority

W63 #212 Slice 1 provider/query foundation is complete and protected-main verified. This PR starts admitted W63 Slice 2 on protected-main 65725a1ab329921fad6dea7e930ba9151e4981c7.

Scope

add bounded historical-metrics maintenance policy and lease/fencing contracts;
add a provider-neutral ADO maintenance store using the existing SQLite/PostgreSQL history connection factory;
acquire a single maintenance lease with monotonic fencing tokens;
serialize HA lease state with PostgreSQL row locking and SQLite write locking;
roll raw samples into deterministic 5-minute aggregate windows before raw deletion;
merge late raw samples cumulatively into an existing rollup rather than replacing prior evidence;
delete raw samples only after the corresponding rollup write succeeds in the same transaction;
expire rollups independently from raw retention;
bound each cycle by both max rollup windows and a server-owned duration deadline;
interrupt long-running SQLite maintenance through sqlite3_interrupt;
add a hosted maintenance worker with finite cycle interval and fail-closed retry behavior;
add SQLite rollup/late-data/fencing regressions and PostgreSQL single-active-owner evidence.

Safety boundary

no raw Kafka/generated payload or arbitrary request content is persisted;
no Kafka topic is used as Kafdeck history storage;
no maintenance worker can continue writes after a newer fencing generation takes over;
no raw retention deletion can occur before rollup persistence for that window commits;
missing evidence remains missing/unknown; rollup does not fabricate zeros;
no new external dependency or provider is introduced;
W64 historical trend/SLO integration remains dependency-gated on this maintenance contract.

Exact head: 4876a55da9660657063ffb9fc4c808ab1590b2b8.

Review remediation

PostgreSQL rollup/raw-delete transaction now uses REPEATABLE READ so concurrent appends after the rollup snapshot remain invisible to that cycle's delete and are preserved for the next cycle;
expired rollup retention is moved to a separate fenced transaction and capped at 1,000 rows/cycle by default (10,000 hard), so raw-rollup progress commits independently and retention backlog drains in bounded batches;
regression evidence covers PostgreSQL isolation mode and multi-cycle retention batching.

Canonical resynchronization

rebased semantically onto protected main 4beeab13ab9d251024d95aa28bcf4c6b8f3e9f8a;
canonical exact head: 8f62a84f0289b0adf507876656c75675fb243b06;
one DCO-signed commit;
raw sample identity ledger preserves append idempotency across rollup deletion;
per-window work is capped at 32 raw samples with set-based raw deletion;
legacy Partial state survives migration;
legacy rollup coverage remains unknown after late merges;
maintenance lease validity is checked against execution-time UTC, not only caller cycle time;
fresh exact-head CI, CODEOWNER approval and Codex review required before merge.

Final exact-head reconciliation

protected-main parent: 4beeab13ab9d251024d95aa28bcf4c6b8f3e9f8a;
canonical exact head: 4876a55da9660657063ffb9fc4c808ab1590b2b8;
one DCO-signed commit;
Final canonical head includes deterministic-window test anchoring, one-sided coverage rejection and exact full-batch window accounting.
all prior approvals are treated stale; fresh exact-head CI, CODEOWNER approval and Codex review are required.

chatgpt-codex-connector · 2026-09-28T12:45:18Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T17:09:22.499478Z	`4876a55`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

ammarheidari · 2026-09-28T12:45:24Z

@codex review

Please review exact W63 Slice 2 head 458c82ac44daed167d3ab4e1d15964167b67df53, especially HA lease/fencing, late-data rollup accumulation, raw-before-rollup deletion ordering, SQLite interruption, and bounded cycle semantics.

ammarheidari · 2026-09-28T12:47:52Z

@codex review

Please re-review exact remediated W63 Slice 2 head d7541ef7b241790d0add16c11f0b99edc95b03de. Prior CI-only xUnit analyzer failures were corrected without changing the maintenance authority model.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7541ef7b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T12:57:12Z

@codex review

Please re-review exact remediated head f1f4335e05be8abce934f532a977ebdeb67489fc. P1 concurrent-append loss is addressed with PostgreSQL REPEATABLE READ rollup transactions; P2 retention starvation is addressed with a separately committed bounded deletion batch.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1f4335e05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T13:37:40Z

@codex review

Please review canonical W63 Slice 2 head ea4fece41315dbfbfc758c1dc2f9b15744d9fb70, including rollup truth preservation, REPEATABLE READ late-write safety, bounded retention batching, HA lease/fencing, and schema-v2 regressions.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea4fece413

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T13:48:28Z

@codex review

Please re-review canonical W63 Slice 2 head 5190a36c24852a52980324e2085302cef2e5cbf0 after legacy-rollup coverage truth and incremental per-window commit remediation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5190a36c24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T13:54:55Z

@codex review

Please re-review exact canonical head 2523def677641cd7ab2b3dcfeb75dc8246160b94 after the latest compile-boundary remediation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2523def677

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T14:01:58Z

@codex review

Please re-review exact canonical head 0753ba189701e345103af1747e8edc16f9b5db53 after the latest bounded-work / contract-consistency remediation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0753ba1897

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ammarheidari · 2026-09-28T16:51:21Z

@codex review

Please review canonical exact head 61b7efe7f1b8ef69813154eaebfe770b532c07e0, focusing on raw-retry idempotency, legacy coverage/state preservation, execution-time lease fencing, and bounded set-based maintenance progress.

ammarheidari · 2026-09-28T16:54:58Z

@codex review

Please review final canonical exact head 8f62a84f0289b0adf507876656c75675fb243b06; prior product findings are remediated and the latest change only aligns deterministic rollup-window regressions.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f62a84f02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Refs #212 Signed-off-by: Ammar Heidari <ammar@arad-itc.org>

ammarheidari · 2026-09-28T17:04:24Z

@codex review

Please review final canonical exact head 4876a55da9660657063ffb9fc4c808ab1590b2b8 after the latest bounded-contract remediation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4876a55da9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-28T17:09:19Z

+            var sum =
+                group.Sum(row => row.Sum);
+            var count =
+                group.Sum(row => row.Count);


Prevent valid samples from overflowing rollup aggregation

Two raw samples can each pass HistoricalMetricSample.Validate() while their combined Count exceeds long.MaxValue; Enumerable.Sum then throws OverflowException here and rolls back the batch. Because every subsequent cycle selects the same oldest window, that valid input permanently blocks rollup processing for all later windows. Similarly, finite Sum values can combine into infinity, bypassing the finite-value invariant. Bound or safely validate accumulated aggregates before attempting the upsert.

Useful? React with 👍 / 👎.

Refs #212 #236 Signed-off-by: Ammar Heidari <ammar@arad-itc.org>

ammarheidari · 2026-09-28T17:13:46Z

Post-merge P2 aggregate-overflow finding is being corrected in PR #241. The finding remains unresolved until #241 is exact-head clean, merged and protected-main verified.

ammarheidari requested a review from akhiabanchian as a code owner September 28, 2026 12:45

ammarheidari mentioned this pull request Sep 28, 2026

v0.8 W63 — Historical metrics provider, retention and query contracts #212

Closed

ammarheidari force-pushed the w63-history-maintenance branch from d857046 to d7541ef Compare September 28, 2026 12:47

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs Outdated

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs

ammarheidari force-pushed the w63-history-maintenance branch from 9e96b88 to f1f4335 Compare September 28, 2026 12:56

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs Outdated

akhiabanchian previously approved these changes Sep 28, 2026

View reviewed changes

ammarheidari dismissed akhiabanchian’s stale review via 8473289 September 28, 2026 13:04

ammarheidari force-pushed the w63-history-maintenance branch from fd10f7d to ea4fece Compare September 28, 2026 13:37

ammarheidari requested a review from akhiabanchian September 28, 2026 13:37

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread src/backend/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricStore.cs

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs Outdated

ammarheidari force-pushed the w63-history-maintenance branch from 4d5988e to 5190a36 Compare September 28, 2026 13:48

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs

ammarheidari force-pushed the w63-history-maintenance branch from 3d9b664 to 2523def Compare September 28, 2026 13:54

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs

ammarheidari force-pushed the w63-history-maintenance branch from 6d564b5 to 0753ba1 Compare September 28, 2026 14:01

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

akhiabanchian previously approved these changes Sep 28, 2026

View reviewed changes

ammarheidari dismissed akhiabanchian’s stale review via 141db7c September 28, 2026 16:49

ammarheidari force-pushed the w63-history-maintenance branch from 9a1374e to 61b7efe Compare September 28, 2026 16:50

ammarheidari requested a review from akhiabanchian September 28, 2026 16:51

ammarheidari force-pushed the w63-history-maintenance branch from 3de501e to 8f62a84 Compare September 28, 2026 16:54

ammarheidari mentioned this pull request Sep 28, 2026

v0.8 — Observability, Automation & Platform APIs planning and implementation tracker #209

Open

2 of 11 tasks

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread tests/Kafdeck.Architecture.Tests/V08W63HistoricalMetricsPersistenceTests.cs

Comment thread src/backend/Kafdeck.Core/Observability/HistoricalMetrics.cs

Comment thread ...end/Infrastructure/Kafdeck.Infrastructure.Persistence/AdoHistoricalMetricMaintenanceStore.cs Outdated

akhiabanchian previously approved these changes Sep 28, 2026

View reviewed changes

ammarheidari dismissed akhiabanchian’s stale review via e2603e3 September 28, 2026 17:03

feat(history): add fenced rollup and retention maintenance

4876a55

Refs #212 Signed-off-by: Ammar Heidari <ammar@arad-itc.org>

ammarheidari force-pushed the w63-history-maintenance branch from e2603e3 to 4876a55 Compare September 28, 2026 17:03

ammarheidari requested a review from akhiabanchian September 28, 2026 17:04

akhiabanchian approved these changes Sep 28, 2026

View reviewed changes

akhiabanchian merged commit c1e6f1f into main Sep 28, 2026
9 checks passed

akhiabanchian deleted the w63-history-maintenance branch September 28, 2026 17:09

chatgpt-codex-connector Bot reviewed Sep 28, 2026

View reviewed changes

ammarheidari added a commit that referenced this pull request Sep 28, 2026

fix(history): bound rollup aggregate overflow

e3a667f

Refs #212 #236 Signed-off-by: Ammar Heidari <ammar@arad-itc.org>

ammarheidari mentioned this pull request Sep 28, 2026

fix(history): bound W63 rollup aggregate overflow #241

Merged

Conversation

ammarheidari commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Authority

Scope

Safety boundary

Review remediation

Canonical resynchronization

Final exact-head reconciliation

Uh oh!

chatgpt-codex-connector Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ammarheidari commented Sep 28, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

ammarheidari commented Sep 28, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 28, 2026 •

edited

Loading