Repository navigation
M3: drift detection — two-tier engine, fail-closed enforcement, exports, five-page dashboard - #3
Merged
Merged
Conversation
…lity notes Pre-M3 pre-work per plan Global Constraints: commit deck assets to main, ignore brag-output renders and legacy testdata residue. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…s documented Co-Authored-By: Claude Code <noreply@anthropic.com>
…/3, P-2) Co-Authored-By: Claude Code <noreply@anthropic.com>
…dders (R-M3-5) Co-Authored-By: Claude Code <noreply@anthropic.com>
Sixth service, services/drift (Postgres only, no Kafka): stateless window
recompute from the DB current view, tier-1 invariants from event zero,
INSERT-only drift_windows writes, single fail-closed loop.
- windows.py (pure): compute_windows closes on count_window rows OR when the
window's first row ages past window_time_s against the NEWEST row's
parsed_at (data-only horizon, never wall-clock); trailing partial held
back; FieldStats(nulls/violations/values) per mapped OCSF path plus
recorded unmapped.* keys; new_unmapped only against the baseline's
first-window key set; tier1_violation mirrors validation._value_sanity
(IPs strict dotted-quad str, ports/severity int-not-bool in range, time
ISO-parsable str; None is a null, never a violation).
- detect.py: Window -> drift_windows tuples; EMPTY HISTOGRAM GUARD (Task 3
review carry) — zero-parsed windows emit only the __rule__ row and empty
value lists never reach the shape machinery; __rule__ carries
match_rate = parsed/(parsed+parse_errors); action_taken stays NULL until
Task 5's enforcement.
- store.py: lazy-psycopg + autocommit house style; load_active_rules
(ActiveDriftRule = Rule + id + quarantined_fields), fetch_current_view
(unsuperseded active-version rows), insert_windows ON CONFLICT DO NOTHING
in one txn, baseline_for/insert_baseline (existence-guarded, once),
first_window_unmapped (baseline's first-window key set). INSERT-only —
the 007 rules UPDATE grants are untouched until Task 5.
- app.py: run_drift_loop(cfg, stop) — onboarding-loop shape, every failure
logged with traceback and retried next poll; main() single-threaded (ONE
loop, main thread).
- config.py: DATABASE_URL + ULPF_DRIFT_{WINDOW_COUNT,WINDOW_TIME_S,
BASELINE_WINDOWS,POLL_S}; defaults equal deploy/.env.example.
- compose: drift service, no profile (always on), drift_role DSN + four
knob passthroughs, depends_on migrate; docker compose config renders
under default/slm-4b/slm-8b.
57 new tests (windows 23, detect 12, store 12, config 3, app 7); suite
338 passed; ruff clean.
Co-Authored-By: Claude Code <noreply@anthropic.com>
…/11) Task 5 of the M3 drift plan, extending Task 4's loop: - detect.py: Finding + evaluate_window (tier-1 always, tier-2 null/match/ shape/volume ladders only against a baseline profile, per-field severity = max across signals; quarantined fields still measured and still escalate; unmapped.* are new-key-emergence signals only, alert-only, decide_action bypassed per the Task 4 review ruling); merge_findings (R-M3-11 per-field max); aggregate_baseline (FIRST-N-window aggregation, mean null/match, summed-then-normalized shape, __rule__ count bounds). - Shape-scale ruling: profiles store normalized shape_dist, but _window_scale rescales the baseline to the window's observed-value count before js_divergence — ulpf_core's add-1 smoothing is calibrated for count-scale histograms (T3 pins growth toward ln2), so raw proportions would cap every comparison near 0.32 and leave the severe band unreachable. - enforce.py (new): quarantine_field (array_append + NOT (... = ANY ...) + status='active' predicate, rowcount==0 -> False, NO audit on no-op) and deactivate_rule (expected-status predicate + deactivated_at), each audit paired in-transaction with actor 'drift'; record_windows overlays decided severity/action_taken onto measured rows (ON CONFLICT DO NOTHING); enforce_findings worst-first with ONE deactivate per fingerprint per scan and skip-after-deactivate. - store.py: fetch_window_history (ordered input for the baseline slice). - app.py: frozen pipeline order per scan — scan -> evaluate -> record windows -> FIRST baseline insert when due -> enforce worst-first; the establishing scan is tier-1-only by construction (documented ruling). 45 new tests (detect/enforce/app/store); full suite 383 green; ruff clean. Co-Authored-By: Claude Code <noreply@anthropic.com>
…preserved (R-M3-6) Co-Authored-By: Claude Code <noreply@anthropic.com>
- GET /api/drift/metrics: latest drift_windows row per (fingerprint_id,
field) via DISTINCT ON ... window_start DESC, DEFAULT_LIMIT clamp
- GET /api/drift/alerts (controller P-4): one kind-discriminated list —
window rows (severity IN minor/moderate/severe) merged with audit rows
(field_quarantined/field_unquarantined/rule_deactivated, any actor),
both latest-first, each source and the merged list clamp-capped
- POST /api/rules/{fp}/unquarantine -> writes.unquarantine_field:
array_remove UPDATE under the 007 rules_role grant, TOCTOU predicates
(status='active' AND field = ANY(quarantined_fields)), existence
pre-check splits 404 (unknown fingerprint) from 409 (inactive rule /
field not quarantined), field_unquarantined audit paired inside the
same transaction, post-state quarantined_fields returned
- tests: +11 (8 app/query pins in test_gateway.py, 3 write-path pins in
test_writes.py), purely additive; suite 399 passed, ruff clean
Co-Authored-By: Claude Code <noreply@anthropic.com>
Both exports share fetch_export_rows (parsed-only current view, newest first, EXPORT_MAX=50000 clamp): /api/export/ocsf streams one OCSF doc per line (application/x-ndjson, non-parsed rows skipped — never a null line); /api/export/parquet returns a snappy attachment with a flat stable schema (ocsf as JSON string, ISO timestamps before the table). pyarrow pinned to 18.1.0 per controller P-6 (cp312+cp313 wheels; host==CI==image). Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The fifth and final §12 page (M3 Task 10): field health across the
latest closed drift windows (severity badge, null/match/violation
rates, shape top class, action), the ONE merged latest-first
enforcement/alert feed (window rows ∪ audit actions, kind-discriminated,
rendered verbatim per ruling P-4 — no client re-merge or re-sort), and
unmapped.* extension opportunities as distinct 'new telemetry key
observed' cards. The Un-quarantine button (actor 'ui') POSTs
/api/rules/{fp}/unquarantine and refetches both feeds; gating is a
latest-wins fold over the feed's field_quarantined/field_unquarantined
audit rows, because GET /api/rules/{fp} does not actually return
rules.quarantined_fields (gateway _RULE_COLUMNS carries 13 keys, none
of them quarantined_fields; Python is out of T10's scope) — both
mutations audit only real transitions, so the fold is exact within the
endpoint's row cap.
Shell: TABS gains Events (Task 9's page, previously unwired) and
Drift & Health, in spec §12 order; new App.test.tsx pins five-tab
switching (mocked './api', stubbed EventSource). types.ts mirrors the
drift endpoints key for key (DriftMetricRow, DriftAlertRow,
UnquarantineBody/Response); api.ts gains getDriftMetrics/getDriftAlerts.
Co-Authored-By: Claude Code <noreply@anthropic.com>
…ealth gates un-quarantine via getRuleHistory (T10, P-8)
Gateway (additive only): _RULE_COLUMNS gains quarantined_fields (14th key,
column order mirroring migration 001's rules table); both surfaces built
from it — GET /api/rules and GET /api/rules/{fp} — carry the key. No URL,
status-code, or existing-key changes; the DB column and gateway_role SELECT
predate this (migration 001). Fake-conn pins updated: RULE_ROW_KEYS +
rule_row() carry the key, both fetch_rules/fetch_rule_history SQL tests pin
the full literal SELECT list, and test_rule_history_shape pins TEXT[]
reaching the wire as a JSON array of strings.
Web: DriftHealth's Un-quarantine button now gates on the ACTIVE version's
quarantined_fields fetched per DISTINCT metrics fingerprint via the existing
getRuleHistory (P-8); the audit-feed latest-wins fold is removed (dead
code). types.ts RuleRow mirrors the 14th key exactly (string[]). Tests
reworked: gate pins (active-version gating incl. newest-inactive case,
one-history-per-distinct-fingerprint, POST {field, actor:'ui'} + full
refetch, 409 banner); Rules/ReviewQueue fixtures gain the additive key.
Co-Authored-By: Claude Code <noreply@anthropic.com>
…g corruption (T11, R-P3/P-9) --mode firmware-drift (+ --drift-after SECONDS, default 5): after the drift boundary, syslog lines handed to the sender pass through corrupt_syslog() with a phase escalating 0->1 over the remaining run budget. Selection is R-P3's Knuth-hash predicate on the line index (strict <; seed = index, no RNG state); selected lines split 1:2 per R-P9: SRC->SRCADDR rename (null-rate spike) or SRC=uplink-trust-0x4f (tier-1 violation). Only line content mutates — counts, the other corpora, and the M1 stdout contract are unchanged; steady/new-appliance paths untouched (drift_after=None). 19 new tests in simulator/tests/test_drift.py; simulator 49 passed, full suite 428 passed (baseline 409), ruff clean. Co-Authored-By: Claude Code <noreply@anthropic.com>
Live Task 12 gate finding: psycopg's Json() does not convert datetime, so the field_quarantined/rule_deactivated audit INSERT raised TypeError at execution time and rolled the paired mutation back — the deactivate never landed while the service retried every scan. Fake-conn unit tests never serialize, so the bug class was invisible to them; the two touched tests now pin that the detail survives plain json.dumps. Co-Authored-By: Claude Code <noreply@anthropic.com>
Live Task 12 gate finding: Task 5 added services/drift/enforce.py but never extended the Dockerfile's explicit COPY list, so the built image crash-looped with ModuleNotFoundError: No module named 'drift.enforce'. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ration I-L run FIRST (before the unchanged A-H block): the drift demo needs the syslog baseline state to evolve during the firmware-drift run, and steady A-H traffic first would arm a degenerate baseline that makes J unreachable. - tune_drift_run replicates the simulator's deterministic corruption math host-side and plans --eps/--drift-after/--duration (one run at view offset 0; a minor establishing pre-run + a main run otherwise) so the J-window's tier-1 violation rate lands mid-band [0.27, 0.43] on a window that STARTS inside the run (I's window_start delta filter and J's poll both require it). - K stops the drift container before the human restore (severe windows are permanent rows of the version's view; a live loop re-deactivates the restored rule within one poll) and seeds a few steady lines after the deactivation is confirmed so the onboarding re-entry (sample_count growth) is race-free. - restore_syslog_rule: version-bump dance (successor without the SRC mapping) — the M2 sweep supersedes every poisoned v1 row, emptying v1's drift view before drift restarts, so consecutive runs pass on the accumulating DB. - Demo env: ULPF_DRIFT_WINDOW_COUNT=30 + a time horizon no demo gap can cross (count-close only; the brief's 15 s horizon slices the days-spread M1/M2 history into one-row windows and arms an all-zero baseline — observed live), and ULPF_SAMPLE_CAPTURE_CAP=2000 via a new compose passthrough (the accumulating DB saturates the 200/fingerprint cap and K's re-entry signal becomes unobservable — observed live). - Simulator runs now pass --build: profile-gated services are not built by up --build, so runs were executing an M2-era image without firmware-drift mode (observed live). Co-Authored-By: Claude Code <noreply@anthropic.com>
R-M3-1 (DB current-view polling, rejected alternatives, R17 topic note), the stateless window model, migration 007's grant matrix verbatim, the two-tier detection model, quarantine new-parses-only semantics, audit vocabulary additions, per-source volume deferred to M4, and the measured live findings from the Task 12 exit gate (enforce audit serialization, the accumulating-DB window-horizon and sample-cap findings, the reactivation-needs-a-version-bump recovery note). Co-Authored-By: Claude Code <noreply@anthropic.com>
The python job installs every service's requirements except drift's; its psycopg pin is identical to gateway's today, but the suite collects services/drift tests, so a future drift dep change must reach CI through this line rather than by accident. pyarrow==18.1.0 rides gateway's requirements (py3.12 manylinux wheel) — documented in the workflow header. No test-count pin exists in the file (plain pytest). Co-Authored-By: Claude Code <noreply@anthropic.com>
Live Task 12 finding: under this box's load the collector's librdkafka producer queue delayed messages by many minutes and retry-duplicated posted corpora, breaking K's sample poll and G/H's per-run deltas on otherwise-green runs. A graceful collector restart runs the app's shutdown flush (uvicorn SIGTERM -> lifespan -> producer.flush), bounding delivery to seconds. The collector is a stateless forwarder; the restart is benign. Co-Authored-By: Claude Code <noreply@anthropic.com>
…pin, docstring number (M3 final review) Co-Authored-By: Claude Code <noreply@anthropic.com>
…2 restored with measured convergence The final-review fix wave replaced ~0.32 with ~0.057 on the strength of two reviewer derivations. The re-review challenged it; executed against the real js_divergence: the described pre-rescale call (count-scale hist vs stored proportions) caps 0.2404@N=20 -> 0.3178@N=10k (~0.32); 0.0566 is the cap of a props-vs-props shape detect.py never makes. Rescale rationale unchanged (the actual rescaled call reaches 0.6853 ~ ln2, full ladder range). Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this ships (spec §15 M3 exit)
services/drift/): stateless window recomputation from the DB current view (R-M3-1), tier-1 invariant detection from stored JSONB, tier-2 statistical field-health with per-(fp, version) baselines, graduated fail-closed enforcement — field quarantine → rule deactivation → automatic re-entry into onboarding; humans override any state (un-quarantine, reactivate)drift_rolegrants (natural PKs, no serials — retires the 004–006 sequence bug class); file pinned by a regression testquarantined_fieldsat apply time — mappings filtered, values preserved inunmapped(R-M3-6: new parses only)quarantined_fieldsexposed additively on rule surfaces--mode firmware-drift— deterministic escalating syslog corruption (seed = line index, no RNG)docs/m3-drift.mdruling record (grant matrix byte-verified against 007)Evidence
58576f8; the two subsequent commits (73f809adocs+pin-test,7169b49docstring) are non-behavioral, verified as such by scoped re-reviewgit archive→python:3.12-slim→ ci.yml steps verbatim (ruff + 429 pytest)COPY enforce.pyin the drift imageKey deviations from plan (all review-ruled)
Redpanda-protocol naming held; K reaches severe via the tier-2 null ladder (disclosed + reasoned);
WINDOW_TIME_Suses a count-close horizon (15 s arms a degenerate baseline on accumulated history — documented);ULPF_SAMPLE_CAPTURE_CAPpassthrough (smoke raises 2000); K-restore via version-bump dance so 12/12-twice is possible on one DB; collector producer drain bounded (env-specific librdkafka delay/dup).