Repository navigation
M2: onboarding loop: candidates, review queue, versioned rules, backlog re-parse - #2
Merged
Merged
Conversation
…rors, env example) Co-Authored-By: Claude Code <noreply@anthropic.com>
…hardening Co-Authored-By: Claude Code <noreply@anthropic.com>
… hardcoding) validate_candidate runs all eight CHECK_KEYS (caps/allow-list via the M1 gate, per-line parse + IP/port/severity sanity, held-out match rate, orphan mappings over observed fields, 20-mutant seeded adversarial probe, hardcoded-literal token diff) and returns a frozen CandidateReport; report_to_json emits the rules.validation JSONB shape. The bench's mutation machinery moves verbatim into ulpf_core.mutations (single definition; bench passes vary_text=True, the probe mutates values only — patterns may pin free words like DROP). kv_rule test pattern accepts DROP|ACCEPT: the brief's ACME held-out line is an ACCEPT event while its own test expects held_out_match_rate 1.0. Co-Authored-By: Claude Code <noreply@anthropic.com>
…7 txn/callback fixes Pipeline now ships one EventEnvelope per event (parsed, unparsed, parse_error) to normalized.events keyed by fingerprint_id — the R17 contract M3's drift service will consume — and captures onboarding samples for rule-less fingerprints in the SAME transaction as the normalized rows (grouped per-batch count query, per-fingerprint cap via ULPF_SAMPLE_CAPTURE_CAP, ON CONFLICT (raw_id) DO NOTHING). T7 minors: connect_db runs autocommit and persist_batch/persist_normalized wrap exactly one `with conn.transaction():` block each (zero-event batches open no txn; failures still propagate to the rollback+seek-back path); KafkaProducer passes an on_delivery callback that logs delivery failures at ERROR. Collector caps: IngestRequest.lines max_length=1000, format_hint and RawEnvelope.format_hint max_length=64; TCP lines read with a StreamReader limit so an oversized line closes the connection (README documents the UDP-drops-event vs TCP-kills-connection asymmetry). migration 004: GRANT USAGE on onboarding_samples_id_seq to pipeline_role — the first live sample insert exposed the missing sequence grant (caught by the Task 4 live check; migration 001 granted the table only). Co-Authored-By: Claude Code <noreply@anthropic.com>
…fast-path, validation gate Co-Authored-By: Claude Code <no-reply@anthropic.com>
…SON rules; retry transport errors Co-Authored-By: Claude Code <no-reply@anthropic.com>
Co-Authored-By: Claude Code <no-reply@anthropic.com>
Co-Authored-By: Claude Code <no-reply@anthropic.com>
…try scoping Co-Authored-By: Claude Code <no-reply@anthropic.com>
Three-tab shell (Overview / Review Queue / Rules placeholder) on plain useState — no router, air-gapped. ReviewQueue lists pending_review candidates with the gate's checks grid (keys verbatim), held-out preview pairs, mapping editor (edited_mappings), pattern override textarea (override body), actor/reason inputs, and Approve/Reject with disabled-while-submitting + error banner; list refetches after every action. api gains postJson/getRules/getSamples; types mirror the Task 7 contract key-for-key (RuleRow, ValidationReport, SamplesStatus). T10 polish: TraceDrawer fetch now aborts via AbortController. Co-Authored-By: Claude Code <no-reply@anthropic.com>
The Rules tab is the real registry (Task 9): GET /api/rules with no
status filter, grouped to one summary row per fingerprint (latest
version, status/provenance badges, activated_at). Expanding a row fires
BOTH getRuleHistory (version timeline: version, badge, provenance,
activated_at) and getAudit (the ts - actor - action feed; action
strings from the closed vocabulary render VERBATIM; the history
response's embedded audit array goes unused). Deactivate POSTs
{actor: "anonymous"} to /api/rules/{fp}/deactivate and refetches so
the badge flips. App.tsx swaps the placeholder for the component;
api gains getRuleHistory/getAudit; types gain AuditRow (6 keys, keyed
to the gateway SELECT) + RuleHistoryResponse/AuditResponse.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…nity numbers Co-Authored-By: Claude Code <noreply@anthropic.com>
…ocs: m2 ruling record M2 Task 12 exit gate. smoke.py extends the M1 A-E acceptance smoke with the M2 onboarding-loop asserts F/G/H: F takes an SLM candidate (or the ruled hand-authored manual fallback after ULPF_SMOKE_SLM_WAIT_S, the expected outcome on the default qwen3:4b tier) and G posts the generic zenwall rule, both through approve -> backlog re-parse verified by per-run deltas against the reparse_complete audit row; H re-runs the forced replay with active rules proving raw/normalized/stats/sample idempotency. CI installs the onboarding requirements and pins pytest/ruff; web uses npm ci. docs/m2-rule-lifecycle.md records the M2 rulings (grant matrix, topic envelope, version minting, deferred miner) and the migration-006 sequence-grant completion note the first live smoke run caught; README hero-loop and gate docs updated. Two consecutive green smoke runs on the accumulating database (runs C and D, 8/8 checks each); containerized Linux CI rehearsal green on the final staged tree. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Closes the M2 milestone (spec §15): new appliance → sample capture → candidate rule → human review in the web Review Queue → approved versioned rule live → backlog re-parses, with the full audit trail.
onboarding_samples; emits all-statusEventEnvelopes tonormalized.events(parsed / unparsed / parse_error) keyed by fingerprint; delivery-error callback; autocommit transaction hygiene.validate_candidate— the deterministic gate (caps/allow-list, per-line parse, held-out match, IP/port sanity, orphan mappings, 20-mutant adversarial probe, hardcoded-literal detection).UPDATE (superseded_by_event_id)— the only mutable cell in the event store).rules_role(approve, approve-with-edits, reject, manual authoring, deactivate/reactivate) + review/audit read endpoints. The M1 HTTP API contract is untouched (additive only).--mode new-appliance+newappcorpus.slm-4b/slm-8bprofiles (defaultqwen3:4b).docs/m2-rule-lifecycle.md, measured SLM numbers indocs/m2-slm-sanity.md.Evidence
ruff check .clean; web 25/25 vitest + production buildscripts/smoke.py→ PASS (A–H), twice consecutively on the accumulating DBruff+ fullpytestgreen on the exact final treeKnown residuals
docs/m2-slm-sanity.md: 0.0 held-out match across corpora). The demo path is manual authoring through the same loop; M4 owns the tier/harness question.