Repository navigation
M1 walking skeleton: end-to-end MVP (collector → Redpanda → pipeline → Postgres → gateway → web) - #1
Merged
Conversation
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…n bench Known-format ids (cef_<vendor>, leef_<vendor>, json, syslog, csv, xml) port the legacy-demo heuristics and add csv/xml branches; unknown lines get a provisional auto_<8hex> id from their shape hash so no shared unknown bucket exists (spec §9). Shape hashing collapses numerics, empty key=value values and free-text runs so same-format events hash identically; the bench measures this with 500 seeded mutants across 5 golden corpora (100% stable) and fails on known-format fragmentation. Co-Authored-By: Claude Code <noreply@anthropic.com>
…imeouts libs/ocsf-schema ships the curated Network Activity subset (draft 2020-12, root additionalProperties:false) and validate_document() for activation-time and canary use only (spec §6.4); time/severity_id/action/message are nullable so timestamp-less rules validate pre-pipeline, and jsonschema rides as the optional `activation` extra. ulpf_core.parsing is the one implementation everything imports: legacy apply_rule ported onto the regex module with a per-match ms timeout (spec §10), allow-listed mapping targets enforced at document build, safe severity/port casts (fixes the demo hot-path crash), full OCSF document shape, and EXTENSION_KV as the single shared grammar for M2 onboarding. validate_rule_output adds pattern length/group/nesting caps, compile, allow-list, sample-parse and document-validation checks for the human activation gate. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ation Review findings on 0d4ffe7: the extension key=value scan ran un-timeouted on untrusted line text — a crafted 32KB CEF extension stalled parse() ~3s under the default 50ms timeout (quadratic backtracking of (\w+)= against an a-wall); the scan now runs under the same ms-scale timeout and a timeout returns the standard error (spec §10). Activation validation now fails closed when ocsf-schema is absent (schema validator unavailable -> error, never a silent pass); the validator's internal jsonschema fallback is unchanged. Port casts keep only string values (dict/list -> None), and the exact OCSF_TARGETS contract is pinned by a committed test. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…logs Wraps every inbound line in a RawEnvelope and produces it to raw.logs, keyed by source_id; zero parsing (spec §7). UDP datagrams and TCP lines on :5514 derive source_id from the peer address (udp-<ip>/tcp-<ip>, controller R7 ruling — no device registry until post-M1); HTTP POST /v1/ingest takes the request-supplied source_id and optional format_hint. Envelopes are validated against RawEnvelope before produce; oversized/empty lines -> HTTP 422, produce failures -> HTTP 503 / syslog drop with error log (M1 queue-down deviation; spooling is M3). Tests use an injected fake producer with UDP disabled; root .dockerignore (R12) keeps build contexts from shipping the whole worktree. Co-Authored-By: Claude Code <noreply@anthropic.com>
…se, normalized emit Sole raw.logs consumer (group id exactly "pipeline" for Task 12's forced replay, R3b). Per 500-msg/2s batch, spec §4 ordering: build_rows assembles RawRows with fingerprint_id classified at build time (R3 — pipeline_role has no UPDATE) -> persist_batch commits one txn of staging COPY -> INSERT .. ON CONFLICT (raw_id, received_at) DO NOTHING -> per-partition raw_batches append (batch_seq = COALESCE(MAX,0)+1, prev_hash -> prior merkle_root, genesis 64 zeros, ordered row_hashes JSONB) -> active rules refreshed -> parse (cap re-checked; parse_error + pipeline.dlq with raw_id/fingerprint/error; canary validates every 100th parsed doc) -> normalized.events produced keyed fingerprint_id -> persist_normalized committed on (event_id, parsed_at) -> only then consumer.commit(). parsed_at := raw received_at so replays collide instead of duplicating; superseded_by_event_id stays NULL in M1 (R9). psycopg/confluent_kafka import lazily so unit tests run without them (R14). Dockerfile installs both libs editable (R15); regex~=2024.7 (R8); redpanda-init.sh topics --if-not-exists (R10); root pythonpath gains services/pipeline (R13). Co-Authored-By: Claude Code <noreply@anthropic.com>
…seeded) Co-Authored-By: Claude Code <noreply@anthropic.com>
… head Co-Authored-By: Claude Code <noreply@anthropic.com>
…p by_fingerprint
Review fixes: _json_default now serializes uuid.UUID (dict_row rows carry
UUID objects; SSE died on the first real row), zero_fill_statuses consumes
the {"status","n"} dict rows the query actually returns, by_fingerprint is
capped at 500 (R20), limit=500 boundary accepted, and test fixtures rebuilt
to mirror real psycopg dict_row shapes so this bug class gets caught here.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
scripts/smoke.py drives the live stack: lossless ingest (delta-based, DB never reset), per-fingerprint parsed/unparsed split via /api/stats, raw traceability (HTTP /raw + SQL join with recomputed sha256), forced Kafka replay (group deleted after the consumer stops, pipeline restarted: raw_events unchanged, raw_batches grows), and an independent recomputation of every raw_batches merkle root plus full per-partition chain linkage from genesis. Also: R19 hardening — _is_xml catches RecursionError alongside ParseError so a deeply-nested hostile line falls through to the shape hash instead of wedging a pipeline batch; redpanda-init.sh creates missing topics via list-then-create (v24.1.7 rpk has no --if-not-exists flag); Postgres publishes on host 5433 so host connections cannot land on a native Postgres owning 5432; ruff made repo-clean for CI (frozen legacy-demo excluded; import-order, blind-except and small dead-variable fixes); CI workflow (py3.12 ruff+pytest, node 22 web). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… review
_classify recursed once per '=' in a token, so a hostile single token
('a=' * 1200) raised RecursionError out of shape_hash and wedged a pipeline
batch — the failure class R19 fixed in _is_xml. Classification now loops
partition('=') with identical semantics (KV prefixes, empty value -> W);
golden + mutation-stability suites green unchanged.
Also per review: smoke host deps stated as psycopg[binary] only (Kafka
interaction is rpk via docker compose exec, not confluent_kafka); the
MEMBERS-0 wait tracks saw_members_zero and reports timeouts truthfully;
group-exists reuses group_listed(); CI pins ruff==0.16.8 and installs
simulator requirements; corpora-total comment corrected to 56. Plus:
compose() decodes docker output as UTF-8 explicitly — the Windows ANSI
codepage crashed a subprocess reader thread on build output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er final review CRITICAL-1: consume() advances the consumer position regardless of processing success and no-arg commit() commits that position, so a failed batch (rollback, no commit) was silently covered by the next successful batch's commit — its events never redelivered. Every failure path (persist_batch, persist_normalized, the R16 catch-all) now seeks each affected partition back to the failed batch's first offset before the loop continues; fake-consumer unit tests assert the exact seeks and zero commits on both persist-failure paths and the run()-level crash. CRITICAL-2: _known_format's JSON probe caught only JSONDecodeError, but deeply-nested hostile text (17,995 chars, under the 65,536 cap) raises RecursionError out of json.loads, escaping build_rows via fingerprint_id and blowing up the batch — silent loss before C1, infinite redelivery wedge after; landed together. Regression tests at the fingerprint and build_rows layers, red first. Also: worker comments/docstrings now describe the actual seek-back mechanism; compose notes the single-replica requirement for M1 batch_seq safety (scale-out = M3 with fencing/epochs); README quickstart points at --loop sustained traffic for the dashboard. Co-Authored-By: Claude Code <noreply@anthropic.com>
ruff 0.16.8 EXE001 fired on the linux runner for bench/fingerprint_fragmentation.py, deploy/migrate.py, scripts/smoke.py — each carries a deliberate #!/usr/bin/env python3 but was committed mode 644 (staged from Windows, where the exec bit is invisible to every local check). Set the executable bit via git; file content unchanged. Co-Authored-By: Claude Code <noreply@anthropic.com>
…py3.13 CI runs CPython 3.12, whose C json scanner parses depth-3000 nesting without error, so the Final-C2 regression tests (written against 3.13, which raises there) passed vacuously on the runner. Depth 10,000 raises RecursionError on both generations, and the 60,001-char line stays under the 65,536-char envelope cap, so it still exercises the ingest path. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M1 walking skeleton — complete
docker compose upon one machine → simulated multi-vendor traffic → collector → Redpanda → pipeline (idempotent raw-first persistence + per-partition hash chain + parsing) → Postgres → gateway → React Overview with live SSE feed and raw traceability.What's in here
libs/ulpf-core— single implementation of fingerprinting (shape-hash unknowns), parsing (regex timeout, OCSF allow-list), deterministic UUIDv5 idslibs/ocsf-schema— curated OCSF Network Activity subset + validatorservices/collector— syslog UDP/TCP :5514 + HTTP :8080 →raw.logsservices/pipeline— sole consumer group; raw persisted before parsing in the same batch; staged COPY + ON CONFLICT for replay idempotency; seek-back on failed batches (lossless); validated seed rules; DLQ; canaryservices/gateway— stats / events / raw-trace / SSE / chain-head API (contract frozen at M1)web/— React 19 + TS + Vite Overview page, Caddy-served, air-gapped (no CDN)simulator/— steady-mode traffic player over the golden corporadeploy/— compose stack, least-privilege DB roles, migrations, topic initscripts/smoke.py— acceptance gate: lossless counts, parsed/unparsed split, 20/20 traceability with recomputed hashes, forced-replay idempotency (consumer-group delete → replay → raw counts unchanged, chain appended), full independent verification of every merkle rootlegacy-demo/— the original hackathon demo, moved here frozen (history preserved viagit mv)Verification
Quickstart
cd deploy && docker compose --profile sim up -d --build
python scripts/smoke.py # needs psycopg[binary] on the host
open http://localhost:3000
Postgres publishes on host 5433 (5432 is typically taken by a local Windows Postgres).
M2 (onboarding loop + SLM) builds on the frozen API contract from here.