Skip to content

audit_event_catalog.md

Latest commit

 

History

History
236 lines (184 loc) · 29.8 KB

audit_event_catalog.md

File metadata and controls

236 lines (184 loc) · 29.8 KB
Raw

Audit Event Catalog

Audience: ForgeLM operators, regulators, and downstream verifiers reviewing ForgeLM's training-pipeline audit log. Mirror: audit_event_catalog-tr.md

This catalog enumerates every event ForgeLM may append to audit_log.jsonl, ForgeLM's append-only, hash-chained training-pipeline audit log. It is development-time evidence that can support the technical documentation (Annex IV) and quality-management record-keeping; it does not by itself satisfy EU AI Act Article 12, which requires the deployed high-risk system to log events automatically over its lifetime. Every entry shares a common envelope; the event field selects one of the rows below.

Common envelope

Each line is a single JSON object with at least the following fields:

Field Type Description
timestamp string ISO-8601 UTC timestamp (datetime.now(timezone.utc).isoformat()).
run_id string Stable per-training-run identifier (fg-<uuid12>).
operator string Human-attributable identity, resolved by AuditLogger in priority order: (1) $FORGELM_OPERATOR if set; (2) <getpass.getuser()>@<hostname> when the username can be resolved; (3) anonymous@<hostname> only when getpass.getuser() fails and FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 is set (otherwise the run aborts loudly).
event string Dotted event name from this catalog.
prev_hash string SHA-256 of the previous line ("genesis" for the first entry). Forms the tamper-evident hash chain.
_hmac string? Present only when FORGELM_AUDIT_SECRET is set. HMAC-SHA-256 of the line without _hmac.
payload varies Event-specific keys, listed per row below.

The hash chain advances after the line lands on disk (flush + fsync), so an unclean shutdown leaves the chain intact for resume.

Event vocabulary

Pipeline lifecycle

Event When emitted Payload (in addition to envelope) Article
pipeline.initialized ForgeTrainer.__init__ finished wiring config + audit logger; emitted before any model load. model, trainer_type 12
training.started Trainer begins a fine-tuning run. (no payload — envelope only) 12
training.oom_recovery OOM recovery path halved per_device_train_batch_size and retried (mid-training event). old_batch_size, new_batch_size, new_grad_accum 12 / 15
benchmark.evaluation_completed lm-eval-harness finished evaluating the configured benchmark suite. passed, average, scores, failure_reason (null on a pass) 15
safety.evaluation_completed Safety evaluation finished (Llama Guard / ShieldGemma run). passed, safe_ratio, total_count, evaluation_completed, scored_unsafe_count, unscored_count, safety_score, categories 15
judge.evaluation_completed LLM-as-judge scoring finished. passed, average_score, failure_reason (null on a pass) 15
evaluation.loss_gate_completed Loss/eval-loss auto-revert gate decided (pass or fail) against the configured thresholds. eval_loss carries three distinct states an auditor must be able to tell apart: a number (evaluated, this is the measurement), the string "nan" / "inf" (evaluated, the model diverged — a string because JSON has no non-finite number literal), or null (no measurement exists; the gate failed for a reason that is not about the model, such as an unusable configured ceiling). max_acceptable_loss and baseline_loss use the same number-or-string convention. passed, eval_loss, max_acceptable_loss, baseline_loss 15
pipeline.completed End-to-end CLI run (training + evaluation + export) returned exit code 0. Also emitted by the multi-stage pipeline orchestrator (_finalise_pipeline) with a structurally different payload — a parser must not assume success is always present. single-stage: success, metrics_summary; multi-stage orchestrator: pipeline_run_id, final_status, stopped_at (no success key) 12
pipeline.failed Pipeline aborted with an error before completion. error 12
pipeline.started Multi-stage pipeline orchestrator began a fresh run (not a --resume-from). pipeline_run_id, config_hash, stage_count, stage_names 12
pipeline.force_resume --resume-from proceeded past a stored config-hash mismatch because --force-resume was set. pipeline_run_id, old_config_hash, new_config_hash 12
pipeline.stage_started A pipeline stage began executing (after stage-config merge + validation). pipeline_run_id, stage_name, stage_index, input_model, input_source 12
pipeline.stage_completed A pipeline stage finished — gate_decision=passed on success, failed on a non-revert failure. pipeline_run_id, stage_name, gate_decision, metrics (success only), auto_revert_triggered (failure path only) 12
pipeline.resume_refused --resume-from was refused because a prior stage is still awaiting human approval (Article 14 gate not yet cleared). pipeline_run_id, requested_stage, blocking_stage, blocking_status 12, 14
pipeline.stage_gated A stage halted at the Article 14 human-approval gate (exit 4); the pipeline stops pending operator action. pipeline_run_id, stage_name, gate_decision (approval_pending), staging_path 12, 14
pipeline.stage_reverted A stage's post-train gate auto-reverted the model (auto_revert_triggered=true); the chain stops. pipeline_run_id, stage_name, gate_decision (failed), auto_revert_triggered 12, 15

Article 14 — Human Oversight

Event When emitted Payload Article
human_approval.required A gate marked requires_human_approval: true paused the pipeline awaiting an operator decision. gate, reason, metrics, staging_path, run_id, config_hash 14
human_approval.granted Operator approved the paused gate via forgelm approve. gate, approver, comment, run_id, promote_strategy 14
human_approval.rejected Operator rejected the paused gate via forgelm reject. gate, approver, comment, run_id, staging_path 14

Article 15 — Model Integrity (auto-revert + safety)

Event When emitted Payload Article
model.reverted Auto-revert fired after a quality regression: the saved model directory is deleted. Emitted before the deletion so the record survives a failed rmtree. (Phase 8 — webhook-coupled.) reason (the gate that fired: benchmark / safety / judge / etc.), detail (the human-readable failure reason from the gate) 15
model.revert_failed Auto-revert tried to delete the saved model directory and the delete failed: the artifacts are still on disk. Retracts the preceding model.reverted, which is written before the delete and cannot be edited. Paired with a training.failure webhook rather than training.reverted. reason (the gate that fired), path (the directory that could not be removed), error_class 15
model.integrity_verified Model artifacts, including the model card and deployer instructions, were hashed and model_integrity.json was written. Records generation, not independent verification. artifacts (count of files hashed) 15
audit.classifier_load_failed Safety classifier (e.g., Llama Guard) failed to load. The run still records passed=False. classifier, reason 15

Article 11 + Annex IV — Compliance artefacts

Event When emitted Payload Article
compliance.governance_exported Article 10 data governance report written to disk. output_path, dataset_count 10
compliance.governance_section_missing Governance report exported but the Article 10 data-quality section was absent (no data_audit_report.json). section, expected_path 10
compliance.governance_failed Governance report generation aborted (e.g., schema mismatch). reason 10
compliance.artifacts_exported Annex IV technical documentation bundle (manifest, model card, audit zip). output_dir, files, governance_ok 11, Annex IV
compliance.artifacts_export_failed Annex IV / Article 11 manifest export failed or was torn (disk full, SIGKILL, serialization error). reason 11, Annex IV

Article 17 — GDPR Right-to-Erasure (Phase 21 — forgelm purge)

Event When emitted Payload Article
data.erasure_requested First step of any forgelm purge --row-id / --run-id invocation, BEFORE any deletion. --check-policy is read-only and emits no events. target_kind ∈ {row, staging, artefacts}, target_id (hashed in row mode), salt_source (row mode), corpus_path (row), output_dir (run), justification, dry_run 17
data.erasure_completed Successful deletion finished. All requested fields + bytes_freed, files_modified, pre_erasure_line_number (row mode), match_count (row mode) 17
data.erasure_failed Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity. All requested fields + error_class, error_message 17
data.erasure_warning_memorisation Row erasure × a final_model/ directory exists directly under --output-dir (literal name; --output-dir defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. All completed fields + affected_run_ids 17
data.erasure_warning_synthetic_data_present Row erasure × synthetic_data*.jsonl exists in output_dir. All completed fields + synthetic_files 17
data.erasure_warning_external_copies Loaded config has a non-empty webhook block; downstream consumers may have received notices. All completed fields + webhook_targets (redacted URLs) 17

Article 15 — GDPR Right-of-Access (Phase 38 — forgelm reverse-pii)

Event When emitted Payload Article
data.access_request_query Every forgelm reverse-pii invocation, after the scan completes (or after a mid-scan I/O failure — with error_*). query_hash (salted SHA-256 of raw identifier — never raw; reuses purge's per-output-dir salt), identifier_type ∈ {literal, email, phone, tr_id, us_ssn, iban, credit_card, custom}, scan_mode ∈ {plaintext, hash}, salt_source ∈ {plaintext, per_dir, env_var}, files_scanned (paths), match_count, optional error_class/error_message GDPR Art. 15

Air-gap pre-cache (Phase 35 — forgelm cache-models / cache-tasks)

Event When emitted Payload Article
cache.populate_models_requested forgelm cache-models invocation begins. models, cache_dir, safety_classifier 12
cache.populate_models_completed Every model downloaded successfully. All requested fields + total_size_bytes, count 12
cache.populate_models_failed One or more model downloads failed (transport, disk-full, HF auth). All requested fields + models_completed, error_class, error_message 12
cache.populate_tasks_requested forgelm cache-tasks invocation begins. tasks, cache_dir 12
cache.populate_tasks_completed Every lm-eval task dataset prepared successfully. All requested fields + count 12
cache.populate_tasks_failed Unknown task name, no runnable task enumerated, OR no task staged at all. All requested fields + tasks_completed, error_class, error_message (enumeration failure) or tasks_cached, tasks_failed, tasks_unavailable (nothing staged) 12
cache.populate_tasks_partial Some tasks staged, some did not — the on-disk cache is incomplete. All requested fields + tasks_cached, tasks_failed, tasks_unavailable 12

CLI / migration

Reserved namespace — cli.* is held for future CLI/migration events, but no code in forgelm/ currently emits one. No rows populate this section yet. Nothing enforces the reservation: since the catalog guard no longer whitelists namespaces, a cli.* event would be reconciled exactly like any other.

Audit-system events (meta)

Event When emitted Payload Article
audit.classifier_load_failed (See Article 15 row above.) classifier, reason 15

Adding a new event

  1. Pick a dotted, lower-snake-case name. Prefer an existing namespace (pipeline.*, training.*, evaluation.*, benchmark.*, safety.*, judge.*, model.*, human_approval.*, compliance.*, data.*, cache.*, audit.*, approval.*) but a new namespace is fine — tools/check_audit_event_catalog.py matches any dotted name in an emission context and does not need to be taught about it. It used to keep a hardcoded namespace list, which silently hid evaluation.* from both sides of its reconciliation; that list has been deleted.
  2. Add a row to the table above, including the payload keys and the Article it supports.
  3. Mirror the row to audit_event_catalog-tr.md.
  4. Emit via AuditLogger.log_event(event, **payload). Never call json.dump directly into audit_log.jsonl; the hash chain depends on the canonical writer.

Logs this catalog does not cover

Two other JSONL logs in the tree look like audit trails and are not part of the Article 12 chain. They are recorded here so nobody has to rediscover them.

1. quickstart_audit.jsonl. forgelm quickstart writes <config-dir>/quickstart_audit.jsonl containing exactly one entry, quickstart.model_selection, from a single call site in forgelm/quickstart.py. It is a convenience log, deliberately: unchained (no _hmac, no previous-entry hash), carrying no run context and no resolved model revision, with best-effort writes that are logged and swallowed on failure. That is the right weight for what it records — which template and VRAM figure produced which model choice, before any training run exists to attach it to. A second hash-chained trail rooted in nothing would be weaker compliance evidence than one honest chain plus a clearly-labelled convenience log, so it should not be "upgraded". The Article 12 artefact remains compliance.AuditLogger's audit_log.jsonl, described above.

Two consequences for anyone adding an event there:

  • The catalog guard cannot see that file. tools/check_audit_event_catalog.py misses it because the key is event_type rather than the event its emission regex matches. (It previously missed it for a second, independent reason — quickstart was not in a hardcoded _EVENT_NAMESPACES list. That list is gone; the guard no longer whitelists namespaces, so the event_type key is now the only thing hiding this file.) It reports green having never examined quickstart.py, and its success line now says so explicitly. Do not treat a passing catalog guard as coverage here.
  • What does hold the line is a test. tests/test_quickstart_compat.py::TestAuditLog asserts exactly one event with exactly that event_type. A new event fails there — which is the intended prompt to stop and decide whether it belongs in the real chain instead.

2. safety_trend.jsonl. _append_trend_entry in forgelm/safety/_results.py appends one line per safety evaluation to <output_dir>/safety_trend.jsonl: timestamp, safety_score, safe_ratio, passed, plus scored_unsafe_count, unscored_count and evaluation_completed when the caller supplies the attribution (omitted for library callers still using the pre-existing four-argument signature). Those three carry the unscored decomposition across runs, which is the only place a degrading classifier shows up as a trend at all — a run-over-run slide in safe_ratio driven entirely by a rising unscored_count otherwise reads as a model getting less safe. It is append-only and timestamped and records a gate decision, so it reads like an audit trail at a glance. It is not one: unchained, no run_id, no operator, no HMAC, and written best-effort (OSError / TypeError / ValueError are logged and swallowed so a trend-write failure cannot abort a safety pass that already concluded).

It is a cross-run convenience view, not evidence. Everything in it that matters to Article 12 is already in the chain: the catalogued safety.evaluation_completed event carries passed, safe_ratio, safety_score, total_count, the scored/unscored split (evaluation_completed, scored_unsafe_count, unscored_count), and categories for the same evaluation, inside the hash chain with full run context. An auditor should read audit_log.jsonl; safety_trend.jsonl is for plotting a score over time.

The catalog guard cannot see this file either — it has no event key at all, so no emission-context regex will ever match it. Nothing currently pins its shape the way test_quickstart_compat.py pins quickstart's; if a future change makes this file load-bearing for compliance, the correct move is to emit a catalogued event rather than to harden the trend log.

Tamper-evidence summary

Mechanism Defends against Always on?
SHA-256 hash chain Single-line edits, deletions, reorderings. Yes.
Genesis manifest sidecar Whole-log truncation back to "genesis". Yes (written once on first event).
flock(LOCK_EX) Interleaved writes from concurrent trainers sharing the directory. Yes (Unix); no-op on Windows.
flush + fsync Power-cut / kernel-panic loss between buffer write and chain advance. Yes.
HMAC-SHA-256 per line Forged re-signing after log rewrite. Only when FORGELM_AUDIT_SECRET is set.

Webhook events

Webhook payloads (Slack / Teams / generic HTTP) are a separate vocabulary scoped to operator notifications, not the regulatory record. Webhook events are not appended to audit_log.jsonl; they ride the side-channel notification bus.

Canonical reference: webhook_schema.md is the exhaustive receiver-facing contract — full payload shapes, types, stability guarantees, the outbound extra-field allowlist, and the redaction rules. The table below is scoped to the webhook ↔ audit-log correlation an auditor needs; a receiver author should start there instead. The contributor-facing rules for adding an event are in logging-observability.md.

These eight events are the only events that webhook receivers should expect from WebhookNotifier: five single-stage lifecycle events plus the three-event pipeline.* family the multi-stage orchestrator emits alongside them. Each one mirrors a corresponding audit-log event so a downstream operator can correlate webhook ping → audit entry by run_name + timestamp. Implementation: forgelm/webhook.py.

Webhook event Audit-log mirror Trigger Gated by Required payload fields
training.start training.started train() entered, before model load. webhook.notify_on_start run_name, status="started"
training.success pipeline.completed Run completed without revert or pending approval. With evaluation.auto_revert: true every gate that produced a verdict passed (a safety evaluation with no verdict, safety.evaluation_completed: false, keeps the model and also ends here); with the default auto_revert: false it also fires when a gate failed but was only recorded (model still promoted). webhook.notify_on_success run_name, status="succeeded", metrics
training.failure pipeline.failed Training itself raised (OOM, dataset error, unhandled exception). webhook.notify_on_failure run_name, status="failed", reason (masked, ≤2048 chars)
training.reverted model.reverted A post-training gate (evaluation, safety, judge, benchmark) rejected the run and _revert_model deleted the adapters. webhook.notify_on_failure run_name, status="reverted", reason (masked, ≤2048 chars)
approval.required human_approval.required Run succeeded, evaluation.require_human_approval=true, model staged for review (EU AI Act Art. 14). webhook.notify_on_success run_name, status="awaiting_approval", model_path
pipeline.started pipeline.started A multi-stage pipeline run begins, before any stage executes. webhook.notify_on_start run_name, status="started", stage_count
pipeline.completed pipeline.completed A multi-stage pipeline run reaches its terminal state. Shares the audit event's name (a known wire/audit collision; correlate on the payload field-set). webhook.notify_on_success / webhook.notify_on_failure run_name, status, final_status, stopped_at
pipeline.stage_reverted pipeline.stage_reverted A pipeline stage auto-reverts, before downstream stages are skipped. webhook.notify_on_failure run_name, status="reverted", stage_name, reason (masked, ≤2048 chars)

Why two of these split lifecycle states

  • training.failure vs training.reverted — dashboards need to tell "the trainer crashed" apart from "the trainer succeeded, but the quality / safety / judge gate said no". Both are operationally actionable, but they require different runbooks. Faz 8 introduced notify_reverted precisely so a Slack channel can colour-code the two cases differently (#ff0000 vs #ff9900).
  • approval.required — fires after the run succeeded but before the operator has approved deploy. It is not a failure; it is a pause. Receivers that auto-page on training.failure should not page on approval.required.

Payload schema

Every webhook event ships the same envelope:

{
  "event": "training.start | training.success | training.failure | training.reverted | approval.required | pipeline.started | pipeline.completed | pipeline.stage_reverted",
  "run_name": "<string>",
  "status": "started | succeeded | failed | reverted | awaiting_approval | completed | stopped_at_stage",
  "metrics": {"<name>": <number>, ...},
  "reason": "<masked string or null>",
  "model_path": "<filesystem path or null>",
  "attachments": [{"title": "...", "text": "...", "color": "..."}]
}

metrics, reason, and model_path are always present in the schema; only populated for the events that need them. attachments is the Slack-compatible block — other receivers may ignore it.

Security guarantees

  1. Every free-text field is masked, not just reason. Masking is applied once to the fully assembled payload immediately before serialization, so run_name, reason, model_path, every event-specific string field, and the attachment title / text all pass through forgelm.data_audit.mask_secrets. AWS / GitHub / Slack / OpenAI / Google / JWT / private-key blocks / Azure storage strings do not leave the process. event, status, and the attachment color are exempt and byte-exact — closed sets of code literals receivers route on. If data_audit cannot be imported, every free-text field is replaced with "[REDACTED — secrets masker unavailable]" rather than shipped raw.
  2. Reasons are truncated to 2048 chars. Stack traces longer than that are clipped with "… (truncated)".
  3. No model weights. approval.required carries the staging filesystem path only. Weights stay on disk; the operator already controls that directory.
  4. No webhook URL leakage. URLs are masked in logs to scheme://host (userinfo, path, and query stripped entirely — see _mask_netloc in forgelm/_http.py) and the response body is suppressed on non-2xx.
  5. SSRF guard. Private / loopback / link-local destinations are refused unless webhook.allow_private_destinations=true.

Retention guidance

Webhook payloads are transient. They are not the audit record. Receivers that need long-term history should snapshot the audit JSONL (<output_dir>/audit_log.jsonl) rather than archiving webhook traffic, because the audit log is the append-only hash-chained record and the webhook stream is best-effort.