Skip to content

fix(meta): fit four schemas to the rows DFE lands - #60

Merged
catinspace-au merged 1 commit into
mainfrom
fix/schema-shape-fixes
Oct 4, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/schema-shape-fixes

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

End-to-end checks on 2026-10-04 found four schemas that cannot hold the rows DFE sends them. Each gets a new version entry. The published ones are untouched.

  • meta/m365/dlp 2.0.0. The fetcher's dlp unit is the Management Activity API DLP.All feed (dfe-fetcher crates/fetcher/profiles/m365.yaml), but 1.0.x described Microsoft Graph alerts, so none of its 8 source paths could match. 2.0.0 is the OMAP record. Id, Operation, Workload, UserId come from the fetcher's own fixture (source_m365.rs record(), landed as m365.dlp in dlp_and_exchange_audit_fetch_their_own_feed). CreationTime, RecordType, ObjectId, PolicyDetails, SensitiveInfoDetectionIsIncluded and the three *MetaData objects come from Microsoft's Common and DLP schemas: https://learn.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-schema . PolicyDetails is a collection, so it is Array(JSON). ExceptionInfo stays out: that page types it as a string in one table and a complex type in another.
  • meta/m365/audit_log 2.0.0 drops correlation_id. It read first(Id/CorrelationId) and Id is always present, so it only ever held the event id. The Common schema has no top-level CorrelationId, only AppAccessContext.CorrelationId.
  • meta/aws/cloudwatch_metrics 2.0.0 makes dimensions Array(JSON). The fetcher lands a list of {Name, Value} objects (source_aws.rs fixture) and ClickHouse refused it into JSON with code 117. Map(String, String) refuses the same list with code 27, and nothing reshapes Name/Value pairs into a map.
  • common-header/timeseries 1.0.2 reads _timestamp_received first (dfe-fetcher stamps it), then _timestamp_receiver (dfe-receiver stamps it, on by default), then the old fallbacks.

README and docs/meta-schema.md now say what additional/ holds since #55, list derived/, and stop claiming ClickHouse refuses JSON inside Nullable. 24.8 did (2f28227). 26.3.32.14 creates and fills a Nullable(JSON) column.

Proof, all against dfe-engine main 0d142c2 and ClickHouse 26.3.32.14:

  • validate_schemas.py: 986 schema files, 24 tables, 2 sources, 945 derived, OK. Render gate OK. hyperi-ci check (quality + 141 tests) clean.
  • Each new version, rendered through the engine's SchemaBuilderV2, CREATEs. The fetcher fixture rows INSERT into dlp 2.0.0 and audit_log 2.0.0, and a row shaped by Microsoft's DLP schema reads back policy_details[1].PolicyName.
  • The cloudwatch_metrics fixture row reproduces code 117 on 1.0.1. On 2.0.0 dimensions loads, and the same row then fails on timestamp instead: 1709424000.0 is epoch seconds as a float, which DateTime64(3) refuses (code 27). That is a separate fetcher/loader defect, not fixed here.

Two things this header revision does NOT change on its own: dfe-engine pins unauthored sources at timeseries 1.0.0, and dfe-loader's default json_primary extractor writes now() into _timestamp_received whatever the column comment says.

Merge only, no release.

End-to-end checks on 2026-10-04 found four schemas that cannot hold the rows DFE sends them. Each gets a new version entry. The published ones are untouched.

- meta/m365/dlp 2.0.0. The fetcher's dlp unit is the Management Activity API DLP.All feed (dfe-fetcher profiles/m365.yaml), but 1.0.x described Microsoft Graph alerts, so none of its 8 source paths could match. 2.0.0 is the OMAP record: event_id, operation, workload and user_id from the fetcher's own fixture (source_m365.rs record(), landed as m365.dlp), and creation_time, record_type, object_id, policy_details, sensitive_info_detection_is_included and the three metadata objects from Microsoft's published Common and DLP schemas. PolicyDetails is a collection, so it is Array(JSON).
- meta/m365/audit_log 2.0.0 drops correlation_id. It read first(Id/CorrelationId), and Id is always present, so it only ever held the event id. Microsoft's Common schema has no top-level CorrelationId (only AppAccessContext.CorrelationId), so there is nothing for it to read instead.
- meta/aws/cloudwatch_metrics 2.0.0 makes dimensions Array(JSON). The fetcher lands a list of {Name, Value} objects (source_aws.rs fixture) and ClickHouse refused it into JSON with code 117. Map(String, String) refuses the same list too (code 27), and nothing reshapes Name/Value pairs into a map.
- common-header/timeseries 1.0.2 reads _timestamp_received first, which dfe-fetcher stamps, then _timestamp_receiver, which dfe-receiver stamps, then the old timestamp_received/received_at fallbacks.

README and docs/meta-schema.md now say what additional/ holds since #55, list derived/, and stop claiming ClickHouse refuses JSON inside Nullable. 24.8 did (2f28227). 26.3.32.14 creates and fills a Nullable(JSON) column.

Signed-off-by: Derek <derek@hyperi.io>
@catinspace-au
catinspace-au merged commit 6633db4 into main Oct 4, 2026
21 checks passed
@catinspace-au
catinspace-au deleted the fix/schema-shape-fixes branch October 4, 2026 08:06
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant