Repository navigation
fix(meta): fit four schemas to the rows DFE lands - #60
Merged
Merged
Conversation
End-to-end checks on 2026-10-04 found four schemas that cannot hold the rows DFE sends them. Each gets a new version entry. The published ones are untouched.
- meta/m365/dlp 2.0.0. The fetcher's dlp unit is the Management Activity API DLP.All feed (dfe-fetcher profiles/m365.yaml), but 1.0.x described Microsoft Graph alerts, so none of its 8 source paths could match. 2.0.0 is the OMAP record: event_id, operation, workload and user_id from the fetcher's own fixture (source_m365.rs record(), landed as m365.dlp), and creation_time, record_type, object_id, policy_details, sensitive_info_detection_is_included and the three metadata objects from Microsoft's published Common and DLP schemas. PolicyDetails is a collection, so it is Array(JSON).
- meta/m365/audit_log 2.0.0 drops correlation_id. It read first(Id/CorrelationId), and Id is always present, so it only ever held the event id. Microsoft's Common schema has no top-level CorrelationId (only AppAccessContext.CorrelationId), so there is nothing for it to read instead.
- meta/aws/cloudwatch_metrics 2.0.0 makes dimensions Array(JSON). The fetcher lands a list of {Name, Value} objects (source_aws.rs fixture) and ClickHouse refused it into JSON with code 117. Map(String, String) refuses the same list too (code 27), and nothing reshapes Name/Value pairs into a map.
- common-header/timeseries 1.0.2 reads _timestamp_received first, which dfe-fetcher stamps, then _timestamp_receiver, which dfe-receiver stamps, then the old timestamp_received/received_at fallbacks.
README and docs/meta-schema.md now say what additional/ holds since #55, list derived/, and stop claiming ClickHouse refuses JSON inside Nullable. 24.8 did (2f28227). 26.3.32.14 creates and fills a Nullable(JSON) column.
Signed-off-by: Derek <derek@hyperi.io>
|
Released in v0.2.9 -- https://github.com/hyperi-io/dfe-schemas/releases/tag/v0.2.9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End-to-end checks on 2026-10-04 found four schemas that cannot hold the rows DFE sends them. Each gets a new version entry. The published ones are untouched.
dlpunit is the Management Activity APIDLP.Allfeed (dfe-fetchercrates/fetcher/profiles/m365.yaml), but 1.0.x described Microsoft Graph alerts, so none of its 8 source paths could match. 2.0.0 is the OMAP record.Id,Operation,Workload,UserIdcome from the fetcher's own fixture (source_m365.rsrecord(), landed asm365.dlpindlp_and_exchange_audit_fetch_their_own_feed).CreationTime,RecordType,ObjectId,PolicyDetails,SensitiveInfoDetectionIsIncludedand the three*MetaDataobjects come from Microsoft's Common and DLP schemas: https://learn.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-schema .PolicyDetailsis a collection, so it isArray(JSON).ExceptionInfostays out: that page types it as a string in one table and a complex type in another.correlation_id. It readfirst(Id/CorrelationId)andIdis always present, so it only ever held the event id. The Common schema has no top-levelCorrelationId, onlyAppAccessContext.CorrelationId.dimensionsArray(JSON). The fetcher lands a list of{Name, Value}objects (source_aws.rsfixture) and ClickHouse refused it into JSON with code 117.Map(String, String)refuses the same list with code 27, and nothing reshapes Name/Value pairs into a map._timestamp_receivedfirst (dfe-fetcher stamps it), then_timestamp_receiver(dfe-receiver stamps it, on by default), then the old fallbacks.README and
docs/meta-schema.mdnow say whatadditional/holds since #55, listderived/, and stop claiming ClickHouse refuses JSON inside Nullable. 24.8 did (2f28227). 26.3.32.14 creates and fills aNullable(JSON)column.Proof, all against dfe-engine main 0d142c2 and ClickHouse 26.3.32.14:
validate_schemas.py: 986 schema files, 24 tables, 2 sources, 945 derived, OK. Render gate OK. hyperi-ci check (quality + 141 tests) clean.policy_details[1].PolicyName.dimensionsloads, and the same row then fails ontimestampinstead:1709424000.0is epoch seconds as a float, which DateTime64(3) refuses (code 27). That is a separate fetcher/loader defect, not fixed here.Two things this header revision does NOT change on its own: dfe-engine pins unauthored sources at timeseries 1.0.0, and dfe-loader's default json_primary extractor writes now() into
_timestamp_receivedwhatever the column comment says.Merge only, no release.