Repository navigation
fix: point the field maps at real ECS columns - #63
Merged
Merged
Conversation
The Sigma, ECS, CIM and OCSF default maps targeted a short column vocabulary no meta schema defines, so ClickHouse refuses every view built from them (code 47, UNKNOWN_IDENTIFIER). Image went to the basename in process_name, so a full-path comparison could never match. The Sigma map is now the logsource-independent set dfe-schemagen derives from the pysigma-backend-elasticsearch 2.1.1 ECS pipelines: 121 entries, every target the ECS path with dots as underscores. Image, ParentImage, TargetFilename and ImageLoaded resolve to full paths, EventID to event_code, TargetObject to registry_path. The ECS, CIM and OCSF maps keep their keys and point at the same ECS columns, OCSF following the pairing in Elastic's amazon_security_lake pipeline. tests/test_field_maps.py checks every target of every map against timeseries + meta/elastic/ecs, and pins the Sigma fields whose wrong neighbour column also exists.
|
Released in v0.2.9 -- https://github.com/hyperi-io/dfe-schemas/releases/tag/v0.2.9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four default field maps pointed at a short column vocabulary no meta schema defines. ClickHouse refuses a view when any one target is missing (code 47, UNKNOWN_IDENTIFIER), so every view built from these maps failed. Image went to the basename, so a full-path match could never fire.
dfe-engine's tests/unit/test_fieldmap/test_registry.py pins the old Sigma map, so it needs editing in the same rollout that bumps dfe-schemas.
Closes #46