Skip to content

fix: point the field maps at real ECS columns - #63

Merged
catinspace-au merged 1 commit into
mainfrom
fix/sigma-ecs-map
Oct 6, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/sigma-ecs-map

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

The four default field maps pointed at a short column vocabulary no meta schema defines. ClickHouse refuses a view when any one target is missing (code 47, UNKNOWN_IDENTIFIER), so every view built from these maps failed. Image went to the basename, so a full-path match could never fire.

  • Sigma map is now the logsource-independent set dfe-schemagen (dfe-transform-elastic-dev, crates/dfe-schemagen/sigma-ecs-map.yaml) derives from pysigma-backend-elasticsearch 2.1.1, the pysigma <2 line. 121 entries, identical key for key.
  • Every target is the ECS path with dots as underscores, which is how meta/elastic/ecs.yaml names all 1018 of its columns.
  • Image, ParentImage, TargetFilename and ImageLoaded now resolve to full paths. EventID goes to event_code, TargetObject to registry_path.
  • ECS, CIM and OCSF maps keep their keys and point at the same ECS columns. OCSF follows the pairing in Elastic's amazon_security_lake pipeline.
  • Dropped from the Sigma default: CommandLine, User, LogonType, RegistryKey and RegistryValue. pySigma maps CommandLine and User to a different ECS field depending on the rule's logsource, so they need a source map. LogonType has no ECS field. RegistryKey and RegistryValue are not Sigma taxonomy fields and appear in no SigmaHQ rule.
  • tests/test_field_maps.py checks every target of every map against timeseries + meta/elastic/ecs, and pins the Sigma fields whose wrong neighbour column also exists. It fails on main (10 failed) and passes here.

dfe-engine's tests/unit/test_fieldmap/test_registry.py pins the old Sigma map, so it needs editing in the same rollout that bumps dfe-schemas.

Closes #46

The Sigma, ECS, CIM and OCSF default maps targeted a short column vocabulary no meta schema defines, so ClickHouse refuses every view built from them (code 47, UNKNOWN_IDENTIFIER). Image went to the basename in process_name, so a full-path comparison could never match.

The Sigma map is now the logsource-independent set dfe-schemagen derives from the pysigma-backend-elasticsearch 2.1.1 ECS pipelines: 121 entries, every target the ECS path with dots as underscores. Image, ParentImage, TargetFilename and ImageLoaded resolve to full paths, EventID to event_code, TargetObject to registry_path. The ECS, CIM and OCSF maps keep their keys and point at the same ECS columns, OCSF following the pairing in Elastic's amazon_security_lake pipeline.

tests/test_field_maps.py checks every target of every map against timeseries + meta/elastic/ecs, and pins the Sigma fields whose wrong neighbour column also exists.
@catinspace-au
catinspace-au merged commit c7428dc into main Oct 6, 2026
21 checks passed
@catinspace-au
catinspace-au deleted the fix/sigma-ecs-map branch October 6, 2026 03:31
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Sigma field map targets column names no meta schema defines, and mis-maps Image to a basename

1 participant