Skip to content

enhancement: establish evidence and confidence contracts for incident summaries #20

Description

@CoreyLeath-code

Audit finding

SentinelAI documents LLM-powered incident intelligence and an LLM Guard service. Repository search returned no committed reasoning or confidence_score implementation path, so incident-summary evidence and confidence remain an explicit gap rather than an implied capability.

Evidence inspected

  • README.md: describes drift/anomaly monitoring, LLM-powered incident intelligence, and a rule-based summary fallback.

  • tests/ and CI configuration are present.

  • Searches for reasoning and confidence_score returned no code paths.

Proposed enhancement

At the existing incident-summary boundary, add optional user-facing rationale derived only from supplied telemetry, validation state, execution mode, and source identifiers when available. Do not expose chain-of-thought, fabricate sources, or present synthetic benchmark accuracy as incident-quality evidence.

Acceptance criteria

  • Preserve the current API with a backward-compatible metadata extension.

  • Validate required telemetry fields and return insufficient evidence when support is incomplete.

  • Add focused normal, fallback, malformed, and insufficient-evidence tests.

  • Add non-sensitive structured validation and latency logging.

  • Document evidence source and limitations without publishing unreproducible metrics.

NEEDS HUMAN DECISION

Choose the operational definition and calibration method before a numeric incident-summary confidence score is exposed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions