Skip to content

Check semantic consistency between config investigation hits and summaries #94

Description

@stacknil

Context

The config-change investigation demo publishes schema-valid hit and summary artifacts that reviewers read together. Individual schema validation does not by itself establish that shared identity, severity, evidence counts, and bounded-correlation claims remain semantically consistent across those two surfaces.

Review the current default-branch artifacts and contracts:

  • demos/config-change-investigation-demo/artifacts/investigation_hits.json
  • demos/config-change-investigation-demo/artifacts/investigation_summary.json
  • schemas/config_investigation_hits.schema.json
  • schemas/investigation_summary.schema.json
  • docs/evidence-pipeline-contract.md
  • docs/vocabulary.md

This is distinct from #76, which traced one summary record to its governing schema. This issue asks for a relational, cross-artifact check without assuming what the conclusion should be.

Diagnostic question

For each investigation represented in both artifacts, are shared identifiers, target context, severity, triggering change, evidence counts, and provenance/correlation language mutually consistent? If a difference exists, is it intentional semantics, a documentation ambiguity, stale generated evidence, or a defect?

Evidence required

  • Default-branch commit, environment, and exact validation/inspection commands.
  • A per-investigation matrix mapping the relevant fields in investigation_hits.json to the corresponding summary fields and governing schema/documentation clauses.
  • Separate schema-validity results for both artifacts.
  • For every apparent difference, the smallest evidence-backed classification; do not infer an incident verdict from correlation alone.

Please post the matrix and classification before opening a PR.

Change threshold

A change is justified only for a concrete cross-artifact contradiction, stale generated artifact, or missing public contract. Keep any follow-up scoped to the affected generator, fixture, schema test, or wording; do not redesign both schemas speculatively.

Behavior that must remain unchanged

  • Existing public schema/version routing and deterministic generated artifacts.
  • Bounded-correlation language and the separation between evidence, finding, and investigation.
  • Demo scope and reviewer-facing artifact names.

Boundaries

  • Use checked-in sanitized artifacts only; no live telemetry or external incident data.
  • Do not add a new demo, SIEM/dashboard integration, incident verdict, or production-readiness claim.
  • Do not update generated artifacts or goldens until the semantic change is independently justified.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededtestingTest coverage and edge-case behavior

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions