diff --git a/CHANGELOG.md b/CHANGELOG.md index 85d8952..09075b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to tagged releases are documented here. ## Unreleased +- **Graph matrix runs and validations bind themselves to the document they loaded** (ADR-0030, + closes #132): the matrix suite entry, its validation twin, and the direct single-graph test + and validation envelopes carry `graphSha256` — the bare-hex digest of the exact bytes the run + decoded, read off the one load. Equality with `experimental_get_graph`'s own `sha256` proves + rows fetched in one call and the document fetched in another describe one revision; the member + is absent exactly when the document did not load, beside the detail or diagnostics that say + why. Additive under VERSIONING.md's MINOR rule; the evaluator's conformance claim is + unaffected and stated, in full and only, in `CONFORMANCE.md`. + - **Graph rows members are held to their exact spelling** — `encoding/json` case-folds member names, so a rows document carrying `"Cases"` or `"ID"` bound past the strict decoder and was silently read as the members it is not. The hold the pack matrix has carried since ADR-0025, diff --git a/docs/adr/0030-bind-graph-matrix-runs-and-validations-to-the-loaded-document.md b/docs/adr/0030-bind-graph-matrix-runs-and-validations-to-the-loaded-document.md new file mode 100644 index 0000000..fee12c9 --- /dev/null +++ b/docs/adr/0030-bind-graph-matrix-runs-and-validations-to-the-loaded-document.md @@ -0,0 +1,79 @@ +--- +status: accepted +date: 2026-08-24 +deciders: maintainer +--- + +# Bind graph matrix runs and validations to the loaded document + +## Context and problem statement + +A wire-only client can fetch a configured graph document (`experimental_get_graph`, +[ADR-0029](0029-serve-graphs-and-their-inventory.md), digest included) and run its matrix +(`experimental_test_graphs`, [ADR-0026](0026-run-the-declared-graph-matrix-over-mcp.md)) — two +calls, two reads, one file that may be edited between them. The matrix payload carries no digest +of the graph document its run loaded, so a client joining the two — nodes by name, edge witnesses +by coverage index — cannot prove the rows and the structure describe one revision, and a review +of exactly such a client (issue #132) found the join silently combining revision-A rows with +revision-B arrows across an edit. The client's interim mitigation is connection-epoch gating, +which bounds staleness without ever proving sameness. + +The value already exists at exactly the right moment: `graph.Load` computes the digest of the +exact bytes it decoded, the lock ([ADR-0019](0019-reviewed-set-lock.md)) already pins graph +digests per configured id, and the walk holds the loaded document in hand when it builds each +entry. Only the wire omits it. + +## Decision drivers + +- The echo rule every graph payload already follows: report what this run actually read, off the + one read, never off a second one that could name a different revision. +- The payload digest convention settled in ADR-0029's rounds: a member named for its algorithm + carries bare hex; the `sha256:`-prefixed spelling belongs to the lock and audit records. +- Absence must be honest: a document that did not load has no bytes to bind, and a digest of + nothing would be an invention beside the detail that says why. +- VERSIONING.md's MINOR rule: additive members move no `outputVersion`. + +## Considered options + +- **A. `graphSha256` on the four run payloads** — the suite entry, its validation twin, and the + direct single-graph test and validation envelopes — read off the loaded document's own digest. +- **B. Client-side binding only** (epoch gating, as the desk does today). +- **C. A digest member on the rows instead of the entry.** + +## Decision outcome + +Chosen option: **A**. Option B bounds staleness and proves nothing — it is the mitigation this +member exists to retire, not an answer. Option C repeats one fact per row on a surface whose +report budget was redesigned once already for exactly that multiplication (ADR-0026); the +document is loaded at most once per entry — a rowless entry skips before loading — and the +entry is where a per-load fact belongs. + +Settled constraints: + +1. **Member and format.** `graphSha256`, bare hex, on `GraphSuiteEntry`, `GraphValidationEntry`, + `GraphTest`, and `GraphValidation` — the digest `graph.Load` computed from the exact bytes + this run decoded, with the lock/audit `sha256:` prefix stripped at the payload boundary, + matching every other payload digest member. +2. **Present exactly when the document loaded.** An entry whose document could not be read or + loaded carries no digest, beside the detail or diagnostics that say why; a rows failure after + a successful load keeps it, because the bytes the binding names did load. The direct + envelopes exist only after their caller's load succeeded, so their member is required where + the walk entries' is `omitempty`. +3. **The binding it enables, stated for consumers:** equality with `experimental_get_graph`'s + `sha256` proves the served document and this run's results are about one revision; inequality + proves an edit happened between the calls. It is a binding of bytes, not a verdict about + either revision. +4. **Scope.** The matrix and validation surfaces, exactly. The graph *evaluate* composite + already binds differently — the audit record carries the graph digest per ADR-0018's record + design — and extending the envelope there is its own decision if a consumer ever needs it. + +### Consequences + +- Good, because the client's document/matrix join becomes provable instead of epoch-bounded, and + the desk's recorded mitigation can be retired for a real binding. +- Good, because the member costs one string per entry, charged by the existing report budget on + the budgeted matrix-test paths; the validation walk has no budget, and one string per entry + does not create the multiplication a budget exists for. +- Neutral, because human renderings are unchanged: the member exists for machine consumers doing + the join; a person reading the walk report is not comparing digests. +- Neutral, because `outputVersion` stays: additive members under the MINOR rule. diff --git a/docs/adr/README.md b/docs/adr/README.md index d1102cc..b383eec 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -126,3 +126,4 @@ authority, and following it confers no conformance status on anything. | [0027](0027-pin-the-evaluation-trace-contract.md) | Pin the evaluation trace: deterministic, complete, ordered, and still informative | accepted | | [0028](0028-declare-an-evaluation-a-rehearsal.md) | Declare an evaluation a rehearsal, so exploration never writes a decision record | accepted | | [0029](0029-serve-graphs-and-their-inventory.md) | Serve the configured graphs and their inventory over the wire, read-only | accepted | +| [0030](0030-bind-graph-matrix-runs-and-validations-to-the-loaded-document.md) | Bind graph matrix runs and validations to the loaded document | accepted | diff --git a/internal/cli/graph.go b/internal/cli/graph.go index b1e8800..2066cea 100644 --- a/internal/cli/graph.go +++ b/internal/cli/graph.go @@ -158,6 +158,7 @@ func (a *App) graphValidateCommand() *cobra.Command { GraphPath: graphPath, GraphID: document.ID, GraphVersion: document.Version, + GraphSHA256: strings.TrimPrefix(document.Digest, "sha256:"), Diagnostics: document.Semantic(loaded), } if len(output.Diagnostics) > 0 { diff --git a/internal/cli/graph_test.go b/internal/cli/graph_test.go index 54d6a35..fdcf527 100644 --- a/internal/cli/graph_test.go +++ b/internal/cli/graph_test.go @@ -1,6 +1,8 @@ package cli import ( + "crypto/sha256" + "encoding/hex" "encoding/json" "os" "path/filepath" @@ -85,6 +87,13 @@ func TestGraphValidateCommand(t *testing.T) { if !found { t.Fatalf("the cycle must be a diagnostic: %+v", validation.Diagnostics) } + // The envelope binds the exact bytes this validation decoded (ADR-0030): + // bare hex on the wire, and present even when the document is invalid — + // it loaded, and these findings are about that revision. + sum := sha256.Sum256([]byte(looped)) + if !strings.Contains(stdout, `"graphSha256":"`+hex.EncodeToString(sum[:])+`"`) { + t.Fatalf("the validation envelope binds the loaded bytes: %q", stdout) + } } func TestGraphEvaluateCommandComposesAndReportsJSON(t *testing.T) { diff --git a/internal/graph/digest_test.go b/internal/graph/digest_test.go new file mode 100644 index 0000000..3ab6b17 --- /dev/null +++ b/internal/graph/digest_test.go @@ -0,0 +1,153 @@ +package graph + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +// rawJSON marshals a payload the way the wire does, so the assertions below +// discriminate the contract — the member's spelling and presence in bytes — +// not just the Go field a renderer might never emit. +func rawJSON(t *testing.T, payload any) string { + t.Helper() + data, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + return string(data) +} + +func fixtureDigest(t *testing.T) string { + t.Helper() + data, err := os.ReadFile(filepath.Join("testdata", "project", "onboarding.graph.json")) + if err != nil { + t.Fatal(err) + } + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} + +// Every graph matrix run and validation binds itself to the exact document +// bytes it loaded (ADR-0030): the bare-hex digest a consumer holds against +// the served document's own sha256, so rows fetched in one call and structure +// fetched in another are provably about one revision. The digest is read off +// the one load, and it is absent exactly when the document did not load. +func TestGraphRunsCarryTheDocumentDigest(t *testing.T) { + want := fixtureDigest(t) + binding := `"graphSha256":"` + want + `"` + + loaded := fixtureProject(t) + tested, failure := TestProject(loaded, newEngine(t), "", Options{Command: "test"}) + if failure != nil { + t.Fatal(failure.Message) + } + if len(tested.Graphs) != 1 || !strings.Contains(rawJSON(t, tested.Graphs[0]), binding) { + t.Fatalf("the test entry binds the loaded bytes on the wire: %s", rawJSON(t, tested.Graphs[0])) + } + + validated, failure := ValidateProject(loaded, "", "validate") + if failure != nil { + t.Fatal(failure.Message) + } + if len(validated.Graphs) != 1 || !strings.Contains(rawJSON(t, validated.Graphs[0]), binding) { + t.Fatalf("the validation entry binds the loaded bytes on the wire: %s", rawJSON(t, validated.Graphs[0])) + } +} + +// The direct single-graph envelope echoes the digest of the document it was +// handed — the one load its caller performed — never a second read of the +// path. A digest planted on the document and impossible for the on-disk +// bytes proves which one the envelope reports. +func TestDirectTestEchoesTheHandedDocumentDigest(t *testing.T) { + loaded := fixtureProject(t) + graphBytes, err := os.ReadFile(filepath.Join("testdata", "project", "onboarding.graph.json")) + if err != nil { + t.Fatal(err) + } + document, loadFailure := Load(graphBytes, "onboarding.graph.json") + if loadFailure != nil { + t.Fatal(loadFailure.Message) + } + planted := strings.Repeat("ab", 32) + document.Digest = "sha256:" + planted + rowsBytes, err := os.ReadFile(filepath.Join("testdata", "project", "onboarding.rows.json")) + if err != nil { + t.Fatal(err) + } + rows, rowsFailure := LoadRows(rowsBytes, "onboarding.rows.json") + if rowsFailure != nil { + t.Fatal(rowsFailure.Message) + } + output, failure := Test(loaded, newEngine(t), document, "onboarding.graph.json", "onboarding.rows.json", rows, Options{Command: "test"}) + if failure != nil { + t.Fatal(failure.Message) + } + if !strings.Contains(rawJSON(t, output), `"graphSha256":"`+planted+`"`) { + t.Fatalf("the envelope echoes the handed document's digest, bare hex: got %q", output.GraphSHA256) + } +} + +// A rows failure after a successful document load keeps the digest: the +// document did load, those are the bytes the binding names, and the detail +// says what stopped the run. +func TestRowsFailureRetainsTheDocumentDigest(t *testing.T) { + files := map[string]string{} + for _, name := range []string{"jpack.json", "onboarding.graph.json", "sanctions-screening-0.1.0.pack.json", "vendor-onboarding-0.1.0.pack.json"} { + data, err := os.ReadFile(filepath.Join("testdata", "project", name)) + if err != nil { + t.Fatal(err) + } + files[name] = string(data) + } + files["onboarding.rows.json"] = "{not json" + loaded := writeProject(t, files) + tested, failure := TestProject(loaded, newEngine(t), "", Options{Command: "test"}) + if failure != nil { + t.Fatal(failure.Message) + } + entry := tested.Graphs[0] + if len(tested.Graphs) != 1 || entry.Status != "mismatch" || entry.Detail == "" || + !strings.Contains(rawJSON(t, entry), `"graphSha256":"`+fixtureDigest(t)+`"`) { + t.Fatalf("loaded document, failed rows: digest stays beside the detail: %s", rawJSON(t, entry)) + } +} + +// A document that did not load has no bytes to bind: the member is absent +// from the wire — not empty — beside the detail or diagnostics that say why, +// on the test walk and the validation walk both. +func TestUnloadableGraphCarriesNoDigest(t *testing.T) { + files := map[string]string{} + for _, name := range []string{"jpack.json", "onboarding.rows.json", "sanctions-screening-0.1.0.pack.json", "vendor-onboarding-0.1.0.pack.json"} { + data, err := os.ReadFile(filepath.Join("testdata", "project", name)) + if err != nil { + t.Fatal(err) + } + files[name] = string(data) + } + files["onboarding.graph.json"] = "{not json" + loaded := writeProject(t, files) + + tested, failure := TestProject(loaded, newEngine(t), "", Options{Command: "test"}) + if failure != nil { + t.Fatal(failure.Message) + } + if len(tested.Graphs) != 1 || tested.Graphs[0].Detail == "" || + strings.Contains(rawJSON(t, tested.Graphs[0]), "graphSha256") { + t.Fatalf("no load, no digest member, a detail instead: %s", rawJSON(t, tested.Graphs[0])) + } + + validated, failure := ValidateProject(loaded, "", "validate") + if failure != nil { + t.Fatal(failure.Message) + } + if len(validated.Graphs) != 1 || validated.Graphs[0].Status != "invalid" || + len(validated.Graphs[0].Diagnostics) == 0 || + strings.Contains(rawJSON(t, validated.Graphs[0]), "graphSha256") { + t.Fatalf("no load, no digest member, diagnostics instead: %s", rawJSON(t, validated.Graphs[0])) + } +} diff --git a/internal/graph/rows.go b/internal/graph/rows.go index d4b882e..46bad6d 100644 --- a/internal/graph/rows.go +++ b/internal/graph/rows.go @@ -261,6 +261,7 @@ func Test(loaded *project.Project, engine *evaluation.Engine, doc Document, grap RowsPath: rowsPath, GraphID: doc.ID, GraphVersion: doc.Version, + GraphSHA256: strings.TrimPrefix(doc.Digest, "sha256:"), Rows: make([]result.GraphTestRow, 0, len(rows.Cases)), } spent := 0 diff --git a/internal/graph/suite.go b/internal/graph/suite.go index 176974b..ccc8bef 100644 --- a/internal/graph/suite.go +++ b/internal/graph/suite.go @@ -4,6 +4,7 @@ import ( "encoding/json" "errors" "fmt" + "strings" "unicode/utf8" @@ -125,6 +126,7 @@ func testEntry(loaded *project.Project, engine *evaluation.Engine, id string, en return mismatch(detail) } report.GraphID, report.GraphVersion = document.ID, document.Version + report.GraphSHA256 = strings.TrimPrefix(document.Digest, "sha256:") rowsBytes, err := loaded.ReadGraphRows(entry, MaxRowsBytes) if err != nil { if errors.Is(err, fssecure.ErrTooLarge) { @@ -210,6 +212,7 @@ func validateEntry(loaded *project.Project, id string, entry project.Graph) resu return report } report.GraphID, report.GraphVersion = document.ID, document.Version + report.GraphSHA256 = strings.TrimPrefix(document.Digest, "sha256:") report.Diagnostics = append(report.Diagnostics, document.Semantic(loaded)...) if len(report.Diagnostics) > 0 { report.Status = "invalid" diff --git a/internal/result/graph.go b/internal/result/graph.go index 5d5be74..6b5301f 100644 --- a/internal/result/graph.go +++ b/internal/result/graph.go @@ -175,6 +175,9 @@ type GraphSchema struct { // references against the project — every node resolves through the // configuration, every evidence feed names a requirement its target pack // declares. Diagnostics carry every finding rather than the first one. +// GraphSHA256 is the bare-hex digest of the exact bytes this validation +// decoded (ADR-0030); the payload exists only after the document loaded, so +// unlike the walk entries the member is required. type GraphValidation struct { OutputVersion string `json:"outputVersion"` Tool Tool `json:"tool"` @@ -186,6 +189,7 @@ type GraphValidation struct { GraphPath string `json:"graphPath"` GraphID string `json:"graphId,omitempty"` GraphVersion string `json:"graphVersion,omitempty"` + GraphSHA256 string `json:"graphSha256"` Diagnostics []Diagnostic `json:"diagnostics"` } @@ -245,24 +249,27 @@ type GraphTestRow struct { // additive, so outputVersion is unchanged by the VERSIONING.md machine-output // rules. type GraphTest struct { - OutputVersion string `json:"outputVersion"` - Tool Tool `json:"tool"` - Command string `json:"command"` - Status string `json:"status"` - Experimental bool `json:"experimental"` - ConformanceClaimReference string `json:"conformanceClaimReference"` - Label string `json:"label"` - Kind string `json:"kind"` - FormatVersion string `json:"formatVersion"` - EvaluatorSpecVersion string `json:"evaluatorSpecVersion"` - ConfigPath string `json:"configPath"` - GraphPath string `json:"graphPath"` - RowsPath string `json:"rowsPath"` - GraphID string `json:"graphId"` - GraphVersion string `json:"graphVersion"` - Summary SuiteSummary `json:"summary"` - Rows []GraphTestRow `json:"rows"` - Coverage []MatrixProbe `json:"coverage,omitempty"` + OutputVersion string `json:"outputVersion"` + Tool Tool `json:"tool"` + Command string `json:"command"` + Status string `json:"status"` + Experimental bool `json:"experimental"` + ConformanceClaimReference string `json:"conformanceClaimReference"` + Label string `json:"label"` + Kind string `json:"kind"` + FormatVersion string `json:"formatVersion"` + EvaluatorSpecVersion string `json:"evaluatorSpecVersion"` + ConfigPath string `json:"configPath"` + GraphPath string `json:"graphPath"` + RowsPath string `json:"rowsPath"` + GraphID string `json:"graphId"` + GraphVersion string `json:"graphVersion"` + // GraphSHA256 binds this run to the exact document bytes it loaded + // (ADR-0030), bare hex per the payload convention. + GraphSHA256 string `json:"graphSha256"` + Summary SuiteSummary `json:"summary"` + Rows []GraphTestRow `json:"rows"` + Coverage []MatrixProbe `json:"coverage,omitempty"` } // GraphSuiteEntry is one configured graph's matrix run inside the project walk @@ -273,16 +280,22 @@ type GraphTest struct { // which case the identity echoes are empty rather than guessed and Rows and // Coverage are absent. type GraphSuiteEntry struct { - ID string `json:"id"` - Path string `json:"path"` - RowsPath string `json:"rowsPath,omitempty"` - GraphID string `json:"graphId,omitempty"` - GraphVersion string `json:"graphVersion,omitempty"` - Status string `json:"status"` - Summary SuiteSummary `json:"summary"` - Rows []GraphTestRow `json:"rows,omitempty"` - Coverage []MatrixProbe `json:"coverage,omitempty"` - Detail string `json:"detail,omitempty"` + ID string `json:"id"` + Path string `json:"path"` + RowsPath string `json:"rowsPath,omitempty"` + GraphID string `json:"graphId,omitempty"` + GraphVersion string `json:"graphVersion,omitempty"` + // GraphSHA256 is the bare-hex digest of the exact bytes this entry's run + // loaded (ADR-0030), read off the one load: a consumer holding the served + // document from another call binds these rows to that revision by equality + // rather than by hope. Absent exactly when the document did not load, + // beside the detail that says why. + GraphSHA256 string `json:"graphSha256,omitempty"` + Status string `json:"status"` + Summary SuiteSummary `json:"summary"` + Rows []GraphTestRow `json:"rows,omitempty"` + Coverage []MatrixProbe `json:"coverage,omitempty"` + Detail string `json:"detail,omitempty"` } // GraphSuite is the project graph-matrix walk: every configured graph's rows, @@ -314,13 +327,19 @@ type GraphSuite struct { // applies — plus the declared rows document's containment when one is // declared. Diagnostics carry every finding rather than the first one. type GraphValidationEntry struct { - ID string `json:"id"` - Path string `json:"path"` - RowsPath string `json:"rowsPath,omitempty"` - GraphID string `json:"graphId,omitempty"` - GraphVersion string `json:"graphVersion,omitempty"` - Status string `json:"status"` - Diagnostics []Diagnostic `json:"diagnostics"` + ID string `json:"id"` + Path string `json:"path"` + RowsPath string `json:"rowsPath,omitempty"` + GraphID string `json:"graphId,omitempty"` + GraphVersion string `json:"graphVersion,omitempty"` + // GraphSHA256 is the bare-hex digest of the exact bytes this walk loaded + // (ADR-0030): the member that lets a consumer holding the document from + // another call know these results are about the same revision, read off + // the one load rather than a second one. Absent exactly when the document + // did not load, beside the diagnostics that say why. + GraphSHA256 string `json:"graphSha256,omitempty"` + Status string `json:"status"` + Diagnostics []Diagnostic `json:"diagnostics"` } // GraphValidationSuite is the project graph-validation walk, on