Version-matrix golden generation: classify versions by firing-set and assert mutation coverage by sourcehawk · Pull Request #140 · sourcehawk/operator-component-framework

sourcehawk · 2026-06-01T02:08:15Z

Closes #131

What

An optional, test-only helper for verifying version-spanning resources, plus the read-only framework introspection it needs.

Framework (read-only, inert unless called):

concepts.MutationInspector (RegisteredMutations() []string, FiringSet() ([]string, error)) implemented on generic.BaseResource, every primitive Resource, and *component.Component (union across managed resources).
golden.Serialize / golden.SerializeComponent, exported wrappers over the existing serializer so generated goldens are byte-identical to hand-written ones.

pkg/testing/goldengen (new test-only package):

One declarative Config[T] drives everything: a consumer-supplied version universe, fixtures with Requires/Forbids expectations (optionally pinned to a version via For), and a Build closure producing a Unit.
Run sweeps every fixture across every version, classifies versions into behaviorally-distinct regimes by firing-set, writes one golden per regime at a representative version, emits a reviewable manifest.yaml, and asserts per-fixture gating.
AssertComplete (from the consumer's TestMain) proves every registered mutation is either required by a fixture or explicitly excluded.
LoadMatrix optionally loads the whole declaration from a matrix.yaml, with fixtures supplied inline or by specFile.

Docs and example: docs/testing.md and a runnable examples/version-matrix.

Why

Consumers verified version-gated mutations with hand-picked goldens at a few versions, which missed silent regime gaps (correct at 8.9, no-op at 8.8) and never proved a mutation fires where assumed. This derives the minimal regime coverage and asserts gating intentionally.

Design notes

The framework stays black-box about gating: it never parses constraints, only evaluates Gate.Enabled() and classifies by firing-set, which is immune to version interpolation. Representative selection takes the first supplied version in each regime, which lands on the inclusive boundary when versions are listed ascending.

Verification

make all and make build-examples are green on the integrated branch. The examples/version-matrix example exercises the whole flow end to end.

Spec, implementation plan, and orchestration state for the optional test-only goldengen helper: sweep consumer-supplied versions, classify by firing-set, generate minimal goldens, assert mutation coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…136) * feat(concepts): add MutationInspector interface (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(generic): implement MutationInspector on BaseResource (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(component): aggregate MutationInspector across managed resources (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(primitives): delegate MutationInspector on every primitive resource (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(golden): export Serialize and SerializeComponent (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(generic): gofmt comment alignment in inspect test (#132) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…oldens (#137) * feat(goldengen): Unit interface and resource/component adapters (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(goldengen): config types and validation (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(goldengen): firing-set classification into regimes (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(goldengen): Requires/Forbids gating assertions (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(goldengen): coverage manifest type (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(goldengen): Generator Run with sweep, gating, goldens, manifest (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(goldengen): satisfy staticcheck and gofmt (#133) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add LoadMatrix, which reads a YAML matrix file and returns a runnable Config. Each fixture's CR comes from an inline spec or an external specFile (exactly one), unmarshalled into a fresh typed spec. specFile paths resolve relative to the matrix file. The result is validated via Config.Validate before returning. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…139) * feat(goldengen): AssertComplete accounting assertion (#134) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): version-matrix golden generation example (#134) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add docs/testing.md for golden and goldengen (#134) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds optional, test-only tooling to derive minimal golden coverage across a version matrix by classifying versions into “gating regimes” based on the mutation firing-set, plus the read-only framework introspection and serializer exports needed to make generated goldens byte-identical to existing golden snapshots.

Changes:

Introduces concepts.MutationInspector and implements it on generic.BaseResource, all primitives, and *component.Component (union across managed resources).
Exports golden.Serialize / golden.SerializeComponent and refactors component golden comparison to reuse the shared serializer.
Adds new test-only package pkg/testing/goldengen (matrix config + sweep/gating assertions + regime classification + manifest + accounting + optional YAML loader), plus docs and a runnable examples/version-matrix.

Reviewed changes

Copilot reviewed 67 out of 67 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
README.md	Adds link to new testing documentation.
docs/testing.md	Documents `golden` snapshots and `goldengen` version-matrix generation, manifest, accounting, and YAML loader.
docs/superpowers/states/2026-06-01-version-matrix-goldens-state.md	Adds orchestration state record for the feature work.
docs/superpowers/specs/2026-06-01-version-matrix-goldens-design.md	Adds approved design spec for version-matrix goldens.
docs/superpowers/plans/2026-06-01-version-matrix-goldens-plan.md	Adds implementation plan for the feature.
pkg/component/concepts/mutation_inspector.go	Defines the new `MutationInspector` interface.
pkg/component/component.go	Aggregates mutation registration + firing-set across managed component resources.
pkg/component/component_inspect_test.go	Tests component-level union behavior and error propagation.
pkg/generic/resource_base.go	Implements mutation registration + firing-set computation on `BaseResource`.
pkg/generic/resource_base_inspect_test.go	Tests `BaseResource` mutation introspection behavior.
pkg/testing/golden/golden.go	Exports serializers and refactors component YAML comparison to reuse them.
pkg/testing/golden/serialize_test.go	Verifies exported serializer output matches existing golden comparison output.
pkg/testing/goldengen/unit.go	Introduces `Unit` abstraction and adapters for resources/components.
pkg/testing/goldengen/unit_test.go	Tests resource/component adapters and rendering.
pkg/testing/goldengen/helpers_test.go	Test helpers for building primitives/components and schemes.
pkg/testing/goldengen/config.go	Adds matrix config/fixture/expectation types and validation.
pkg/testing/goldengen/config_test.go	Tests config validation invariants.
pkg/testing/goldengen/classify.go	Implements firing-set regime classification.
pkg/testing/goldengen/classify_test.go	Tests regime classification behavior and ordering independence.
pkg/testing/goldengen/gating.go	Implements Requires/Forbids gating assertion lattice.
pkg/testing/goldengen/gating_test.go	Tests gating assertions and failure modes.
pkg/testing/goldengen/manifest.go	Adds manifest types and YAML rendering.
pkg/testing/goldengen/manifest_test.go	Tests manifest YAML contains expected content.
pkg/testing/goldengen/generator.go	Implements sweep, golden generation/comparison, manifest writing, and `AssertComplete`.
pkg/testing/goldengen/generator_test.go	End-to-end generator test writing goldens + manifest and re-verifying without update.
pkg/testing/goldengen/accounting_test.go	Tests `AssertComplete` accounting behavior.
pkg/testing/goldengen/loadmatrix.go	Adds YAML matrix loader to produce a runnable `Config[T]`.
pkg/testing/goldengen/loadmatrix_test.go	Tests inline vs specFile loading and error cases.
pkg/testing/goldengen/testdata/matrix_badfor.yaml	Loader test fixture: expectation `for` not in versions.
pkg/testing/goldengen/testdata/matrix_both.yaml	Loader test fixture: both `spec` and `specFile` set.
pkg/testing/goldengen/testdata/matrix_inline.yaml	Loader test fixture: inline CR spec.
pkg/testing/goldengen/testdata/matrix_neither.yaml	Loader test fixture: neither `spec` nor `specFile` set.
pkg/testing/goldengen/testdata/matrix_specfile.yaml	Loader test fixture: external `specFile` reference.
pkg/testing/goldengen/testdata/fixtures/cm.yaml	Loader test CR manifest referenced via `specFile`.
pkg/primitives/clusterrole/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/clusterrolebinding/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/configmap/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/cronjob/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/daemonset/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/deployment/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/hpa/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/ingress/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/job/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/networkpolicy/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/pdb/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/pod/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/pv/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/pvc/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/replicaset/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/role/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/rolebinding/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/secret/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/service/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/serviceaccount/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/statefulset/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/statefulset/resource_inspect_test.go	Tests statefulset resource satisfies `MutationInspector`.
pkg/primitives/unstructured/integration/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/unstructured/static/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/unstructured/task/resource.go	Delegates `MutationInspector` methods to underlying base resource.
pkg/primitives/unstructured/workload/resource.go	Delegates `MutationInspector` methods to underlying base resource.
examples/version-matrix/version_matrix_test.go	Runnable example exercising sweep, regime classification, goldens, manifest, and accounting.
examples/version-matrix/resources/statefulset.go	Example workload with version-gated mutations using `feature.NewVersionGate`.
examples/version-matrix/app/owner.go	Re-exports shared example CRD types for the example.
examples/version-matrix/README.md	Walkthrough of the example and how to run/update goldens.
examples/version-matrix/testdata/version_matrix/manifest.yaml	Checked-in manifest produced by the example.
examples/version-matrix/testdata/version_matrix/default/8.7.0.yaml	Checked-in golden for the pre-8.9 regime representative.
examples/version-matrix/testdata/version_matrix/default/8.9.0.yaml	Checked-in golden for the >=8.9 regime representative.

#131) Build the example test scheme from runtime.NewScheme() instead of mutating the shared global client-go scheme, avoiding cross-package leakage and test races. Unmarshal LoadMatrix fixture specs into &spec so value-type T works as well as the common pointer T. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated 4 comments.

… FiringSet, harden Validate (#131) - Remove the unused Config.Scheme field and the LoadMatrix scheme parameter. The scheme is read only by goldengen.Resource/Component, which the Build closure already passes it to, so the config-level field did nothing. - Deduplicate FiringSet by mutation name in first-occurrence order, matching RegisteredMutations, so a name registered twice no longer appears twice and cannot cause a spurious regime split. - Reject empty or duplicate Versions and empty Exclude entries in Config.Validate, with tests. - Fix a stale test comment about a non-existent third golden. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated 2 comments.

…oc (#131) Two docs/testing.md references to a Config-level scheme survived the Config.Scheme removal: the matrix-mirrors-Config line and a LoadMatrix example call. Both corrected. Also reword the AssertComplete GoDoc to lead with its intent (every registered mutation is either exercised by a fixture's Requires or explicitly excluded) and to state that Forbids does not count toward coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Tear down the spec, plan, and orchestration state now that the feature is complete and verified. The design rationale lives in epic #131 and docs/testing.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 64 out of 64 changed files in this pull request and generated 2 comments.

A mutation Name is the identifier gating and error reporting refer to, so two mutations sharing one on the same resource is ambiguous and silently masks a mis-targeted or dead mutation behind its namesake. ValidateBase now rejects a duplicate Name at build time, framework-wide, with a name-only check that evaluates no feature gates. This catches the collision at the source for every consumer, so MutationInspector.RegisteredMutations stays a clean set. Also fix a stale docs/testing.md reference (DebugLoggingMutation) and document the uniqueness rule in docs/primitives.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

sourcehawk and others added 15 commits June 1, 2026 03:10

docs: record sub-PR approval mode for version-matrix goldens (#131)

c2dcac4

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: PR #136 ready for #132 (#131)

672d0fe

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: #132 self-merged via #136, contracts locked (#131)

dab18fa

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: normalize planning artifact formatting (#131)

b97ab00

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: #133 in progress (#131)

1f79d9b

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: #133 self-merged via #137, goldengen API locked (#131)

f05b5de

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: wave 3 in progress, #134/#135 file ownership scoped (#131)

e7a30ad

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: wave 3 self-merged via #138/#139 (#131)

1d820cb

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: integration checkpoint green, all waves merged (#131)

60f372f

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: feature complete, status review (#131)

3998d1c

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 1, 2026 02:08

Copilot started reviewing on behalf of sourcehawk June 1, 2026 02:08 View session

docs: integration PR #140 opened (#131)

bea0c3e

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread pkg/testing/goldengen/loadmatrix.go

Comment thread examples/version-matrix/version_matrix_test.go

Comment thread docs/superpowers/specs/2026-06-01-version-matrix-goldens-design.md Outdated

Comment thread docs/superpowers/plans/2026-06-01-version-matrix-goldens-plan.md Outdated

sourcehawk requested a review from Copilot June 1, 2026 02:17

Copilot started reviewing on behalf of sourcehawk June 1, 2026 02:17 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread pkg/testing/goldengen/config.go

Comment thread pkg/testing/goldengen/config.go

Comment thread pkg/generic/resource_base.go

Comment thread pkg/testing/goldengen/generator_test.go Outdated

sourcehawk requested a review from Copilot June 1, 2026 09:43

Copilot started reviewing on behalf of sourcehawk June 1, 2026 09:43 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread docs/testing.md Outdated

Comment thread docs/testing.md

sourcehawk and others added 2 commits June 1, 2026 11:50

chore: remove version-matrix planning artifacts (#131)

9c89a65

Tear down the spec, plan, and orchestration state now that the feature is complete and verified. The design rationale lives in epic #131 and docs/testing.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 1, 2026 10:04

Copilot started reviewing on behalf of sourcehawk June 1, 2026 10:04 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread docs/testing.md Outdated

Comment thread pkg/component/concepts/mutation_inspector.go

sourcehawk mentioned this pull request Jun 1, 2026

Version-matrix golden generation: derive minimal goldens and assert mutation coverage #131

Closed

sourcehawk merged commit 249f04b into main Jun 1, 2026
3 checks passed

sourcehawk deleted the feature/version-matrix-goldens branch June 1, 2026 11:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Version-matrix golden generation: classify versions by firing-set and assert mutation coverage#140

Version-matrix golden generation: classify versions by firing-set and assert mutation coverage#140
sourcehawk merged 21 commits into
mainfrom
feature/version-matrix-goldens

sourcehawk commented Jun 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sourcehawk commented Jun 1, 2026

What

Why

Design notes

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants