Repository navigation
Add policy testing, NetworkPolicy fence, and drift audit - #5
Draft
kylecrawshaw wants to merge 2 commits into
Draft
kylecrawshaw wants to merge 2 commits into
kylecrawshaw wants to merge 2 commits into
Conversation
Two gaps in the Kyverno path, both about the difference between "the policy ran" and "the deployment is in the state you meant". The side door. Rewriting the Service moved Service traffic behind the gate and left the pod IP answering on the app's own port to anything in the cluster. The example README described the fence and told you to write it yourself, because its failure mode is CNI-dependent. That reasoning stands, so the new generate-anteroom-networkpolicy policy is off by default — but it is a value now rather than a snippet, with the two ways it fails silently spelled out: kindnetd does not implement NetworkPolicy at all (it applies cleanly and fences nothing), and where a CNI does enforce it, the app container's probes target precisely the port being fenced. networkPolicy.extraIngress is the node-CIDR escape hatch. Ingress-only on purpose: naming Egress would take DNS out from under every gated pod. Removal. A mutation is stored, not overlaid, so removing a Service's proxied label left it targeting the port name "anteroom" with nothing able to reconstruct the original — the trap the README warned about under "remove the Service label first". Kyverno can undo it, but only if the original was written down first, so the rewrite now records spec.ports verbatim into an annotation and a third rule restores them on the update that removes the label. The array is stored whole because a port with no explicit targetPort has none for a reason, and a port-to-value map cannot say "absent". The rest of the cleanup story turned out to be Kyverno's already, and is now documented rather than left to be discovered: a synchronized generate rule deletes its downstream when the trigger stops matching, so dropping a namespace label removes the ConfigMap, the NetworkPolicy and the cloned Secret. Uninstall is asymmetric — generated-from-data downstreams are deleted in every namespace, clones are retained — hence orphanDownstreamOnPolicyDelete. A running pod's sidecar is the one thing no policy can remove, since container lists are immutable after admission; that needs a rollout, and audit-anteroom-drift reports the inverse state (labeled, no gate) which is otherwise indistinguishable from the app being down. Fixed while here: the rewrite rule errored on a Service with no spec.ports (Kyverno raises on the missing field rather than yielding an empty list), and a mutate error is an admission denial under the default failurePolicy — so labeling an ExternalName Service by mistake got it refused instead of skipped. Verified with charts/kyverno-policies/tests/run.sh, which evaluates each policy against fixtures using `kyverno apply` and diffs the result against committed expectations — including the cases that motivated the design (an implicit targetPort surviving the round trip, the port-less Service, the drift rule applied alone so it sees an unmutated pod the way a background scan does). Wired into a new CI job, since nothing checked the charts before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018788NnHJYEyyhhuuV2BuST
…ence
main landed `admin.networkPolicy.enabled` while this branch was open, and it
generates very nearly the policy this branch added: same trigger, same
podSelector, same `policyTypes: [Ingress]`, 8080 from anywhere, an
extraIngress escape hatch, and the same CNI and kubelet-probe caveats. Its
switch is named for the admin listener because fencing an exposed /metrics is
what it was built for, but it carries its own guard, so it already closes the
side door with `admin.expose` left off.
Two policies selecting the same pods would have been additive in Kubernetes
and confusing everywhere else — two objects, two names, two docs sections, and
union semantics for a reader to work out. So this drops the duplicate
(`policies.networkPolicy`, the top-level `networkPolicy:` block, and
clusterpolicy-generate-networkpolicy.yaml) and keeps main's, which is the
author's shipped design.
Nothing is lost by that. What this branch had and main's rule did not, it now
has:
- The fence goldens in tests/ were regenerated against main's rule, so the
tests now cover `generate-admin-networkpolicy` — including the collector
ingress and extraIngress appending — rather than a policy that no longer
exists.
- `orphanDownstreamOnPolicyDelete` now applies to the NetworkPolicy too,
not just the ConfigMap. Both are generated from data, so both are deleted
when the policy is, and neither should be surprising at `helm uninstall`.
- main's rule hardcoded `port: 8080` a second time; it now uses the
gatePort constant this branch added, so the fence cannot drift from the
containerPort the injection policy sets.
Kept from main verbatim: rbac.yaml (its `admin.networkPolicy.enabled` guard is
the correct condition now), the admin container port on the injected sidecar,
and the "Scraping the fleet" documentation. Chart version goes to 0.3.0 since
main already shipped 0.2.0.
Conflicts resolved in values.yaml, README.md.gotmpl, rbac.yaml,
clusterpolicy-inject-sidecar.yaml, and the generated README.md (regenerated
with helm-docs rather than hand-merged). No Go file diverges from main.
Verified with helm lint, the 12-case policy suite, go vet, and go test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018788NnHJYEyyhhuuV2BuST
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive offline testing for Kyverno policies, introduces two new policies (NetworkPolicy fence and drift audit), and enhances the Service rewrite policy with automatic restoration when labels are removed.
Key Changes
Testing Infrastructure
charts/kyverno-policies/tests/run.sh: A comprehensive test harness that useskyverno applyto evaluate policies offline against fixture resources, comparing output against golden fileschartsjob in.github/workflows/ci.yamlhelm-testtarget to MakefileNew Policies
clusterpolicy-generate-networkpolicy.yaml: Opt-in policy that generates a NetworkPolicy per namespace, admitting only TCP 8080 (the gate's port) to gated pods, closing the side door where the application's own port remains accessible on the pod IPclusterpolicy-audit-drift.yaml: Audit-only policy that reports pods carrying the inject label but no gate container—the invisible failure state when a pod was admitted while injection was disabledEnhanced Service Rewrite Policy
clusterpolicy-route-service.yaml: Now includes three rules instead of one:spec.portsin annotation before rewriting (precondition: incoming object is un-rewritten)proxiedlabel is removedpolicies.restoreServicevalue (default: true)Configuration & Documentation
policies.restoreService,policies.networkPolicy, andpolicies.auditDriftchart valueskyverno-policies.gatePortto keep the gate port (8080) in one placeNotable Implementation Details
portat resolution time)background: trueto scan existing pods, not just newly admitted oneshttps://claude.ai/code/session_018788NnHJYEyyhhuuV2BuST