Repository navigation
Add DCM Documentation Release Validation Plan - #27
Merged
Merged
Conversation
Signed-off-by: Thomas Stetson <tstetson@redhat.com>
testetson22
requested review from
LinskId,
NoamNakash,
gpb88,
jordigilh and
tkiss28
October 19, 2026 16:21
Code Review by Qodo
1. Provider checks fail on current docs
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 9 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="test-plans/documentation-release-validation.md" line_range="273-274" />
<code_context>
+
+1. Follow the UI page to locate the documented route or Backstage path.
+2. Confirm the page loads and exposes the documented DCM navigation.
+3. Introduce a disposable service failure and follow the troubleshooting
+ commands for logs, status, and restart.
+4. Restore the service and verify the health check recovers.
+
</code_context>
<issue_to_address>
**Shared services stay disrupted**
When DOC-010 runs against a shared stack, or execution stops after introducing the failure but before restoring the service, the troubleshooting steps take a shared service down, interrupting other users; if execution stops before step 4, the service remains failed and later checks also run against the disrupted stack.
Run this test on an isolated stack and ensure the service is restored even when execution is interrupted.
Also at `test-plans/documentation-release-validation.md:275`.
</issue_to_address>
### Comment 2
<location path="test-plans/documentation-release-validation.md" line_range="197" />
<code_context>
+**Steps:**
+
+1. Follow `Create Placement Policy` using the documented policy file.
+2. Create, list, get, update, and delete a disposable policy.
+3. Exercise any documented filter, ordering, enabled-state, or merge-patch
+ examples.
</code_context>
<issue_to_address>
**Global policy redirects other placements**
When DOC-006 is run against a control plane shared with other users, the documented policy is `GLOBAL` and always selects `kubevirt-service-provider`, so creating and exercising it redirects other users’ instance placements on the same control plane during validation.
Require an isolated disposable control plane for DOC-006, or use a policy fixture scoped so it cannot affect other users’ placements.
</issue_to_address>
### Comment 3
<location path="test-plans/documentation-release-validation.md" line_range="183-184" />
<code_context>
+2. Create the item using the documented CLI command and optional ID.
+3. Retrieve it in table, YAML, and JSON output.
+4. Verify the item through the catalog-item API or control-plane state.
+5. Repeat the corresponding User Guide list, filter, pagination, and delete
+ examples where the release supports them.
+
+**Expected:** The YAML schema is accepted, the created resource contains the
</code_context>
<issue_to_address>
**Dependent checks lose prerequisites**
When the plan is followed in order and the documented delete operations are supported, dOC-005 can delete the catalog item before DOC-007 creates an instance from it, and DOC-006 deletes the policy before its placement check; DOC-009 also requires get checks without reseeding these records. The dependent checks then fail on missing resources or IDs.
Reorder the checks so dependent workflows run before deletes, or recreate the required resources before each dependent check.
Also at `test-plans/documentation-release-validation.md:197-201`, `test-plans/documentation-release-validation.md:213`, `test-plans/documentation-release-validation.md:222`, `test-plans/documentation-release-validation.md:255`.
</issue_to_address>
### Comment 4
<location path="test-plans/documentation-release-validation.md" line_range="83" />
<code_context>
+ authentication flow; never record secret values in results.
+
+Live tests must use disposable IDs and clean up created policies, catalog items,
+instances, providers, and containers after execution.
+
+## Execution Order
</code_context>
<issue_to_address>
**Registered agents remain after validation**
When an agent registration succeeds during DOC-008 or DOC-012, the cleanup requirement covers providers but not registered agents. A successful registration therefore leaves agent records in the control plane; later runs can encounter stale agents or route placement checks to an earlier run's agent.
Include registered agents in cleanup and verify that each run removes only the agent it created.
Also at `test-plans/documentation-release-validation.md:236-238`, `test-plans/documentation-release-validation.md:311-314`.
</issue_to_address>
### Comment 5
<location path="test-plans/documentation-release-validation.md" line_range="145-146" />
<code_context>
+1. Follow `Getting Started > Local Setup` from a clean disposable workspace.
+2. Clone the documented control-plane and CLI repositories.
+3. Start the documented core services and environment agent with the supported
+ SP profile.
+4. Run the documented `podman-compose ps`, control-plane health, agent health,
+ and agent provider-list checks.
+5. Build the CLI using the documented command and run `dcm version`.
</code_context>
<issue_to_address>
**Agent checks lack documented procedures**
When the plan is run against the current documentation set, the repository has no documented agent setup or `dcm agent list/get` examples, so operators cannot follow these steps as written and DOC-003 and DOC-009 cannot pass using the published instructions.
Add the missing agent setup and CLI instructions, or make the test explicitly report missing agent documentation as a gap rather than require undocumented steps.
Also at `test-plans/documentation-release-validation.md:254-255`.
</issue_to_address>
### Comment 6
<location path="test-plans/documentation-release-validation.md" line_range="167-168" />
<code_context>
+4. Run the documented shell completion commands for the supported shells.
+5. Confirm documented command names and flags using `dcm --help`, including
+ `dcm agent list` and `dcm agent get` where documented.
+6. Fail the test if any supported page still uses a removed `dcm sp provider`
+ command.
+
+**Expected:** Configuration, precedence, output formats, completion commands,
</code_context>
<issue_to_address>
**Current guides fail the release gate**
When the validation plan is run against the current supported documentation without first resynchronizing those guides, dOC-004 rejects the `dcm sp provider` commands still shown in `content/docs/user-guide/providers.md:18-21`, and DOC-008 flags the multiple-provider workflow in `content/docs/getting-started/register-another-provider.md:15-49`. A release run against the current supported guides therefore fails these checks unless it also performs the broader documentation resynchronization that the plan says is outside this gate.
Either include the affected guide updates in this release gate or scope these checks to pages actually covered by the release.
Also at `test-plans/documentation-release-validation.md:240-242`.
</issue_to_address>
### Comment 7
<location path="test-plans/documentation-release-validation.md" line_range="130-131" />
<code_context>
+3. Run `make check-spell`.
+4. Inspect generated output for Hugo errors, unresolved shortcodes, missing
+ images, and unexpected draft or future-dated pages.
+5. Check internal links and anchors in the generated site, including links
+ generated by the enhancement content integration.
+
+**Expected:** All commands pass; generated pages contain no broken internal
</code_context>
<issue_to_address>
**Generated links go unchecked**
When a release operator follows the listed automated checks without independently selecting a link checker, step 5 names no tool or command, and the Makefile has no link-check target, so no prescribed check verifies generated-site links or anchors. The build can pass with broken links, leaving DOC-002 without a defined link-check result.
Specify a reproducible link-check command and the result that causes the release check to fail.
</issue_to_address>
### Comment 8
<location path="test-plans/documentation-release-validation.md" line_range="253-254" />
<code_context>
+
+**Steps:**
+
+1. For Agents, Service Types, Catalog Items, Catalog Item Instances, Policies,
+ and Service Provider Resources, run at least one list and get example.
+2. Run every documented optional flag at least once: filters, pagination, output
+ format, ordering, and `--show-deleted` where applicable.
</code_context>
<issue_to_address>
**Provider CLI coverage is omitted**
When the release includes provider list/get documentation, dOC-009 requires list/get checks for several user-guide areas but omits Providers, so changes to the supported provider list/get workflow can go untested even when obsolete commands have been removed.
Add Providers to DOC-009 and exercise the supported list/get commands or API workflow with valid IDs and output examples.
</issue_to_address>
### Comment 9
<location path="test-plans/documentation-release-validation.md" line_range="389" />
<code_context>
+
+## Evidence And Results
+
+Create a dated result file beside this plan using the following structure:
+
+```text
</code_context>
<issue_to_address>
**Result files escape ignore rule**
When an operator names the dated result file without the literal `-test-results-` substring, the `.gitignore` pattern `test-plans/*-test-results-*.md` does not match the result file, so Git can include the generated evidence in the documentation PR.
Specify a result filename convention that includes `-test-results-`.
Also at `test-plans/documentation-release-validation.md:390`.
</issue_to_address>Sourcery assessment
Approval pending. 9 findings to address first.
Blocking findings: test-plans/documentation-release-validation.md:274, test-plans/documentation-release-validation.md:197, test-plans/documentation-release-validation.md:184, test-plans/documentation-release-validation.md:83, test-plans/documentation-release-validation.md:146, and 4 more
Signed-off-by: Thomas Stetson <tstetson@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a reusable acceptance plan for validating DCM documentation
against the control plane, CLI, deployment utilities, environment agent, service
providers, and enhancement sources.
Key Changes
environment-agent, provider, enhancement, and published-site checks.
explicit cross-references to implementation and deployment sources.
can remain local or be attached separately without adding generated files to
the documentation PR.
Documentation release validation (
test-plans/documentation-release-validation.md)Reusable test cases covering repository/source inventory, Hugo quality checks,
local setup, CLI discovery, catalog and policy workflows, environment-agent
routing, provider resources, UI/troubleshooting, authentication, enhancements,
regression review, and published-site smoke validation.
Test Flow
CI Impact
validation baseline.
deployment or provider environment in CI.
Validation Run
make buildmake check-formatmake check-spellgit diff --checkChecklist
Summary by Sourcery
Establish a reusable, source-pinned acceptance process for validating DCM documentation against current implementations and deployed behavior.
New Features:
Enhancements:
Documentation:
Tests:
Chores: