test: validate the V-PROGRAM flow on the pre-release network

[odesenfans]

Why

Validation branch for the full V-PROGRAM flow (aleph-rs vprogram CLI + aleph-vm 2.0 + the new scheduler) on pre-release versions of every component, so V-PROGRAM can be signed off end-to-end before anything ships. Stacks on #27 (aleph-vm dev / gRPC split validation): this branch contains #27's commits plus the V-PROGRAM pins; targeting main so the integration tests trigger.

Pins (manifesto.yml)

Component	Pin	What it brings
pyaleph	0.10.3-rc7	V-PROGRAM message handler, vms DB migration 0064, cost model
aleph-cli	0.17.0-rc1	vprogram create/show/call with RA-TLS attestation verification (subsumes #32)
scheduler-rs / scheduler-api	vprogram-rc1	one-off images from od/vprogram-scheduling (scheduler#193): V-PROGRAM ingestion, SEV-SNP capability matching, v_programs allocation bucket
aleph-vm	branch od/vprogram-integration	dev (gRPC split, #1012) + full vprogram stack #1050..#1079: SNP launch path, measured workload volumes, host-mapped attestation port (#1079)

Wire compatibility checked: scheduler#193's allocation bucket is named v_programs, matching aleph-vm#1052's /control/allocations model, and the CLI resolves the attested endpoint from the mapped_ports[8443] entry that aleph-vm#1079 populates.

Current state and next steps

This PR currently proves the existing 25-test suite passes on the fully pinned pre-release network (nothing V-PROGRAM-specific runs yet). Coming next on this branch:

1. test_vprograms.py: deploy the fib-service workload (aleph-vm#1074..#1076, nix-reproducible dm-verity roothash) with aleph vprogram create --wait, assert placement on the SEV-SNP TEE server, then aleph vprogram call with the expected launch measurement: the first genuine end-to-end RA-TLS attestation test in CI.
2. First run computes the expected measurement with aleph-vm#1078's sev-snp-measure helper; switching to CLI-integrated measurement (aleph-rs follow-up) as soon as that lands.

Revert

Validation pins, not permanent: each component moves to its tagged release as they ship.

[foxpatch-aleph]

Well-structured PR that pins all components to V-PROGRAM-capable pre-releases with thorough documentation. The CI log collection refactor (SSH/SUDO variables, --since instead of -n, adding agent logs) is clean and well-motivated. The gRPC socket addition correctly forces the two-process split. No security issues or correctness bugs in the changed code itself. However, the gRPC split introduction means tee-reset.sh should also stop aleph-vm-agent.service (currently only stops the supervisor), and the stale sevctl comment in confidential-artifacts.sh should be updated to match the pr-tests.yml comment update. The read_vm_version() function is now broken for the current manifesto (KeyError on missing 'version' key) but is not called in the current code path — worth making resilient with .get('version', '') for safety.

scripts/tee-reset.sh (line 28): With the gRPC split now forced by ALEPH_VM_SUPERVISOR_GRPC_SOCKET in supervisor.env (crn-up.sh:363), the TEE server also runs aleph-vm-agent.service. This line should also stop the agent before wiping /var/lib/aleph/vm/* (which includes supervisor.sock), otherwise the agent may survive the reset and error out when its socket disappears.

scripts/confidential-artifacts.sh (line 25): This comment is now stale — the PR updates the same rationale in pr-tests.yml (line 283-285) to say all hosts run Ubuntu 24.04, but this comment still says "the TEE server's distro (and its .deb's glibc baseline) may be newer than ours." Consider updating for consistency.

scripts/crn-up.sh (line 75): read_vm_version() does m['components']['aleph-vm']['version'] which will now KeyError since the manifesto replaced version with branch. It's not called in the current code path (the if [ -n "$branch" ] guard on line 328 protects it), but consider using m['components']['aleph-vm'].get('version', '') to avoid a hard crash if someone later removes the branch pin without restoring the version key.

[foxpatch-aleph]

Well-structured V-PROGRAM validation PR with solid test coverage, good security practices (sha256-pinned fixtures, fail-closed attestation test), and clean integration with existing test infrastructure. The code reuses existing patterns correctly and the workflow changes (gRPC socket, log collection, artifact preparation) are properly commented. No blocking issues found — the observations above are non-blocking nits and robustness suggestions.

tests/test_vprograms.py (line 30): Consider matching by message type (e.g., obj.get("type") == "VPROGRAM") like _create_program does with m.get("type") == "PROGRAM" in test_programs.py:79. The "verification" in content heuristic is fragile if the CLI output format changes.

tests/test_vprograms.py (line 65): Substring matching an IP in a URL is technically vulnerable to false positives (e.g., 1.2.3.4 matches 21.2.3.40). Unlikely with real DigitalOcean IPs, but parsing the hostname from the endpoint URL would be more precise.

.github/workflows/pr-tests.yml (line 337): The since="45 min ago" window may not cover the full test suite. The V-PROGRAM test alone can take up to 20 min (CREATE_WAIT_SECS=900 + 300s timeout margin), and it runs after test_confidential (900s) and others. Consider 60 or 90 min to ensure early failures are captured.

scripts/vprogram-artifacts.sh (line 32): bundle-info.json is downloaded and checksum-verified but never used by any test or script. If it's not needed at runtime, consider removing it from the download loop to save a (small) amount of CI time.

scripts/crn-up.sh (line 75): read_vm_version does m['components']['aleph-vm']['version'] which will raise KeyError if the manifesto uses branch instead of version (as it does in this PR). Using .get('version', '') would prevent a crash if ALEPH_VM_VERSION is set without ALEPH_VM_BRANCH during the temporary pin period.

[foxpatch-aleph]

The PR is well-structured with good documentation, proper sha256-pinned fixture downloads, and thoughtful workflow improvements (TEE log collection, gRPC split forcing). However, the new test has two genuine issues: (1) no teardown — unlike every other VM-creating test in the suite, a failed assertion leaves a V-PROGRAM VM running on the shared TEE server, and (2) no per-test timeout, so a hang can consume the entire 60-minute CI budget. The missing scheduler readiness check (present in test_confidential.py) risks a 900s placement timeout, and bundle-info.json is downloaded and checksummed but never used.

tests/test_vprograms.py (line 53): Missing teardown: every other VM-creating test in this suite (test_programs.py, test_confidential.py) wraps its assertions in try/finally to delete/forget the VM. A failed assertion here leaves a V-PROGRAM VM running on the shared TEE server, consuming resources that can affect subsequent CI runs. Add a try/finally that calls aleph_cli("vprogram", "delete", item_hash, "-y", "--chain", "eth", check=False) (or whatever the vprogram forget/delete subcommand is).

tests/test_vprograms.py (line 53): Missing @pytest.mark.timeout: this test can run 900s (create wait) + 600s (endpoint poll) + 3×120s (calls) = ~30 min with no per-test guard. Other tests in the suite use @pytest.mark.timeout(900) or @pytest.mark.timeout(600). A hang here would consume the entire 60-minute CI budget. Consider @pytest.mark.timeout(1800) or similar.

tests/test_vprograms.py (line 53): No scheduler readiness pre-check: test_confidential.py polls the scheduler API (lines 93-111) to confirm the TEE node is seen as confidential-capable before creating an instance. Without this, the V-PROGRAM create can hit a 900s timeout if the scheduler hasn't yet observed the TEE node's SEV-SNP capability. Consider adding an equivalent check before vprogram create.

scripts/vprogram-artifacts.sh (line 32): bundle-info.json is downloaded and sha256-verified but never used by either the test or the conftest fixtures. Only snp-image.tar.gz, manifest-template.json, and fib-workload.ext4 are referenced. Either the test should consume it (e.g. to assert the bundle's sha256 matches the manifest) or it should be removed from the download loop.

[foxpatch-aleph]

This PR adds V-PROGRAM end-to-end validation on a pre-release network: a new test (test_vprograms.py) deploys a measured SEV-SNP workload, verifies attested calls via RA-TLS, checks fail-closed behavior on wrong measurements, and probes IPv6 reachability. Supporting changes include fixture download with sha256 verification (vprogram-artifacts.sh), CI workflow updates for TEE network diagnostics and log collection improvements, tee-reset.sh hardening against leaked controller units, and gRPC split + Rust supervisor configuration for confidential CRNs. The code is well-commented with clear rationale for non-obvious decisions, subprocess calls use safe list-based invocation (no shell injection risk), and retry logic is bounded with appropriate deadlines. Minor observations: the curl_probe in the retry loop is only captured once (stale if the endpoint becomes reachable later), and the assumption that objs[-1] is the ready payload is tied to the pinned CLI version. Neither blocks merging.

tests/test_vprograms.py (line 96): The curl_probe is captured only on the first CLI failure (if curl_probe is None). If the endpoint becomes reachable later (e.g. after the guest agent binds :8443) but the CLI still fails for a different reason (e.g. attestation verification), the assertion message will show a stale HTTP {curl_probe} value from the first failure. Consider re-capturing the probe on each failure, or at least noting in the comment that the probe is a first-failure snapshot only.

tests/test_vprograms.py (line 67): ready = objs[-1] assumes the last JSON object in the --json stream is the wait/ready payload. This is tied to the pinned CLI version (0.17.0-rc9) behavior. If the CLI ever emits an additional message after the ready payload, this would silently pick the wrong object. A defensive check like assert ready.get("ready") is not None before the existing assertion would catch this mismatch early.

scripts/vprogram-artifacts.sh (line 6): The manifest STORE hash in the comment is split across two lines (c4e882eb21bd82393e831dd1c828efd2676741c5de2ac1f + b0d966917e104bf0b), making it hard to read. Consider joining it on one line or adding a note that it's a line-wrap.

.github/workflows/pr-tests.yml (line 367): The grep ... tee-network.txt | while read loop reads from and appends to the same file. This works because grep reads the file fully before piping, but it's a pattern that could confuse future maintainers. A brief comment noting that the appends don't affect the grep input would help.

[foxpatch-aleph]

Well-structured PR that adds the first genuine end-to-end V-PROGRAM RA-TLS attestation test to CI, along with supporting infrastructure (fixture download script, pytest fixtures, TEE network diagnostics, and operational fixes for tee-reset.sh). The test logic is sound: it validates placement on the TEE server, exercises the full attested call path, tests fail-closed behavior on wrong measurements, and includes IPv6 reachability as a soft probe. Comments are exceptionally thorough, documenting race conditions, temporary workarounds (TCB floor bypass), and the reasoning behind each non-obvious decision. The workflow diagnostics capture every piece of the attestation-port DNAT and IPv6/ndppd path, which will be critical for debugging. No correctness bugs or security issues found.

tests/test_vprograms.py (line 96): Minor: curl_probe is only captured on the first failure (line 104 guard). If the first failure is a transient race and a later failure is a different root cause, the AssertionError will show the stale first-failure curl result as the diagnostic. Consider refreshing curl_probe on each non-transient failure, or document that the first-failure capture is intentional (the comment explains the intent but the stale-value edge case is not called out).

tests/test_vprograms.py (line 142): Minor: guest_ipv6 is read from shown, which was fetched at line 85 — potentially minutes before this IPv6 probe runs (after all attested calls complete). If the VM restarted between show and the IPv6 probe, the address could be stale. Low risk in practice given the short test window, but worth a note.

.github/workflows/pr-tests.yml (line 391): The grep -E "dnat.*:8443" tee-network.txt | while read -r rule loop appends to tee-network.txt with >> while grep is reading it. This is safe in practice because the appended lines (=== probe: / attestation DNAT rule:) don't match the dnat.*:8443 pattern, but it's a fragile pattern. Consider grepping into a temporary variable first, then iterating, to make the non-overlap guarantee explicit.

scripts/vprogram-artifacts.sh (line 47): Bash associative arrays (declare -A) do not guarantee iteration order, so the download sequence is non-deterministic. Harmless since each download is independent, but if you ever add ordering dependencies (e.g., manifest depends on bundle being present), switch to an ordered approach.

[foxpatch-aleph]

Well-documented V-PROGRAM validation PR with proper fixture pinning, thorough attested-call coverage, and comprehensive CI diagnostics. Two non-blocking issues: the test lacks cleanup/teardown (unlike the established try/finally pattern in test_confidential.py and test_programs.py), and a missing @pytest.mark.timeout decorator for a test that can run 15+ minutes.

tests/test_vprograms.py (line 48): Missing cleanup/teardown. Both test_confidential.py (line 204: finally: delete_instance(aleph_cli, vm_hash)) and test_programs.py (line 140-143: finally: aleph_cli("program", "delete", ...)) use try/finally to clean up after themselves. This test leaves the V-PROGRAM running on the TEE server after completion. While tee-reset.sh wipes state in CI, a local run would leave a running VM. Consider wrapping the body in try/finally with a vprogram delete (or equivalent) call.

tests/test_vprograms.py (line 48): Missing @pytest.mark.timeout. The test can run for up to ~1200s (CREATE_WAIT_SECS=900 + 300s timeout margin) plus the 120s attested-call retry loop and 60s IPv6 probe. The existing tests use @pytest.mark.timeout(600) or @pytest.mark.timeout(900). Without this, a hung CLI call could run until the CI job's 60-minute timeout instead of failing fast.

tests/test_vprograms.py (line 67): Assuming objs[-1] is the ready payload is correct per the CLI's --wait contract (documented in the comment at line 71), but it relies on the CLI never emitting additional JSON objects after the ready payload. If the CLI ever adds a trailing diagnostic object, this would silently pick up the wrong thing. Low risk given the contract, but a next((o for o in objs if o.get("ready") is not None), None) would be more robust.

tests/test_vprograms.py (line 116): The transient-retry logic only matches the exact string "error sending request" in stderr. If the CLI changes its error message wording (e.g. in a future rc), this retry loop would stop retrying and immediately fail. Consider matching a broader pattern like "error sending request" or "transport" or "connection" to be more resilient, or document that this string is part of the CLI's stable error surface.

.github/workflows/pr-tests.yml (line 381): The IPv6 CIDR regex 2[0-9a-f:]+::[0-9a-f]+0/124 assumes the host tap address ends in hex 0 and starts with 2 (global unicast). This is a CI diagnostic step (not a test assertion), so it's acceptable, but if the addressing scheme ever changes the probes will silently match nothing and the diagnostic will be incomplete without any error.

[foxpatch-aleph]

Well-structured PR introducing the first end-to-end V-PROGRAM test with comprehensive attestation verification, thorough documentation of timing constraints and temporary TCB workarounds, and proper sha256-pinned fixture downloads. The CI diagnostics additions (network probes, agent logs, controller unit cleanup) are well-motivated by specific observed failures. No bugs or security issues found.

scripts/crn-up.sh (line 76): Latent fragility (pre-existing, not introduced by this PR): read_vm_version() does m['components']['aleph-vm']['version'] which will KeyError when the manifesto uses branch: instead of version:. It is currently unreachable because read_vm_branch() returns non-empty and the caller only invokes read_vm_version() in the else branch, but adding a .get('version', '') with an empty-string guard would make the mutual exclusivity explicit and prevent a confusing crash if someone clears the branch without restoring the version key.

tests/test_vprograms.py (line 67): The assumption that objs[-1] is the readiness payload depends on the CLI emitting --wait's ready object last in the JSON stream. This is currently correct per aleph-rs#318, but a defensive approach would be to search for the object with ready key (like _vprogram_message searches for type == V-PROGRAM) rather than assuming positional ordering. Not blocking — the current code works and the assertion on ready.get('ready') is True will catch any ordering regression clearly.

tests/test_vprograms.py (line 142): The guest_ipv6 field is read from shown (the vprogram show result) via shown.get('ipv6_ip'). If the CLI changes this field name, the probe will silently skip with a warning rather than failing — which is the intended behavior for this first run, but worth noting as a coupling to track when promoting to an assertion.

[foxpatch-aleph]

Well-structured V-PROGRAM validation PR with thorough diagnostics and good test coverage of the attested call path. The test correctly validates placement, attestation, fail-closed behavior, and IPv6 reachability (as a warning). Two issues worth addressing: the test lacks a try/finally cleanup block (unlike every other E2E test in the suite), and the TEE network diagnostics A/B test for kernel proxy-NDP will silently never run because it reads from a file still being written by a redirect block. The overall approach — pinning pre-release components, adding fixtures with sha256 verification, and extensive diagnostics — is solid.

tests/test_vprograms.py (line 48): Missing try/finally cleanup. Every other E2E VM test in the suite deletes its VM in a finally block (test_confidential.py:204 finally: delete_instance(...), test_programs.py:143 finally: aleph_cli("program", "delete", ...)). Without cleanup, a failed V-PROGRAM test leaves a running VM on the TEE server consuming resources until tee-reset.sh runs at end-of-CI. If the test fails early (e.g. in the create step), the V-PROGRAM is left deployed on the network with no teardown. Consider wrapping the body after item_hash is obtained in a try/finally that calls aleph_cli("vprogram", "delete", item_hash, "-y", "--chain", "eth", check=False).

.github/workflows/pr-tests.yml (line 391): The uplink variable will always be empty. The grep reads tee-network.txt, but that file is the target of the enclosing { ... } > tee-network.txt 2>&1 redirect block (line 400) — its content is only flushed to disk when the block completes, so grep -oE '^proxy [a-z0-9]+' tee-network.txt reads an empty (or nonexistent) file. The A/B kernel proxy-NDP test at lines 392-398 is therefore silently dead code. To fix, either move the A/B test outside the redirect block, or capture the ndppd config output into a shell variable before the block and grep that variable.

.github/workflows/pr-tests.yml (line 382): Unsafe interpolation: python3 -c "... '$cidr' ..." injects $cidr (derived from ip addr output via grep) into a Python string. The regex 2[0-9a-f:]+::[0-9a-f]+0/124 is tight enough to prevent exploitation in practice, but this is still an unsafe pattern — a malformed address could break the Python parser. Consider passing the CIDR as an argument: python3 -c "import ipaddress, sys; n=ipaddress.ip_interface(sys.argv[1]); print(n.ip + 1)" "$cidr".

.github/workflows/pr-tests.yml (line 404): The DNAT probe loop does grep -E "dnat.*:8443" tee-network.txt | while read -r rule but the file is still being written by the enclosing redirect block. Similar to the uplink issue, at this point tee-network.txt exists but may not have all its content flushed yet (bash line-buffering for > file redirects means prior writes are visible, but the nft ruleset output at line 357 should already be flushed). This particular case likely works because nft output comes early in the block, but it's fragile — the file is not fully written until line 400's block closes. Moving the DNAT probing outside the redirect block would be more robust.

tests/test_vprograms.py (line 95): Retry timing: the deadline is time.time() + 120 but each aleph_cli call has timeout=120. If the first call takes the full 120s (e.g. a hung TLS handshake), the deadline expires before any retry can happen, so the loop exits after a single attempt. Consider either reducing the per-call timeout (e.g. 30s) or increasing the deadline to allow at least 2-3 retries.

[foxpatch-aleph]

Well-structured PR that adds end-to-end V-PROGRAM validation on a fully pinned pre-release network. The new test (test_vprograms.py) correctly exercises the full attested call path with fail-closed verification, the fixture download script (vprogram-artifacts.sh) verifies sha256 checksums, and the CI diagnostics are thorough. The temporary TCB floor workaround is clearly documented with a removal condition. I have a few minor observations but no blocking issues.

tests/test_vprograms.py (line 86): The test asserts shown["measurements"] is truthy but doesn't verify the measurements match what was pinned at create time. Since the CLI computes measurements at create time and the CRN launches from the same bundle, a create→show measurement comparison would strengthen the test and catch a regression where the CLI and CRN compute different measurements for the same bundle. Consider asserting a specific measurement value (e.g. the 952167b9... mentioned in vprogram-artifacts.sh) appears in the show output.

tests/conftest.py (line 511): The manifest upload uses the default 180s timeout from _upload_with_balance_retry, while the bundle upload (line 500) uses 300s. The manifest is tiny so this is fine, but consider passing an explicit timeout for symmetry and to document the intent.

scripts/vprogram-artifacts.sh (line 48): for asset in "${!CHECKSUMS[@]}" iterates bash associative array keys in non-deterministic order (hash order). Each download is independent and verified, so this is functionally correct, but it means the log output order varies between runs. Not worth changing, just noting for anyone comparing CI logs between runs.

.github/workflows/pr-tests.yml (line 381): The regex 2[0-9a-f:]+::[0-9a-f]+0/124 assumes the /124 network's host portion ends in 0 (so n.IP + 1 gives the guest). This is diagnostic-only code that runs if: always(), so it won't affect test pass/fail, but if the guest isn't at network + 1 the probes will target the wrong address and the diagnostics will be misleading.

manifesto.yml (line 27): PR description says aleph-cli is pinned to 0.17.0-rc1, but the actual pin is 0.17.0-rc9. The code comments explain the rc9 bump (TCB floor recalibration), so the code is self-documenting, but the PR description should be updated to avoid confusion for reviewers.

manifesto.yml (line 60): PR description says aleph-vm is pinned to branch od/vprogram-integration, but the actual pin is od/vprogram-integration-2 (superseding #1114 via #1124). The code comment is accurate; the PR description is stale.

[foxpatch-aleph]

Well-structured V-PROGRAM validation PR with thorough documentation and a solid end-to-end test. The TCB floor override is clearly temporary and well-justified. However, the tee-network.txt dual-capture diagnostic has a self-referential read bug that makes it dead code (grep reads from the file being written to, which is empty at that point), the new test is missing a @pytest.mark.timeout decorator that every other VM test has, and there's no V-PROGRAM cleanup in a finally block unlike the established pattern in test_programs.py and test_confidential.py. None of these are blockers for a validation branch, but the dual-capture bug means an intended diagnostic is silently non-functional.

.github/workflows/pr-tests.yml (line 401): The entire diagnostic block (lines 359-418) redirects to tee-network.txt via } > tee-network.txt 2>&1 at line 418, but lines 401-402 grep from tee-network.txt itself. The shell truncates the file when it opens it for the redirect, so these greps read an empty file — uplink and tap are always empty, and the dual-capture block (lines 403-416) is silently skipped. To fix: write the diagnostic to a temp variable/file first, then grep from it, or restructure so the initial network state is captured before the redirect block begins.

.github/workflows/pr-tests.yml (line 401): The grep pattern ^proxy [a-z0-9]+ is meant to extract the uplink interface from ip -6 neigh show proxy output, but that command typically outputs lines like 2001:bc8:702:32d:: dev enp1s0 proxy — the word proxy is at the end, not the start. Even if the file were populated, this pattern would not match. Consider parsing from ip -6 route show default or ip addr output instead.

tests/test_vprograms.py (line 48): Missing @pytest.mark.timeout() decorator. Every other VM test in the suite has one (300s-1800s). Without it, a hung operation (e.g. vprogram create --wait stalling) could run until the 60-minute CI job timeout, leaving no time for log collection or cleanup. Suggest @pytest.mark.timeout(1200) to match the 900s create wait + retry margins.

tests/test_vprograms.py (line 65): No cleanup in a finally block. test_programs.py and test_confidential.py both delete their VMs in finally to avoid leaking running instances on failure. A V-PROGRAM left running on the TEE server could interfere with the reset step or subsequent runs. Consider adding a finally: block that calls aleph_cli("vprogram", "delete", item_hash, ...) (or whatever the CLI delete command is), even if best-effort.

scripts/vprogram-artifacts.sh (line 48): Bash associative arrays iterate in unspecified order, so the download order of the three fixtures is non-deterministic. Harmless (each download is independent), but the output ordering in CI logs will vary between runs. If deterministic output is desired, iterate over a sorted key list instead of ${!CHECKSUMS[@]}.

[foxpatch-aleph]

Well-structured V-PROGRAM validation PR with thorough diagnostics and good test coverage of the attested call path, IPv6 reachability, and fail-closed behavior. The code is well-documented with run-number references and inline rationale. No bugs or security issues found; the comments below are minor improvements for robustness and consistency.

tests/test_vprograms.py (line 47): Consider adding a try/finally block to delete the V-PROGRAM after the test, consistent with test_programs.py (which calls program delete) and test_confidential.py (which calls delete_instance). The TEE server is reset by tee-reset.sh, but the V-PROGRAM message persists on the testnet.

tests/test_vprograms.py (line 31): Consider adding @pytest.mark.timeout(1200) (or similar) to match the pattern in test_confidential.py which uses @pytest.mark.timeout(900). The total test duration (900s create wait + 120s call retries + 60s IPv6 probe + fail-closed call) is long and a timeout marker would help catch hangs.

.github/workflows/pr-tests.yml (line 401): The diagnostics step reads tee-network.txt with grep while the same shell is still writing to it via the { ... } > tee-network.txt 2>&1 redirect on line 359/418. This works because earlier sections are already flushed, but it is fragile. Consider extracting the interface names into shell variables earlier (e.g. capturing uplink and tap right after the ip addr / ndppd outputs) instead of re-reading the file.

tests/test_vprograms.py (line 103): curl_probe is captured only on the first failure and never updated on subsequent retries. If the first failure is transient (transport) but a later one is attestation-related, the stale probe value in the assertion message could be misleading. Consider re-probing on each failure or at least noting in the assertion that the probe was from the first failure.

.github/workflows/pr-tests.yml (line 275): The IPv6 pool 2001:bc8:702:32d::/64 is hardcoded in the workflow. If the Scaleway flexible IP changes, this requires a workflow edit. Consider using a GitHub variable (e.g. ${{ vars.STATIC_CRN_IPV6_POOL }}) for maintainability, consistent with how DO_SSH_KEY_FINGERPRINT is already handled.

[foxpatch-aleph]

Well-structured V-PROGRAM validation PR with thorough documentation of non-obvious decisions. The new test covers the full attested call path, fail-closed measurement verification, and IPv6 reachability. One notable gap: the test lacks a try/finally cleanup block (unlike all comparable tests in the suite), leaving the V-PROGRAM VM running on the TEE server if the test fails partway through, and the V-PROGRAM message is never FORGOTTEN from the chain. The test also lacks a @pytest.mark.timeout decorator that every other long-running test has. The TCB floor workaround is well-documented and clearly temporary. The diagnostics collection is comprehensive but contains a fragile IPv6 regex that is non-blocking since it's diagnostics-only. No security issues found — the sha256 verification of fixtures, isolated CLI config, and proper credential handling are all correct.

tests/test_vprograms.py (line 47): No try/finally cleanup block — every comparable test (test_programs.py:140-143, test_confidential.py:204-205) deletes the VM/message in a finally. If the test fails after vprogram create, the VM stays running on the TEE server and the V-PROGRAM message persists on the chain. Consider adding a finally: block that calls aleph_cli("vprogram", "delete", item_hash, ...) or equivalent, using check=False for best-effort cleanup.

tests/test_vprograms.py (line 47): Missing @pytest.mark.timeout(...) decorator. Every other long-running test has one (e.g., test_confidential.py uses 900s, test_programs.py uses 600s). Given CREATE_WAIT_SECS = 900 plus retry loops, a per-test timeout guard would prevent a hung polling loop from burning the entire 60-minute CI budget.

tests/conftest.py (line 508): manifest["bundle"]["ref"] = bundle_hash will raise an opaque KeyError if the manifest template's schema changes (e.g., the key is runtime instead of bundle). An explicit assertion or .get() with a descriptive error would make schema drift easier to diagnose.

.github/workflows/pr-tests.yml (line 385): The IPv6 regex 2[0-9a-f:]+::[0-9a-f]+0/124 assumes the tap address ends in ...0 (so the guest gets ...1 via n.ip + 1). This is correct for the current 2001:bc8:702:32d::/64 pool but could silently miss addresses if the allocation doesn't start at ...0. Non-blocking since this is diagnostics-only code.

scripts/vprogram-artifacts.sh (line 36): The sha256 hashes are duplicated between CHECKSUMS and SOURCES (each hash appears twice). A single array could derive both the checksum and the URL, reducing the risk of a copy-paste mismatch when updating fixtures.

.github/workflows/pr-tests.yml (line 275): STATIC_CRN_IPV6_POOL=2001:bc8:702:32d::/64 is hardcoded in the workflow. Consider using a GitHub variable or secret so it can be updated without a code change if the Scaleway flexible IP changes.

[foxpatch-aleph]

This PR adds the V-PROGRAM end-to-end validation flow to the testnet integration suite. It pins pre-release versions of pyaleph, aleph-cli, scheduler-rs, scheduler-api, and aleph-vm (from a feature branch) to validate the full SEV-SNP attestation pipeline. The new test file (test_vprograms.py) is comprehensive: it deploys a measured workload, asserts placement on the TEE server, exercises attested calls with RA-TLS verification, probes IPv6 reachability, and validates fail-closed behavior on wrong measurements. The supporting infrastructure (vprogram-artifacts.sh, conftest fixtures, crn-up.sh env changes, tee-reset.sh fixes, CI workflow diagnostics) is well-documented and follows the existing patterns in the codebase. The code is correct, with no logic errors or security issues. The only minor observations are non-blocking: a diagnostic regex that may not match the expected output format (harmless due to the guard clause), and the hardcoded IPv6 pool in the workflow (acceptable for a test environment with a known static server).

.github/workflows/pr-tests.yml (line 401): The regex ^proxy [a-z0-9]+ is intended to extract the uplink interface from ip -6 neigh show proxy output, but that command outputs lines like 2001:bc8:702:32d:: dev enp1s0 proxy (address first, then dev <iface>, then proxy). The ^proxy anchor at the start of the line will not match this format, so uplink will likely be empty and the dual tcpdump capture will be skipped. The ip addr output (written earlier) does contain iface vmtapN lines, so the tap extraction should work. Since the whole block is guarded by if [ -n "$uplink" ] && [ -n "$tap" ] and is purely diagnostic, this is non-blocking, but the regex could be fixed to dev ([a-z0-9]+).* proxy for correctness.

.github/workflows/pr-tests.yml (line 275): The IPv6 pool 2001:bc8:702:32d::/64 is hardcoded in the workflow. Consider moving this to a GitHub variable (vars.SCALEWAY_IPV6_POOL) or a secret, consistent with how AMD_SEV_SNP_HOST is handled — it is environment-specific and would need updating if the TEE server's flexible IP changes. Non-blocking since this is a test-only CI workflow with a known static server.

tests/test_vprograms.py (line 103): The curl_probe is captured only once (on the first failure) and reused in all subsequent retry failure messages. If the first failure is a transient race (endpoint not yet bound) and a later failure is attestation-related, the error message will show the stale first-probe HTTP code rather than the state at the time of the final failure. Minor diagnostic limitation; the retry logic itself is correct.