test: release-stack run: pyaleph 0.11.2 / aleph-cli 0.18.0 / scheduler 0.2.1 / aleph-vm 2.0.1 rc - #38
test: release-stack run: pyaleph 0.11.2 / aleph-cli 0.18.0 / scheduler 0.2.1 / aleph-vm 2.0.1 rc#38odesenfans wants to merge 67 commits into
Conversation
Point the CRN deployment at the aleph-vm dev-accelerate branch (PR #977: gRPC supervisor rework, integration suite, directory-based VM migration) so the testnet migration test exercises it end to end before merge. crn-up.sh fetches aleph-vm.debian-12.deb from the branch's CI build (build-deb-package-and-integration-tests.yml); the latest dev-accelerate run is green and produces that artifact. Revert to version: 1.13.0 once aleph-vm#977 lands.
scheduler 0.1.2 (#188) makes node_watcher poll the v2 executions endpoint, so a gracefully-stopping VM stays observed (present as STOPPED) instead of dropping out of the v1 running-only list and being reported 'missing' for the ~34s graceful-shutdown window. That is what broke test_instance_stop_start on the dev-accelerate aleph-vm (whose stop is a graceful guest shutdown vs the old near-instant teardown). Pairs with aleph-vm's stop=stop_vm change, which keeps the stopped VM in the pool so v2 still lists it.
aleph-vm#977 (supervisor rework) merged into dev, so the temporary dev-accelerate branch pin is obsolete (that branch will be deleted). Point at dev to keep the migration and graceful-stop tests running against the merged code until it reaches main and a release is cut. Scheduler stays at 0.1.2 (graceful-stop tolerance).
Validate Phase 2 (gRPC wire + two-service packaging) on testnet. Phase 1 (#980) has merged to dev; phase2 carries it plus the Phase 2 changes.
The static AMD SEV-SNP server runs Ubuntu 24.04 (Python 3.12), but the branch path deployed the debian-12 .deb (cp311), so its vendored pydantic_core native extension failed to import there and the supervisor crash-looped (confidential test: :4020 connection refused). Provision CRN droplets as ubuntu-24.04 (CCN already is) and fetch/deploy the ubuntu-24.04 branch .deb so the Python ABI matches on every host including the TEE server.
Diagnosing the confidential create/registration on the SEV box: in the two-service split the agent serves /control/* and /v2/about/executions/list, so the confidential allocation result and the awaiting_confidential_init state live in its journal, not the supervisor daemon's. Also switch the TEE journals from -n 5000 to --since: the supervisor crash-loops during the static-server reinstall and blew the line budget before the test window.
Validate PR-1 of the agent/supervisor boundary split (contract layer + import-linter, behavior-neutral) on testnet.
…re origin The test_confidential init-session 'VM not found' failure reproduced 2x on #986 (od/agent-supervisor-contract), whose confidential-path code is import-only vs dev. Run against dev to determine if the breakage is pre-existing (#984/#985) rather than caused by #986. Will revert the pin afterwards.
…l failure is pre-existing on dev Isolation run (manifesto pinned to dev) failed test_confidential identically (VM not found at init-session), proving the breakage is pre-existing on dev (#984/#985), not caused by #986. Restoring the pin to the PR-1 branch.
…#990) Validate the complete combined stack (tip branch builds a .deb with all four PRs) on testnet integration. Combined-stack GitHub CI is green (PR #991). Known: the confidential SSH test is a pre-existing dev failure (needs the .deb-selection fix + SEV hw); the other 23 are expected to pass.
The generated supervisor.env omitted ALEPH_VM_SUPERVISOR_GRPC_SOCKET, so the agent ran its own in-process VmPool and never dialed the supervisor daemon. The daemon ran idle alongside it. Every prior green #27 validated the in-process path, not the two-process gRPC connector the boundary work is for. Set the socket so the agent talks to the daemon over gRPC. Expect split-mode gaps to surface (endpoints that still reach the in-process pool: backups, restore, confidential, migration, network recreation, GPU reservation, persistent programs).
The manifesto pinned od/fix-confidential-reconcile-teardown, deleted after #987 merged, so crn-up.sh could not fetch a build. dev now carries the full boundary stack + the firecracker Configuration vm_id fix, and crn-up.sh sets the gRPC socket. Deploy dev to exercise the two-process connector end-to-end for the first time.
…(aleph-vm#1001-#1010)
…24/24 on the testnet)
…-accelerate # Conflicts: # .github/workflows/pr-tests.yml # scripts/crn-up.sh
Point every component at its V-PROGRAM-capable pre-release so the full flow (CLI create/show/call with RA-TLS attestation) can be validated on the testnet: - pyaleph 0.10.3-rc7: first pinned release with the V-PROGRAM message handler, vms DB migration and cost model. - aleph-cli 0.17.0-rc1: vprogram command set (create/show/call with RA-TLS attestation verification). Subsumes the pin from PR #32. - scheduler-rs / scheduler-api vprogram-rc1: one-off images built from od/vprogram-scheduling (scheduler#193), adding V-PROGRAM ingestion, SEV-SNP capability matching and the v_programs allocation bucket. - aleph-vm od/vprogram-integration: dev (gRPC split) plus the full vprogram stack (#1050..#1079) including the SNP launch path, measured workload volumes and the attestation port host mapping.
Adds the first genuine end-to-end attestation test: deploy the fib-service measured workload with the Rust CLI, wait for scheduler placement on the SEV-SNP TEE server, then exercise vprogram call, which only returns a body after the AMD report chain verifies, the TLS key is bound to the report, and the launch measurement matches the ones the CLI pinned at create time. A wrong --expected-measurement must fail without printing any body. Pieces: - scripts/vprogram-artifacts.sh: fetch the prebuilt fixtures (runtime bundle, manifest template, fib-workload.ext4) from the vprogram-fixtures-1 release, sha256-pinned. The fixtures are nix-reproducible builds of aleph-vm od/vprogram-integration rev de9ed350; rebuild instructions in the release notes. - conftest: vprogram_dir + vprogram_runtime_hash fixtures. The bundle is uploaded per run (fresh CCN each time); the manifest template's zeroed bundle.ref is patched with the run's STORE hash before upload. - local-up.sh: export ALEPH_TESTNET_VPROGRAM_DIR when the fixtures dir exists; the test skips otherwise. - pr-tests.yml: fetch fixtures on the CCN + install cryptsetup-bin there (vprogram create runs veritysetup locally to hash the workload).
… sample First live run (31373843773) taught two things: - vprogram create --json emits a submission receipt (type: V-PROGRAM, item_hash, message_status) rather than a full message envelope; match on the type field instead of content.verification. - The rc1 CLI resolves the attested endpoint exactly once at readiness, before the CRN maps the attestation port (the mapping happens only after the SNP guest's measured boot: aleph-vm agent/run.py reconcile_vprogram_port_forwards runs post-_wait_until_running). Poll vprogram show as a fallback so the attestation path still gets exercised; drop the fallback once aleph-rs#318 ships in the pinned CLI.
Round-2 run (31378982391) launched the V-PROGRAM but the guest never started: the agent created the VM then hit 'did not reach RUNNING within 120.0s' twice, and no aleph-vm-controller@ unit ever appeared for it. Root cause: the aleph-vm deb ships both supervisor daemon implementations behind ALEPH_VM_SUPERVISOR_IMPL (default python), and the V-PROGRAM SNP auto-launch only exists in the Rust daemon (lifecycle.rs: session-dir derivation, verity sidecar force-insert, measured cmdline with workload_roothash; none of it exists in the python daemon). The python daemon parks the AlephQemuConfidentialInstance execution waiting for the operator session dance that classic confidential instances perform and V-PROGRAMs never do. Set ALEPH_VM_SUPERVISOR_IMPL=rust on the confidential CRN only (the TEE server, the sole node that can run V-PROGRAMs). The Rust daemon also implements the classic CVM session flow (confidential.rs), so test_confidential stays covered; DO CRNs keep the default python daemon that PR #27 has been validating. Also stop + reset-failed all aleph-vm-controller@ units in tee-reset.sh: the state wipe deletes controller.json files but left the units restart-looping (observed at restart counter 844k), flooding the journal and drowning the CI log capture.
Round 3 (31382627461) got the full chain working: measured SNP boot
under the Rust daemon, attestation port mapped, endpoint resolved on
the TEE server, and the guest's aleph-attest-agent bound :8443 with an
attested TLS identity - 3 seconds AFTER the test's first call. The CRN
maps the port at RUNNING (qemu up), a few seconds before guest
userspace finishes starting, so the first connection can land in that
gap.
Retry transport-level failures ('error sending request') for up to two
minutes; attestation verification failures still fail immediately.
Round 4 (31385362211): the SNP guest boots, aleph-attest-agent binds 0.0.0.0:8443, the DNAT host port is allocated and surfaced via the executions list, yet external connections to it fail for minutes (the CLI only sees 'error sending request'). The confidential instance's SSH forward works on the same host, so the failure is specific to the V-PROGRAM mapping. Capture the host-level view after tests: nft ruleset, interfaces, routes, listeners, plus two live probes: TEE host -> guest :8443 directly (guest reachability) and runner -> public host port (full DNAT path). Their combination localizes the break.
Round 5's diagnostics only probed the first dnat-to-:8443 rule, which turned out to be a STALE mapping from an earlier run (nft rules survive tee-reset): port 24006 answered 200 while the run's own port 24008 went unprobed. Loop over all rules so each mapping gets a host->guest and a runner->DNAT probe, with compact status-code output.
…ails Rounds 4-6 all show the identical signature: the guest's attest agent is up and listening, the run's own DNAT rule exists through the failure window (round 5's diagnostics listed it), a stale sibling rule to the same guest IP serves curl 200, yet vprogram call fails with 'error sending request' for the full retry window. Everything points at the CLI's RA-TLS verifier rejecting the handshake (most plausibly a launch measurement mismatch), which reqwest hides behind a transport-looking error until aleph-rs#319 ships. Capture ground truth at failure time: on the first failed call, curl the same endpoint with no verification and carry the HTTP status into the final assertion message. 200 there + CLI failure = verifier rejection, beyond doubt.
rc2 carries aleph-rs#318 (create --wait polls for the attested endpoint within the wait budget) and #319 (attestation rejections are named instead of hiding behind 'error sending request'). - manifesto: aleph-cli 0.17.0-rc1 -> 0.17.0-rc2. - test: remove the show-polling fallback; the ready payload's attested_endpoint is the CLI contract again. The warm-up retry stays (the port maps a few seconds before the guest binds), but with #319 a verification failure no longer matches the transient predicate, so it fails fast with the actual reason.
Root cause of runs 32372006187/32375910277, reproduced in a local qemu boot of the compose runtime: rc10's docker fallback saves the archive from the digest-pinned ref, and docker save name@sha256:... writes RepoTags: null, so the guest's podman load imports a bare image ID that podman-compose cannot match against the pinned image: string (short-name resolution fails without a registries.conf, fail-closed poweroff). Supply the archive ourselves with a TAGGED docker save and --image-archive, which the CLI maps verbatim: podman load restores the tag and compose resolves it from local storage (whoami verified up on :8080 in the local boot). Archive bytes stay verity-measured either way. Drop once the CLI saves archives under a podman-matchable reference. Also raise the job timeout to 75 min: two SNP boots per run pushed the suite past 60 (run 32375910277 was killed at the limit).
…shed aleph-vm aggregation All three registers-schema adopters (pyaleph 0.10.3-rc9, aleph-cli 0.17.0-rc11, aleph-vm od/vprogram-integration-3 rebuilt on dev 2df8d87e) move together; scheduler images bumped to the vprogram-rc2 build.
Run 32416035775 attempt 2 lost the documented nodestatus-balances race: the confidential rootfs upload still saw 'Insufficient balance' after 180s while the same upload passed on attempt 1 and every earlier run of the day. Give the asynchronous funding more room; real cost shortfalls still fail, just later.
The validation aggregation branch is fully merged (aleph-vm #1131, #1147, #1148); dev's tree matches the branch that passed run 32426136992.
scheduler-rs/scheduler-api move to the 2026-08-24 vprogram-rc3 build. aleph-vm stays pinned to dev, which gained the measured-CPU SNP launch (#1145), agent-owned VM disks (#1149) and the SNP vCPU probe retry (#1150) since the last green run.
dev gained agent-side backups with supervisor guest quiescence (#1151), disk admission before resource download (#1153), the dead proto reservation cleanup (#1154) and FAILED reporting for dead controller units (#1152) since the last green run.
The Siena/Zen4c TEE host reports x86 microcode 0x0AA0021C (SPL 28) since its 2026-08-24 reboot, above rc11's Zen4c EntrySign floor of 25, so the --min-tcb microcode=21 --accept-outdated-tcb override the V-PROGRAM tests carried since 2026-08-17 is obsolete. Dropping it also makes the runs exercise the real TCB gate again instead of lowering every silicon line's floor (which is what the CLI's 'accepting a TCB below the network floor' warning was about).
Throwaway harness for the 2.0 release readiness call: install the 1.15.0 release deb, boot a QEMU instance with a disk marker and an extra port forward, upgrade the CRN in place to the dev branch's CI deb via the new crn-up.sh --upgrade, and assert the instance survived untouched: same controller unit main PID and start time, exactly one QEMU process (no double boot), same vm_index in the controller config, same rootfs inode, listed by the new supervisor/agent, no adoption errors in the journals, port forward kept, marker readable, stop/start still working and coming back on the same rootfs file. A second scenario flips ALEPH_VM_SUPERVISOR_IMPL python -> rust -> python on the upgraded node with the same checks. Ported from od/aleph-vm-upgrade-checks onto the validated dev-era stack (pyaleph rc9, scheduler vprogram-rc3, cli rc11), without the TEE host steps: the static SNP server is on mainnet and must not be touched.
…st poll lands on a serving node Run 33010889646: --register restarted the supervisor right after staking, the scheduler's node watcher polled during the restart window, marked the node Unreachable, and the first instance message stayed unscheduled for 5 min (no reschedule trigger on recovery), past the dispatch timeout. Set ALEPH_VM_NODE_HASH, restart, wait for :4020, then link. Scenario B now fails fast with a clear message when the node was never upgraded (no supervisor-launcher), instead of blaming the rust swap.
… live on the TEE server The TEE server joins as static confidential CRN index 1 (baseline 1.15.0 installed on it too). Scenario A now creates a plain instance and an SEV instance (init-session + measured start + secret injection), upgrades every CRN in one pass (UPGRADE_STATIC=1 opts the static server in), and asserts both survived untouched; the SEV VM additionally has to report SEV active and a dm-crypt root after the upgrade, and to come back on the same rootfs after stop, start, re-init-session and secret injection. Scenario B gains an SEV variant: python -> rust -> python swap with the confidential VM live, stop/start under the Rust daemon. Host commands honour the per-CRN ssh-user (the TEE server is non-root with passwordless sudo), like crn-up.sh.
…17.0 / 0.2.0 releases Until now only the confidential TEE server ran ALEPH_VM_SUPERVISOR_IMPL=rust; the DigitalOcean CRNs took the launcher's python default, so every non-confidential test (instances, programs, migration, backups, port forwards, IPv6) on this testnet still validated the Python supervisor. crn-up.sh now writes ALEPH_VM_SUPERVISOR_IMPL (default rust, overridable from the environment) into supervisor.env on every CRN. Version pins: pyaleph 0.11.0 (with its Kubo v0.43.0), aleph-cli 0.17.0, scheduler-rs/api 0.2.0 (V-PROGRAM scheduling, aleph-vm-scheduler#193), aleph-vm from od/drop-attest-cli (PR #1168; #1169 is stacked on it and has no deb build, its kernel change is not part of the deb anyway).
Kubo v0.43.0 refuses to start when the config still carries the pre-0.42
Reprovider.* keys ("deprecated configuration detected. Manually migrate
'Reprovider' fields to 'Provide'"), so pyaleph never got its IPFS daemon
and the CCN failed its readiness wait on the first run of #37.
Run 2 of #37 (33112723550) ran the whole suite on the Rust supervisor: 28/31 green including the SEV instance; the three failures were harness drift, not supervisor behaviour: - aleph-cli 0.17.0 made the V-PROGRAM name a mandatory positional (aleph-rs#361); both vprogram create calls pass one, ahead of the optional-valued --wait. - tests/test_vm_upgrade.py ran under pr-tests without GH_TOKEN and with the deleted `dev` branch as candidate. It is opt-in now: the module skips unless ALEPH_VM_UPGRADE_BRANCH is set, which only upgrade-check.yml does, and that workflow's candidate is `main`. - aleph-vm deb from `main`: #1168 merged, everything dev carried is on main now.
…0.2.1, aleph-vm 2.0.1 rc (#1183) Pins the four components of the 2.0.1 / 0.18.0 release stack: - pyaleph 0.11.2 (strict V-PROGRAM scalars, aleph-message 1.3.1) - aleph-cli 0.18.0 (verified volumes, register-carrying verification) - scheduler-rs/api 0.2.1 (lossy history fetches, SDK 0.18.0) - aleph-vm branch od/bump-aleph-message-131 = main + released aleph-message 1.3.1 (PR #1183), the 2.0.1 release candidate. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
GitHub Actions
Inactive
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
A well-engineered release-stack bump plus a new opt-in upgrade-check harness and V-PROGRAM end-to-end tests. The shell/Python/YAML is careful: fixtures are sha256-pinned and content-addressed, SSH quoting handles non-root sudo on the static TEE server, race conditions (register-before-link, scheduler poll-after-upgrade) are explicitly reasoned about and commented, and the new tests assert unusually deep invariants (controller PID, QEMU PID, rootfs inode, vm_index stability). I verified function-hoisting and set -e interactions in crn-up.sh, the pytest API usage (poll exception handling, pytest.skip.Exception, pytest-timeout dependency), cross-module imports, and script syntax. All findings are non-blocking robustness/diagnostics nits; no correctness or security blockers found.
scripts/crn-up.sh (line 811): Minor race: upgrade_ts is captured right before apt-get install, but the pre-upgrade daemon keeps serving for part of the install window, so a scheduler 200 logged during that window satisfies wait_for_scheduler_poll before the post-restart poll has happened. Capturing the timestamp after the apt-get install line would make the wait exact.
.github/workflows/pr-tests.yml (line 424): This loop greps tee-network.txt while the body appends to the same file, and the appended 'attestation DNAT rule:' lines also match the dnat.*:8443 pattern, so depending on grep's buffering the loop can re-process echoed lines and probe the same rule repeatedly. Snapshot the matches into a temp file first (e.g. grep -E "dnat.*:8443" tee-network.txt > /tmp/dnat.txt then loop over that) to keep the artifact clean.
tests/test_vm_upgrade.py (line 469): _create_confidential_instance leaks the VM if anything after instance create fails (placement wait, unlock): the caller's finally can't help because sev is still None at that point, so cleanup relies entirely on the end-of-run tee-reset. Consider extracting the hash right after create and deleting it in a try/except so failures leave a cleaner state and a re-run of the workflow doesn't inherit a stranded instance message.
.github/workflows/pr-tests.yml (line 344): Now that the job timeout is 75 minutes, a run whose test window starts more than 45 minutes before this step will have its early journal lines (the static-server reinstall window is presumably longer) clipped by the fixed '45 min ago'. Consider a wider window or anchoring on a timestamp captured before 'Run tests' to avoid missing the failure you are actually collecting these logs for.
.github/workflows/upgrade-check.yml (line 74): github.event.number is empty for push and workflow_dispatch events, so CCN/CRN droplet names render as 'testnets-pr--<run_id>' (double dash) in this workflow. Harmless since run_id is unique, but a plain prefix without the pr- part would be cleaner.
tests/test_vprograms.py (line 36): Neither this test nor test_vprogram_compose.py deletes its V-PROGRAM afterwards, so the compose VM (which runs first alphabetically) stays resident on the TEE server while the fib V-PROGRAM boots alongside it. The repo convention (see delete_instance in vm_helpers.py) is that every VM-creating test frees its capacity; consider a best-effort teardown so suite growth here doesn't eat TEE memory.
.github/workflows/pr-tests.yml (line 305): Stale comment: the step says fixtures come from the vprogram-fixtures release, but scripts/vprogram-artifacts.sh now downloads from mainnet native storage / IPFS (per its own header).
…ty volume)
Fixture bump: the 2026.08.31 "1.1" runtime build (aleph-vm c5391963,
PR #1176), the first with verified-volume support: the exec and compose
bundles gain the {verified_volumes} cmdline slot and the guest
/volumes/<i> verity mounts. Both bundles and manifests are published on
mainnet (exec native-storage, compose bundle on IPFS); sha256 pins
updated accordingly.
New test test_vprogram_compose_verified_volume: builds an ext4 volume
with mkfs.ext4 -d (nginx conf + marker.json), deploys a compose
V-PROGRAM with --volume, binds /volumes/0 subpaths into nginx and
asserts the marker comes back through the attested channel. The marker
only exists inside the verity volume, so a hit proves CLI verity
format/publish, guest verity-open + mount, and the podman bind half
(aleph-vm#1176 + aleph-rs#383) end to end.
Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
|
Second run adds what the first didn't cover: the 1.1 V-PROGRAM runtimes (2026.08.31 build from c5391963, first with verified-volume support) as fixtures, plus a new e2e test deploying a compose V-PROGRAM with |
GitHub Actions
Failure
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
A large, careful release-stack run PR: version bumps in manifesto.yml, a new in-place crn-up.sh --upgrade phase with a strict two-phase install/upgrade lifecycle, a new upgrade-check CI harness, prebuilt V-PROGRAM fixtures fetched with pinned sha256s, and three new e2e test modules (upgrade, V-PROGRAM exec, V-PROGRAM compose/verified volumes). The race-condition fixes in register_crn() (restart-before-link, post-upgrade scheduler-poll wait) are well-reasoned and documented, the new tests are appropriately fail-closed, and all shell/Python/YAML parses cleanly. No correctness or security blockers found; the comments below are robustness and hygiene nits in mostly-diagnostic code paths.
.github/workflows/pr-tests.yml (line 424): This loop appends lines into the very file grep is streaming, and the appended "attestation DNAT rule: ..." lines themselves match the dnat.*:8443 pattern. It only terminates because grep typically hits EOF before the first append (each iteration spends ~10s in curl probes). Grepping a snapshot first (e.g. mapfile -r rules < <(grep -E "dnat.*:8443" tee-network.txt)) would make this deterministic instead of racy.
.github/workflows/pr-tests.yml (line 344): The job timeout was raised to 75 minutes but the log window is still 45 min ago — on a slow run the TEE log capture can land outside the window and produce empty artifacts, which defeats the purpose of this step. Consider deriving the window from a start-of-run timestamp or bumping it along with the timeout.
scripts/vprogram-artifacts.sh (line 65): rm -f "$OUT_DIR/bundle-info.json" removes a file nothing in this repo writes (this is its only occurrence). If it's a leftover from an earlier revision of the script, it can go; if something is supposed to produce bundle-info.json, that writer is missing.
.github/workflows/upgrade-check.yml (line 11): The push trigger only fires on branch od/upgrade-check-1.15-to-dev, but this PR's head is od/run-201rc-stack, so this workflow never runs automatically as-is (only via manual dispatch selecting this branch). If the harness is meant to exercise this release candidate, either point the trigger at the actual head branch or note that dispatch is the intended invocation.
scripts/crn-up.sh (line 729): Copy-paste artifact: the # Destroy banner sits directly above the upgrade_crn() section (followed immediately by the # In-place upgrade banner), while the real destroy() further down has no banner. Confusing when navigating the file.
tests/test_vm_upgrade.py (line 356): _wait_forward_host_port duplicates the identically-named helper in tests/test_instance_lifecycle.py:57 verbatim. Moving it to vm_helpers.py (like poll/wait_for_dispatched) would keep the two from drifting.
tests/test_vm_upgrade.py (line 53): Minor doc nit: the comment says the candidate branch defaults to "dev" but the workflow passes CANDIDATE_BRANCH="main"; also the docstring at the top of the module says 'default "dev"'. Aligning the two would help future readers.
…#1184) The 2026.08.31 "1.1" runtime bundles never boot on real SEV-SNP: the minimal guest kernel (aleph-vm#1169) lacks CONFIG_X86_PAT (+ its MTRR dependency) and hangs before the first console line, which run 33502834019 exposed as all three V-PROGRAM tests timing out against VMs the 90s RA-TLS gate kept tearing down. New fixtures are built from aleph-vm od/snp-guest-kernel-pat (main + the fragment fix, aleph-vm#1184), SNP-boot-validated on the TEE server before publishing, and hosted on this repo's vprogram-fixtures-2 prerelease until the fixed runtimes are republished on mainnet. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
|
Run 2 (33502834019) failed: all three V-PROGRAM tests timed out. Root cause (bisected on the TEE server): the 2026.08.31 "1.1" runtime bundles never boot on real SEV-SNP — the minimal guest kernel (aleph-vm#1169) lacks Fix: aleph-vm#1184 (fragment + re-seeded goldens). Run 3 uses rebuilt fixtures from that branch, SNP-boot-validated on the TEE server and hosted on the vprogram-fixtures-2 prerelease. The mainnet-published 08-31 bundles are dead weight (never added to the vm-images aggregate); the fixed runtimes still need a mainnet republish + aggregate update once #1184 merges. |
GitHub Actions
Failure
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Solid, well-documented release-stack pin bump with high-quality new test coverage. Cross-references all check out (vm_helpers exports, conftest fixtures, CLI conventions in test_instance_lifecycle.py, pytest-timeout in pyproject), shell scripts pass bash -n, the register_crn reordering (restart → wait for API → link) is a thoughtful race fix, and the vprogram fixtures are sha256-pinned. No security problems. The one probable genuine defect is in the new TEE network diagnostics step: the tap interface is extracted with a grep for 'iface vmtap' which none of the captured commands can produce, so the dual tcpdump capture is likely dead code. Remaining items are nits and questions.
.github/workflows/pr-tests.yml (line 404): This tap extraction likely never matches: at this point tee-network.txt contains nft ruleset, ip addr (renders N: vmtap0: <...>), ip route / ip -6 route (render dev vmtap0), ss -tlnp, systemctl status ndppd and cat /etc/ndppd.conf — none of which emit the literal iface vmtap<N>. That means [ -n "$uplink" ] && [ -n "$tap" ] is false and the whole dual tcpdump capture (the most valuable part of this step) silently never runs. Did a real run ever populate $tap here? Something like grep -oE 'dev vmtap[0-9]+' from the ip -6 route output, or adding ip -o link show to the capture, would match reliably.
.github/workflows/upgrade-check.yml (line 11): The push trigger lists od/upgrade-check-1.15-to-dev but this PR's head branch is od/run-201rc-stack, so the workflow won't run on this branch, and once merged to main it becomes manual-dispatch-only. If the throwaway-harness framing makes that intentional, consider a short comment saying so (and note the workflow name says '-> dev' while CANDIDATE_BRANCH is main).
scripts/crn-up.sh (line 793): The branch-deb cache path is keyed only by distro variant, not by target branch, so a second --upgrade against a different branch in the same run would silently reuse the stale cached deb (the if [ ! -f "$local_deb" ] check would pass). Harmless for the current single-upgrade flow, but encoding the branch (or its hash) in the filename would make it robust.
scripts/crn-up.sh (line 729): The In-place upgrade section was inserted between the Destroy banner and destroy(), leaving a duplicated ---- separator and an orphaned 'Destroy' header here; destroy() at the bottom now has no banner at all. Cosmetic only.
.github/workflows/upgrade-check.yml (line 194): Minor (copied from pr-tests.yml): wait $CCN_PID "${CRN_PIDS[@]}" returns the exit status of the last pid, so a failed CCN droplet creation is silently masked when CRN creation succeeds — the run then dies later at 'Get Droplet IPs' with a less clear error. A set -e-friendly loop or checking each wait's status would surface the real failure.
tests/test_vm_upgrade.py (line 356): _wait_forward_host_port is a verbatim copy of the one in tests/test_instance_lifecycle.py:57 — vm_helpers.py (which both files already import from) is the natural shared home.
.github/workflows/pr-tests.yml (line 277): The Scaleway flexible-IP /64 is hardcoded in both workflows (here and upgrade-check.yml:275). Since it's provider-config tied to the static TEE server (like DO_SSH_KEY_FINGERPRINT, which already comes from vars.), a repo variable would avoid the two copies drifting apart.
scripts/vprogram-artifacts.sh (line 41): ALEPH_IPFS_URL is declared but never used — dead variable (the comment about the compose bundle's CID mentions IPFS, but the actual download goes through $FIXTURES_URL).
tests/conftest.py (line 543): Tiny doc nit: this says the compose bundle is 297 MB while scripts/vprogram-artifacts.sh says 268 MB.
tee-reset.sh kills QEMU and wipes aleph-vm state but left the tap devices behind (NO-CARRIER, pool addresses still configured). The next run's supervisor restarts its slice allocator and re-assigns the same IPv6 /124 to a fresh tap, so two interfaces own the prefix and the kernel keeps routing the guest's IPv6 into the dead tap (linkdown routes are used by default): run 33512021462 failed only the test_vprograms IPv6 assertion, with `ip -6 route get <guest-v6>` resolving to the stale vmtap8. IPv4 never collides because its pool index keeps advancing. Also stop the lingering aleph-vm-dhcp-* dnsmasq units that hold the taps. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
|
Run 3 (33512021462): the SNP kernel fix works — 28 passed including the first-ever green Single failure: Possible aleph-vm hardening follow-up: supervisor startup GC of vmtap links no controller config owns. |
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
A large but carefully engineered change: the two-phase install/upgrade flow in crn-up.sh, the restart-before-link fix in register_crn, tee-reset hardening (controller units, DHCP units, leftover vmtaps), and the three new test modules are all well-reasoned and well-documented, with excellent run-ID-anchored comments explaining non-obvious workarounds. No blocking correctness or security issues found in the tested paths; secrets handling, shell quoting, and subprocess usage all look sound. The problems I found are confined to the best-effort 'Collect TEE network diagnostics' step in pr-tests.yml (a tap-detection regex that never matches, making the dual tcpdump capture dead code, plus a grep/append feedback loop over the same file) and a few minor nits. None of these block the release-stack pin changes in manifesto.yml.
.github/workflows/pr-tests.yml (line 404): This tap-detection regex can never match anything in tee-network.txt: ip addr output uses the N: vmtapX: <...> format, and none of the other captured outputs (nft ruleset, ip route, ss, ndppd status) emit the literal iface vmtapN. tap is therefore always empty, the [ -n "$uplink" ] && [ -n "$tap" ] guard always fails, and the whole dual tcpdump capture below is dead code. To match the tap from the ip addr capture, a pattern like : vmtap[0-9]+: (stripping the trailing colon) would work; as written the 'dual capture' diagnostics the comments describe never execute.
.github/workflows/pr-tests.yml (line 424): This loop appends to tee-network.txt while grep is streaming it as input, and every appended line (attestation DNAT rule: <rule>) itself matches dnat.*:8443, so grep may re-read the file's new tail and re-process its own output (duplicate probes, or worse if the file grows faster than grep consumes). Safer to snapshot first (e.g. grep -E "dnat.*:8443" tee-network.txt > /tmp/dnat.txt before the loop, or write the echo'd rule headers to a separate file). Best-effort diagnostics so not blocking, but the loop is fragile.
scripts/crn-up.sh (line 665): The readiness gate after the supervisor restart polls /about/usage/system, but on the split-package (2.0) deb this endpoint is served by the agent and can answer even if the agent's gRPC channel to the just-restarted supervisor is broken — exactly the gap _set_supervisor_impl in tests/test_vm_upgrade.py works around by waiting on /about/executions/list (a supervisor-backed request). Consider probing a supervisor-backed endpoint here (and in the equivalent wait in upgrade_crn, line 822) so 'stake a node that is serving' actually proves the full agent→supervisor path.
scripts/crn-up.sh (line 849): Minor: when .local/crn exists but every state dir is skipped (static without UPGRADE_STATIC, or missing droplet-ip), this prints '==> All CRNs upgraded' and exits 0, so --upgrade can succeed having upgraded nothing — the test would then only fail later with a confusing 'the upgrade did not change the installed package' error. Tracking the skip count alongside failures and erroring when zero CRNs were upgraded would make the failure mode clearer.
tests/test_vprogram_compose.py (line 221): message is assigned but only item_hash is used; item_hash = _vprogram_message(objs)["item_hash"] would suffice (as done in the whoami test above).
scripts/vprogram-artifacts.sh (line 41): ALEPH_IPFS_URL is defined but never used (all sources use the GitHub release or Aleph storage). Drop it or note why it's kept.
|
Run 4 (33516936849) is fully green: the tee-reset vmtap cleanup cured the IPv6 blackhole, and the whole suite passes on the release stack — pyaleph 0.11.2 / aleph-cli 0.18.0 / scheduler 0.2.1 / aleph-vm 2.0.1 rc (#1183) — with the fixed 2026.09.01 runtimes (aleph-vm#1184), including SNP V-PROGRAM exec + compose, the new verified-volume e2e, and the IPv6 assertion. Remaining before this PR's pins can point at final artifacts: merge aleph-vm#1184, republish the fixed runtimes on mainnet (upload-runtimes.sh 2026.09.01 + update-aggregate.sh), then swap the vprogram-fixtures-2 prerelease URLs back to mainnet storage pins. |
2 participants
Full-suite run of the release stack:
od/bump-aleph-message-131(aleph-vm#1183): main as of 2026-09-01 (#1178/#1179/#1180/#1182) + released aleph-message 1.3.1Based on #37 (Rust supervisor on every CRN); only the manifesto pins change.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5