test: validate aleph-vm dev-accelerate (supervisor + migration) on testnet

[odesenfans]

Why

Continuous validation of aleph-vm dev on a live testnet: two CRNs in true two-process gRPC split mode, exercising the cross-CRN migration flow. Currently validating the post-#1010 tree: the reattach-hardening round (#1001-#1008, background retry with bounded give-up, on-demand re-adoption) and the per-actor DB split (#1009/#1010, supervisor-owned supervisor.sqlite3 for port mappings with a one-time copy migration, keep_port_mappings on delete+recreate cycles).

What

Pins manifesto.yml -> components.aleph-vm to branch: "dev". crn-up.sh fetches aleph-vm.debian-12.deb from the latest successful dev build of build-deb-package-and-integration-tests.yml and sets ALEPH_VM_SUPERVISOR_GRPC_SOCKET, so the agent dials the supervisor daemon over gRPC instead of embedding its own pool.

What this exercises

tests/test_migration.py: create instance -> write marker over SSH -> unlink the hosting CRN -> wait for scheduler reallocation -> verify the marker survived on the new CRN (disk state preserved across migration). With the DB split, this also covers port-forward persistence through the supervisor DB and the legacy copy migration on first start.

Revert

Temporary pin for validation. Revert to a tagged release once the next aleph-vm release cut from dev ships.

[foxpatch-aleph]

Cleanly pins aleph-vm to the dev-accelerate branch for pre-merge live migration validation. The branch-based deployment path was already wired into crn-up.sh — this is a simple config toggle. The diff is minimal, well-commented, and correctly hands reverting to version: "1.13.0" once alpeh-vm#977 lands. No correctness, security, or quality issues.

manifesto.yml (line 35): The comment clearly flags this as temporary and explains what to revert to. Good practice.

[foxpatch-aleph]

Clean, minimal configuration PR that pins aleph-vm to the dev-accelerate branch for pre-merge validation of the gRPC supervisor rework and directory-based VM migration. Also bumps the scheduler components to 0.1.2 for graceful-stop tolerance. Both changes are well-documented with clear rationale and revert instructions. No code changes, no logic errors, no security concerns.

[foxpatch-aleph]

Configuration-only PR: bumps scheduler-rs/scheduler-api to 0.1.2 (fixes the graceful-stop race where a VM in STOPPED state was reported as "missing") and switches aleph-vm from version pin "1.13.0" to branch "dev" (tracks the merged #977 gRPC supervisor + directory-migration work). The comments are thorough, the revert instructions are clear, and the deployment script's branch/version branching logic handles this correctly.

[foxpatch-aleph]

Adds migration test with VLAN-aware VM scheduling and anaconda bootstrap. Core logic is sound but has a VM fault tolerance gap (no recovery on start failure), a port collision risk for concurrent tests, and a fragile Bash CIDR computation. These match the existing codebase risk profile and do not block for a testnet-facing PR.

aleph/vm/vm_pool.py (line 2265): switch_network: if cloned.start() fails, original is already stopped and released. No rollback path. Keep original alive until cloned.start() succeeds, then stop it.

tests/migration/config.py (line 76): Fixed port 4020 for Postgres forwarding collides under concurrent test runs. Use port range or random_port().

deploy-hypervisor.sh (line 483): CIDR extraction awk script fails silently on multi-IP VMs. Add fallback or grep for scope global and guard with set -e.

[foxpatch-aleph]

The manifest version changes are correct and well-documented, but the PR description omits nearly all of the test/ changes: removal of the confidential VM test, removal of the scheduler-observation gate (wait_for_scheduler_observed), removal of CRN re-link in migration finally block, removal of required_port=None from wait_for_dispatched, and a downgrade of aleph-cli from 0.11.1 to 0.11.0. The PR title/description should reflect the full scope of these changes.

manifesto.yml (line 41): nit: the comment references aleph-vm#980 but the branch name says "phase2". Is this the right branch? The PR description mentions "dev-accelerate" but the actual branch is "od/grpc-only-supervisor-phase2". One of these is out of date.

manifesto.yml (line 15): aleph-cli downgraded from 0.11.1 to 0.11.0 without explanation in the PR description or comments. Is this intentional? If the newer CLI is incompatible with the test branch, please document. If this was accidental (leftover from a rebase), it should be reverted.

tests/test_migration.py (line 42): Why remove wait_for_scheduler_observed? The old comment explained a real race: if the unlink happens before the scheduler's node_watcher poll sees the VM on the source CRN, the reschedule cold-starts the VM instead of migrating it — losing the marker. With scheduler 0.1.2's v2 executions endpoint, is this race fully eliminated, or does this make the test flaky by removing the gate? The commit message should justify this removal.

tests/test_migration.py (line 70): Removed the CRN re-link from the finally block. Without it, if this test unl inks the static TEE CRN, the confidential test will fail because the CRN is no longer linked. If the confidential test was removed intentionally (see below), then this is fine, but the re-link removal should still be documented as it's a non-obvious change in shared-state cleanup behavior.

tests/test_confidential.py (line 1): The entire confidential VM test (205 lines+) is deleted but not mentioned in the PR descri ption at all. There's no explanation in the commit message, title, or comments. If this is intentional (e.g., simplifying CI to focus on migration), it needs to be acknowledged. If it was removed accidentally (e.g., the branch was cut before the confidential test was added), this is a regression and should be restored.

[foxpatch-aleph]

The diff correctly switches aleph-vm from a tagged release to a CI artifact build from a feature branch. The crn-up.sh script properly handles both code paths, and the migration test is well-designed with correct handling of the scheduler race condition. However, the PR description says the branch is 'dev-accelerate' while the actual code pins 'od/grpc-only-supervisor-phase2' — these should be aligned before merging.

manifesto.yml (line 39): PR description references branch "dev-accelerate" but the code pins "od/grpc-only-supervisor-phase2". These should match. Either update the PR description to reflect the actual branch name, or update the manifesto.yml if the branch was renamed.

[foxpatch-aleph]

The code changes are correct and internally consistent: standardizing all droplets to Ubuntu 24.04, updating the CI artifact name accordingly, and pinning to the development branch for testing. The only issue is a mismatch between the PR description (which says branch: "dev-accelerate") and the actual change (branch: "od/grpc-only-supervisor-phase2"). This should be reconciled before merging.

manifesto.yml (line 39): PR description says this should be "dev-accelerate" but the code sets "od/grpc-only-supervisor-phase2". Please update one or the other to match."

[foxpatch-aleph]

Sound PR that migrates from Debian 12 to Ubuntu 24.04 across all CRNs/CCN, fixes the TEE server log collection to use --since instead of -n (addressing the crash-loop noise problem), and captures the new agent service logs. The distro unification simplifies the artifact download path to a single ubuntu-24.04 artifact. No correctness or security issues found — the SSH quoting in the --since parameter is handled correctly via double-quote expansion.

.github/workflows/pr-tests.yml (line 330): Consider using a wider --since window (e.g. '90 min ago') or capturing a start timestamp earlier in the workflow. The 45 min window covers most but not all failure scenarios — a test that fails after minute 45+ of a 60-min timeout workflow could have its tail end of logs cut off. Low risk in practice given the log collection step itself is fast.

manifesto.yml (line 39): PR description says 'dev-accelerate' but the branch pinned here is 'od/grpc-only-supervisor-phase2'. Verify these refer to the same branch (e.g. a rename) or update the PR description to match.

[foxpatch-aleph]

Switches testnet infrastructure from Debian 12 to Ubuntu 24.04 across all droplet types (CCN, CRNs), pins aleph-vm to the dev-accelerate branch for pre-merge validation, improves journalctl log capture to use --since instead of -n to avoid budget issues from supervisor crash-loops, and adds collection of the new aleph-vm-agent.service logs. The migration test properly re-links the unlinked CRN in its cleanup. Coherent, well-documented changes with no correctness or security issues.

scripts/crn-up.sh (line 111): Artifact name hardcoded to ubuntu-24.04 variant — correct now since all hosts use that distro, but brittle if the CI pipeline renames the artifact. Acceptable for a temporary branch pin.

.github/workflows/pr-tests.yml (line 330): Hardcoded --since "45 min ago" works with the 60-min timeout but may miss the window if the workflow runs long. Best-effort collection with || true so acceptable — just flagging the implicit assumption.

manifesto.yml (line 36): The comment still references version 1.13.0's EIP-191 features. Not incorrect, but when reverting to version: "1.13.0", the developer must restore this comment block (it's currently above the branch field). Consider keeping the comment above the version line so it survives the toggle.

[foxpatch-aleph]

The PR contains well-reasoned changes (debian-12→ubuntu-24.04 migration, improved TEE log collection, --allow-downgrades for static CRN reinstalls), but manifesto.yml pins to branch "od/supervisor-vmid-identity" while the PR description explicitly says it should pin to "dev-accelerate" — this mismatch needs to be resolved before merging, as it determines which aleph-vm code is actually tested.

manifesto.yml (line 39): Branch name mismatch: the PR description says branch: "dev-accelerate" but the code pins to branch: "od/supervisor-vmid-identity". These are different branches testing different things. Please confirm which branch is intended and update either the code or the PR description.

scripts/crn-up.sh (line 447): Stale comment: says "a static TEE server may not run debian-12 like the DO droplets do" — but the DO droplets now also run Ubuntu 24.04 (changed in this PR). The debian-12 reference is outdated. Consider: "a static TEE server may run a different distro than the DO droplets".

scripts/crn-up.sh (line 75): With version replaced by branch in manifesto.yml, m['components']['aleph-vm']['version'] will KeyError if ever called without ALEPH_VM_VERSION set. Not currently reachable (install_crn only calls this when branch is empty), but consider using m['components']['aleph-vm'].get('version', '') for robustness.

[foxpatch-aleph]

This PR is a well-executed testnet validation pin: it migrates all CRN droplets from Debian 12 to Ubuntu 24.04 (consistently across the CI workflow, crn-up.sh provisioning, CI artifact name, and release variant fallback), pins aleph-vm to the dev branch to exercise the two-process gRPC split via ALEPH_VM_SUPERVISOR_GRPC_SOCKET, and improves TEE log collection by switching from fixed -n line counts to --since time-based filtering while adding the agent service log. The changes are internally consistent, correctly quoted, and well-documented. The only minor concern (read_vm_version KeyError if called without a version key in manifesto.yml) is unreachable in the current control flow. This is a temporary pin explicitly meant to be reverted to a tagged release, and the changes are appropriate for that purpose.

scripts/crn-up.sh (line 75): Minor (non-blocking): read_vm_version() does m['components']['aleph-vm']['version'] which raises KeyError when manifesto.yml has branch but no version key and ALEPH_VM_VERSION is unset. This is currently unreachable (install_crn only calls it in the else-branch when branch is empty), but a defensive .get('version', '') would make the function safe against future callers.

.github/workflows/pr-tests.yml (line 330): The hardcoded since="45 min ago" assumes the test window starts within the last 45 minutes of the run. Given the 60-minute workflow timeout and that the TEE log collection runs in the always() block, this should be sufficient in practice, but if the provisioning phase runs long the relevant test-window logs could be truncated. Not a blocker for a temporary validation pin.

[foxpatch-aleph]

The PR is well-structured and the core changes (distro migration to ubuntu-24.04, gRPC socket config, TEE log collection improvements) are correct and consistent. The code path where read_vm_version() would KeyError on the new branch-based manifesto is safely unreachable. Two stale comments about debian-12/glibc remain in files that were partially updated by this PR — worth fixing for accuracy but non-blocking.

scripts/crn-up.sh (line 453): Stale comment: "a static TEE server may not run debian-12 like the DO droplets do" — all CRNs now run ubuntu-24.04, and the default at line 456 was updated accordingly. The comment should reflect the new reality (e.g. the static TEE server's distro is auto-detected, defaulting to ubuntu-24.04).

scripts/confidential-artifacts.sh (line 23): Stale comment: "Prefer a debian-12 CRN: its sevctl is built against the oldest glibc" — the pr-tests.yml comment (line 283-285) was updated to say all CRNs and the TEE server now run Ubuntu 24.04, but this companion comment was not. The glibc-baseline rationale no longer applies since all machines share the same distro.

[foxpatch-aleph]

Solid testnet configuration PR that migrates all CRN droplets from debian-12 to ubuntu-24.04, pins aleph-vm to the dev branch to exercise the two-process gRPC split, and improves TEE log collection with --since instead of -n. The changes are internally consistent across pr-tests.yml, crn-up.sh, and manifesto.yml. No correctness or security issues. Two stale debian-12 comments remain in files not touched by this PR (confidential-artifacts.sh and the inline comment in crn-up.sh at the deb_variant fallback), but these are cosmetic documentation drift, not blocking.

scripts/confidential-artifacts.sh (line 22): Stale comment: references "debian-12 CRN" and the glibc baseline rationale for SEVCTL_HOST, but all CRNs and the TEE server now run ubuntu-24.04 (per the pr-tests.yml change). The pr-tests.yml comment was updated but this file was not — consider updating lines 22-26 to match the new reality.

scripts/crn-up.sh (line 452): Stale comment: "a static TEE server may not run debian-12 like the DO droplets do" — with this PR all hosts run ubuntu-24.04, so the debian-12 reference is outdated. The dynamic detection logic (${ID}-${VERSION_ID}) is still correct and worth keeping for robustness, but the comment should reference ubuntu-24.04 instead.

[foxpatch-aleph]

This PR correctly pins aleph-vm to the dev branch for testnet validation of the gRPC two-process split and migration flow, migrates all droplets from debian-12 to ubuntu-24.04 consistently across CI and local provisioning, and improves TEE log collection with --since timing and agent log capture. The shell quoting in the refactored SSH/sudo pattern is correct for both root and non-root cases, and the read_vm_version function safely avoids a KeyError by only being called when branch is unset. There are a few minor documentation inconsistencies (stale debian-12 references in confidential-artifacts.sh and crn-up.sh comments) and the dynamic CRN log collection doesn't capture agent logs like the TEE server now does, but none of these are blocking.

scripts/confidential-artifacts.sh (line 23): This SEVCTL_HOST docstring still references the old debian-12 glibc-baseline rationale. Now that all hosts run Ubuntu 24.04, this comment is stale — the sevctl from any CRN and the TEE server share the same glibc baseline. Consider updating to reflect the new homogeneous distro reality.

scripts/crn-up.sh (line 453): The comment "a static TEE server may not run debian-12 like the DO droplets do" is outdated now that all CRNs and the TEE server run Ubuntu 24.04. Consider updating to reflect the current state.

.github/workflows/pr-tests.yml (line 314): Dynamic CRN log collection only captures supervisor logs, but the TEE server collection now also captures aleph-vm-agent.service. In the two-process gRPC split, the agent runs on all CRNs — adding ssh root@${ip} "journalctl -u aleph-vm-agent.service --no-pager -n 500" > "crn${idx}-agent.txt" 2>&1 || true here would give parity and help debug migration failures on dynamic CRNs.

scripts/tee-reset.sh (line 28): Only aleph-vm-supervisor.service is stopped here. In the two-process gRPC split, the agent (aleph-vm-agent.service) runs as a separate unit. If the .deb's systemd units don't have a BindsTo/PartOf dependency that cascades the stop, leftover agent processes could hold the gRPC socket open during the rm -rf /var/lib/aleph/vm/* on line 36. Worth confirming the unit dependency handles this.

[foxpatch-aleph]

This PR temporarily pins aleph-vm to the dev branch to validate the two-process gRPC split (supervisor + agent) and cross-CRN migration on the testnet. The changes are well-scoped: manifesto.yml switches from version to branch (already supported by crn-up.sh's branch-fetching logic), the gRPC socket env var is added to the shared supervisor.env with a clear explanatory comment, and the CI log collection is improved to use --since instead of -n (avoiding crash-loop noise drowning the test window) while adding agent log capture for the TEE server. Shell quoting and variable expansion in the refactored SSH/SUDO block are correct. One minor asymmetry: the Collect CRN logs step (line 310-316) does not capture aleph-vm-agent.service logs from the DO droplets, even though those CRNs now also run the two-service split — agent logs there could help debug migration failures on non-TEE CRNs.

.github/workflows/pr-tests.yml (line 314): The "Collect CRN logs" step only collects aleph-vm-supervisor.service and docker logs vm-connector from CRN droplets, but with the two-service gRPC split the agent now runs as a separate aleph-vm-agent.service on those CRNs too. The TEE server log collection was updated to also capture the agent (line 333), but the CRN droplet log collection was not. Consider adding ssh root@${ip} "journalctl -u aleph-vm-agent.service --no-pager -n 500" > "crn${idx}-agent.txt" for parity, and add crn*-agent.txt to the artifact upload list.