Add netsim scenarios#352
Open
Rekseto wants to merge 57 commits into
Open
Conversation
Install the astral-agent skill into the Qwen Code operator. The netsim host owns a deploy key (SATFORGE_SKILLS_DEPLOY_KEY); run.sh injects it into the VM, which clones the private satforgedev/skills repo, builds the satforge-skills linker (Go already present from install-astrald), and runs `link astral-agent --target qwen` -> ~/.qwen/skills/astral-agent. Folded into lab.story after install-qwen-code; documented in the task README (one-time deploy-key setup) and netsim/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The astral-agent skill is installed as a tree of symlinks, so the verify file count used `find`, which does not traverse symlinked directories and undercounted. Use `find -L` to follow symlinks (and silence transient errors) so the count reflects the materialized tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thin-prompt, skill-driven swarm task: a two-sentence prompt tells the in-VM Qwen operator to make node1 a User-controlled node by following its astral-agent skill's node-setup playbook (software-User path), without restating the procedure. run.sh base64-ships the prompt over one `netsim ssh` argv and runs `qwen -y` as tester; verify.sh independently reads the persisted User token and asserts apphost.whoami = User id and user.info returns the active contract. Standalone (not in lab.story): `netsim task --stage astrald-lab --save astrald-user bootstrap-user`. Validated end-to-end on a live astrald-lab. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thin-prompt, skill-driven swarm task chained onto bootstrap-user: a two-sentence prompt drives the Qwen operator to claim node2 into the User's swarm via its astral-agent skill's node-claiming playbook (`user.claim`, with nearby handling reachability). verify.sh is an independent both-ends check -- both nodes hold a contract from the same User, node1 lists node2 as a Linked sibling, and a mutual link exists -- parsing the astral-query JSON object-stream line-by-line. Standalone: `netsim task --stage astrald-user --save astrald-swarm link-swarm`. Validated end-to-end (two nodes in one User Swarm). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First scenario past swarm formation: store an astral object on node1 and prove sibling node2 can obtain it by Object ID across the swarm. The thin prompt drives the Qwen operator (acting as its User) to objects.store a text payload and record the id; the cross-swarm fetch lives in verify.sh, not the prompt, which from node2 tries a ladder -- explicit-target <node1>:objects.load, transparent objects.load, then objects.find -- and asserts the bytes match, distinguishing a routing failure from an auth rejection. Standalone: `netsim task --stage astrald-swarm --save astrald-shared share-object`. Drafted; not yet run end-to-end (the cross-swarm read hop is inferred from the docs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post-install probe only waited ~10s for astrald to come up, but a fresh
astrald's first start (node-key generation + SQLite init), right after a
CPU-heavy go build still loads the VM, can take longer -- it flaked on an
otherwise-clean lab build ("astrald did not come up"). Wait up to ~90s, and
on failure dump `systemctl status` + `journalctl -u astrald` so the message
is a real diagnosis instead of opaque. Validated: the lab build passed with
the wider window on both nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ect) The link-swarm and share-object verifiers were shell scripts that gathered astral-query JSON and parsed it with embedded python heredocs -- awkward, and the parsing couldn't be unit-tested without booting a VM. Move all logic into a real verify.py per task (calls `netsim ssh ... astral-query` via subprocess, parses the JSON streams, asserts); verify.sh becomes a thin shim: exec python3 "$NETSIM_TASK_DIR/verify.py" "$@" netsim sets $NETSIM_TASK_DIR to the task dir and only auto-runs run.sh/verify.sh, so verify.py sits alongside and is found cleanly. Behavior-preserving: parsers golden-file tested against captured JSON, and the full pipeline re-run fresh on NFS -- link-swarm verify PASSES, share-object verify reproduces the cross-swarm-fetch diagnostic, both via the new shim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename swarm wording to match astrald master (PR #350) and the updated astral-agent skill: user.claim -> user.adopt, the node-claiming playbook -> node-adoption, and mod.user.swarm_access_action -> mod.user.swarm_membership_action. Docs/comments/prompt wording only; no verifier logic changes.
Add a fourth verifier check: node2 must list node1 as a Linked sibling (user.swarm_status, which derives from node2's own active contract, so no token). This is a direct regression guard for astrald #348 (roster sync to a newly adopted node) and the precondition share-object's write direction relies on. Offline golden test: post-#348 passes, pre-#348 (roster={node2}) correctly fails.
Pivot from the (blocked) cross-swarm read to the now-unblocked write direction: the agent stores an object ON node2 (<node2>:objects.store) and reads it back; verify.py independently proves node2 physically holds it via repo-pinned, ungated objects.load/contains -repo local. Unblocked by #348 (node2 now recognizes node1 -> AuthorizeRelayFor permits the relayed store, which reaches the ungated op_store). Caveat documented: op-level write is unauthenticated (CreateObjectAction still unwired).
Move lab.story into netsim/stories/ and add one story per tested flow (bootstrap-user, link-swarm, share-object), each a thin task list with a start/save stage header so a story doubles as a pass/fail integration test. Refresh netsim/README.md (full task list, swarm pipeline via stories) and reconcile running-as-a-service.md snapshot guidance (disk image: stop; live RAM snapshot: leave running).
The link-swarm and share-object verifiers carried long module docstrings restating rationale already in their README.md. Cut to a one/two-line summary; no logic change.
Cut the per-task READMEs down to a short paragraph (what the task does + the stage it produces); dropped the execution-model, build-facts, verify-internals, deploy-key setup, and security-note sections. No behavior change.
Replace the scattered ~/.netsim/{user.id,user.token,object.*} files with a
single $HOME/info.json (/home/tester/info.json) holding user_id, user_token,
object_id, object_payload, object_readback, object_target. bootstrap-user writes
user_*; share-object merges object_* (keeping user_*); verifiers and smoke-checks
read the JSON (python3 in-VM for shell, host-side json for verify.py). Transient
prompt/log files stay under ~/.netsim.
A drop-in alternative to bootstrap-user: instead of minting fresh entropy, the agent derives the User key from a provided BIP-39 mnemonic (ASTRAL_USER_MNEMONIC) and installs node1's active contract under that existing software User. verify.sh asserts node1 is a User node and, if ASTRAL_USER_ID is set, that the derived id matches exactly (proof the existing key was used). Produces stage astrald-user.
Rename the two first-node User-setup tasks to spell out the key variant: bootstrap-user -> bootstrap-user-software-key (new soft key), import-user -> import-user-software-key (existing soft key, known mnemonic). Renames the task dirs + story files and updates every reference (internal messages, prompt/log basenames, cross-references in link-swarm/share-object, README layout/pipeline). Leaves room for hardware-key variants later.
The previous commit swept in py_compile byte-cache via 'git add -A'. Remove the .pyc artifacts and add netsim/.gitignore for __pycache__/*.pyc.
Bake a valid BIP-39 mnemonic (the canonical all-zero-entropy test vector) into prompt.md instead of an __MNEMONIC__ placeholder; the task is now self-contained and reproducible. run.sh ships prompt.md verbatim (drops the ASTRAL_USER_MNEMONIC requirement and sed substitution). verify.sh's optional ASTRAL_USER_ID assertion is unchanged.
Skills moved off GitHub to ssh://git@git.satforge.dev/satforge/skills.git. Update the default SATFORGE_SKILLS_REPO and the comments/README; drop the GitHub-specific 443 fallback note. Host-key handling (StrictHostKeyChecking= accept-new) already covers the new host, and the deploy-key flow is unchanged (the key must now be registered on git.satforge.dev).
Rename the second-node task to match the swarm vocabulary (user.adopt): link-swarm -> adopt-node. Renames the task dir + story file and updates every reference (internal messages, prompt/log basenames, README layout/pipeline, share-object cross-reference). Stage names unchanged (astrald-user -> astrald-swarm).
…gle-node Both bootstrap-user-software-key and import-user-software-key produce the same single-node stage (a node set up as a User; they differ only in the User key — random vs the embedded mnemonic). Name it astrald-single-node and point adopt-node at it. Stage-name change only (set via --save/--stage); no script logic depends on it.
Split the object lifecycle into two focused scenarios and drop the combined (write-direction) share-object: - object-store (0006): node1 stores an object in its OWN local repo and reads it back; agent-driven, verify re-loads -repo local. astrald-swarm -> astrald-stored. - read-remote-object (0007): node2 reads node1's stored object OVER ASTRAL; host-driven (node2 has no operator), verify runs the <node1>:objects.load ladder and asserts the bytes. astrald-stored -> astrald-read. This is the peer-reads-node1 direction that failed pre-#348 — re-probed on current master. Rewire README pipeline accordingly.
astrald-single-node -> one-node, astrald-swarm -> two-nodes, astrald-stored -> two-nodes-data, astrald-read -> two-nodes-data-read. Stage-name change only (set via --save/--stage in story headers + docs); no script logic depends on it. astrald-lab kept as the base build fixture.
object-store now stores either in node1's own local repo (--target self, default) or on the sibling node2 (--target peer, via <node2>:objects.store) — so one story tests local storage and another tests storing on a peer. run.sh selects the prompt (prompt.md / prompt-peer.md); verify.py checks the holder's local repo (node1 for self, node2 for peer). Adds object-store-peer.story (two-nodes -> two-nodes-data-peer); object-store.story stays self -> two-nodes-data and feeds read-remote-object.
…ters node aliases Replace object-store's abstract --target self|peer with a real astral query target (--target, default localnode; e.g. node2). One prompt template (drops prompt-peer.md): the agent stores on / reads back from <target>, forming the right query itself. verify.py maps target -> holder (localnode/node1 -> node1, node2 -> node2). adopt-node now registers node1/node2 directory aliases (dir.set_alias) on both nodes when the swarm forms, so tasks can address nodes by name. Also fixes adopt-node's stale soft-check (read the User token from info.json, not the removed user.token). object-store-peer.story now passes --target node2.
Strip astral-agent/playbook/skill-location references and harness meta ('the skill
won't mention this'); the operator already has the skill auto-loaded. Prompts now
read like a person's request (still naming astral/astrald), keeping only the task
plus a terse 'save results to ~/info.json' the automated check needs.
The User on node1 permanently bans node2 from the swarm via user.expel, driven by the Qwen operator through its astral-agent skill. verify.py confirms the ban from both ends: node2 lands in user.list_expelled, drops out of user.swarm_status (OpSwarmStatus lists ActiveNodes, which filters the expelledSet), and the node1<->node2 link is torn down. README registers the new task/story and the two-nodes -> two-nodes-expel branch.
The old read-remote-object was host-driven and read node2->node1 anonymously, which can't route (network zone stripped) -- it tested the wrong, unroutable direction. Now it's agent-driven on node1: the agent reads the object (id from ~/info.json, written by object-store --target node2) FROM the peer as the User -- the authenticated, routable direction -- and records what it read. verify.py independently re-reads <peer>:objects.load as the User and asserts the bytes. New read-remote-peer.story chains object-store --target node2 (store on the peer) then read-remote-object (read it back from node1). Drops the old read-remote-object.story; README pipeline updated.
Minimizing the prompts dropped the 'keep existing keys' hint, so object-store's agent overwrote ~/info.json with object_* and wiped the user_token bootstrap wrote -- breaking read-remote-object's verify (which reads the peer as the User). Restore a natural 'leaving the existing entries in place' instruction in object-store and read-remote-object.
Each task writes its own file (no shared accumulator, no merge, no clobbering): - bootstrap/import -> ~/user.json (user_id, user_token) - object-store -> ~/object.json (object_id, object_payload, object_readback) - read-remote-object -> ~/read.json (object_remote) Readers reference the specific file(s) they need: adopt-node + expel-node read user.json; object-store verify reads object.json; read-remote-object verify reads user.json + object.json + read.json. Prompts drop the 'keep existing entries' hint (own file, overwrite is fine). Updates expel-node's reads to user.json too.
object-store now ships a fixed payload.txt to the operator and tells the agent to store that file's contents (deterministic id/bytes) instead of inventing 'distinctive text'; verify.py (object-store and read-remote-object) uses the shipped file as ground truth. Simplify every task prompt to precise, minimal wording and name __TARGET__/__PEER__ as astral nodes.
…ls ref, minimized READMEs - enable-tor: new host task — bring up a node with a Tor endpoint and save it to /root/tor.json (validated live: real onion published + saved). - object-store: agent only stores + records object_id; verify owns the read-back and byte match against the shipped payload.txt. - adopt-node: agent records swarm siblings to ~/siblings.json (sibling_ids); verify asserts it includes the adopted node. - configure-astral-agent: SATFORGE_SKILLS_REF builds the lab against a skills branch (fails loudly if the ref can't be fetched). - prompts: terser, human-style wording (adopt/bootstrap/import/expel/object-store). - READMEs: minimized to astral-docs voice across all tasks.
… scenario 0004)
Restore the two parked tasks for Tor scenario 0004 ("a node leaves the LAN and
links over Tor"), completing the scenario alongside the already-committed
enable-tor building block. Sequenced by tor-link.story (two-nodes -> two-nodes-tor):
enable-tor -> leave-lan -> link-over-tor.
- leave-lan (host): seed node1 with node2's onion while the LAN is up, then
nftables-drop the LAN path between them (WAN/Tor egress stays up). verify.py
asserts node2 can no longer TCP-connect to node1:1791 (only a timeout passes).
- link-over-tor (agent): thin prompt drives the Qwen operator to force the swarm
link over Tor (nodes.new_link -strategies tor) per the astral-agent skill's
linking-over-tor playbook; verify.py independently asserts a nodes.links entry
with Network=tor.
The linking-over-tor playbook is now on skills main (PR #4), so the lab builds
against main with no SATFORGE_SKILLS_REF override.
Checkpoint: not yet validated against the real Tor network (the original parking
gate -- VM WAN NAT -> Tor reachability + the agent's per-turn tool-call cap).
A resumed stage runs astrald + all userspace live; netsim's sync_clock corrects the stale snapshot clock with a ~day forward jump, which makes systemd's Persistent apt-daily/apt-daily-upgrade timers fire and unattended-upgrades saturate the 1-vCPU VMs for minutes -- breaking every resumed scenario (node unreachable, QMP save timeouts). astrald itself tolerates the jump. Fix it once in the image (standard ephemeral-VM hygiene): install-astrald masks apt-daily/apt-daily-upgrade/unattended-upgrades on the fresh build boot, so every stage is born quiet. The per-task quiescing in enable-tor/leave-lan is now redundant and removed (DPkg::Lock::Timeout kept). Validated by a full rebuild: the resumed scenarios no longer saturate (object-store-peer/read-remote/tor-link green, no QMP/ssh-banner timeouts); tor-link clears the real-Tor path end to end.
Expulsion is a membership change, not a disconnect -- a lingering link is permitted -- so verify no longer checks nodes.links. It asserts node2 is in user.list_expelled and gone from user.swarm_status. node2's identity now comes from node1's siblings.json (recorded by adopt-node), not from node2: once expelled, node2 rejects user.info (query rejected (2) untokened, auth_failed with the User token), so it can't identify itself. Verified live against a post-expel stage.
The import/bootstrap prompts said "set up user ... save id + token", which the agent could satisfy by importing the key + minting a token without installing the node's active contract (user.info then rejects). State the end goal -- make this a User node, install the active contract -- so the agent runs the full node-setup flow. Validated: import-user now passes.
…allback, offline tests)
…nonymous WS sessions)
…op dead code, intent comments)
…ternal-checkout path)
…ing it astrald has no carrier/operstate monitor -- mod/ip polls net.InterfaceAddrs() every 3s and advertises one tcp endpoint per assigned IP. An nftables DROP (or a bare link/carrier down, which retains the IPv4 address) is invisible to it: it never withdraws the 10.77 endpoint and never exercises the re-link-over-Tor path. Flush node2's own 10.77 address instead (RTM_DELADDR drops the address and its connected route; the NIC is downed too) -- exactly what astrald observes as leaving the network. verify.py becomes a blind, deterministic check (node2 has no 10.77 address or route) instead of a TCP-timeout probe, which would be unreliable anyway: with a WAN default route the LAN connect falls through to the WAN NAT and times out rather than returning ENETUNREACH. The Tor re-link stays asserted by link-over-tor. Offline-validated (sh -n, py_compile, 14/14 _lib tests); live validation deferred to the next full rebuild.
…QGA time-sync) Install + enable qemu-guest-agent so the host can correct the guest clock out-of-band over virtio-serial on snapshot resume (netsim's qga.sync_time -> guest-set-time), instead of racing sshd while the resume clock-jump storms the 1-vCPU VM. Folded into the existing deps install; enabled so it is baked running into the snapshot and answers on resume. Requires the netsim guest-agent virtio-serial device (satforge/netsim branch intern0/dev/guest-agent-time-sync); inert without it. Adding that device changes the machine model, so this only takes effect on a from-scratch rebuild.
…s netns)
enter-nat moves astrald into netns "priv", so its 127.0.0.1 becomes the netns
loopback and it loses the root-ns Tor. astrald's tor module needs Tor at
127.0.0.1:9050/9051 AND its onion service's local listener is hardcoded
127.0.0.1:0 (mod/tor/src/server.go), which Tor dials on inbound -- so a config
knob can't fix it; Tor must run in the same netns.
configure-nat-tor (runs after enable-tor + enter-nat, per --vm):
1. WAN masquerade for 192.168.99.0/24 so Tor-in-netns reaches the real Tor
network (enter-nat's LAN SNAT still handles 198.51.100.x peer traffic;
routing splits by destination);
2. move tor@default.service into netns "priv" via a NetworkNamespacePath
drop-in (same idiom enter-nat uses for astrald);
3. restart Tor (in netns) then astrald; self-validates by confirming astrald
re-publishes its onion. No astrald source/config change.
Deferred-phase task for the NAT-punch line
(enable-tor . enter-nat . configure-nat-tor . add-reflector . punch-nat).
Syntax-validated (sh -n + embedded python); live validation deferred to the
punch-phase bring-up (needs the enable-tor + enter-nat preconditions).
… kcp helpers WIP checkpoint of the NAT hole-punch scenario (see the "simulate NAT hole punching" task doc), milestone phase — no punch yet. - enter-nat: put a peer's astrald behind its own symmetric true-masquerade NAT (private netns "priv" + veth + port-preserving SNAT to a public TEST-NET alias 198.51.100.<lan-octet>; astrald relaunched inside the netns via a NetworkNamespacePath drop-in). The leave-lan analog for the NAT line. - add-reflector: wire a public reflector node so each NAT'd peer learns its own public endpoint by reflection (ObservedEndpointMessage), arming its nat module. - _lib/astralapi.py: links_by_network() + kcp_links() — the verify helper for the eventual punch (a kcp link is the unique punch signal). nat-eim-probe (throwaway de-risk probe) and the throwaway stories are held back deliberately. No milestone verify yet; enter-nat/add-reflector have no verify.sh. Committed on dev--netsim-scenarios atop configure-nat-tor (963554c).
… on both peers The current NAT milestone is "nat module enabled on both peers", but add-reflector had no verify, so the wiring could silently fail. Add a blind host-side check: for each peer, derive its public TEST-NET alias 198.51.100.<10.77-octet>, query the peer's astrald `ip.public_ip_candidates`, and assert that address is present -- which is exactly what flips nat.enabled (evaluateEnabled = setting-default-on AND len(PublicIPCandidates())>0). Queries via the Go astral-query CLI over the apphost unix socket (not the astral-py WS client): astrald runs inside netns "priv" so its WS port is netns-local, but the unix socket is in the shared mount ns and crosses the net-ns boundary. Syntax-validated; live validation comes with the milestone story (needs enter-nat + add-reflector applied to a NAT'd pair -- no stage carries that yet).
…enter-nat)
Root cause of the long-standing enter-nat failure ("astrald did not come back up
in netns"): astral-query defaults to tcp:127.0.0.1:8625 (lib/apphost
DefaultEndpoint; only the token is env-overridable, not the endpoint). Once
enter-nat moves astrald into netns "priv", that 127.0.0.1 is the netns loopback,
unreachable from the root ns -- so the readiness probe (and add-reflector's peer
calls, the armed verify, configure-nat-tor's onion check) all silently failed.
Fix: prefix every astral-query targeting a NAT'd node with `ip netns exec priv`
(astral-query in the netns hits astrald's netns-local 127.0.0.1:8625). Corrected
the enter-nat header comment (the unix-socket-from-root-ns assumption was wrong).
No astrald change.
…cenario) Completes the NAT hole-punch line. From a source-grounded design pass (signaling is Tor: the tcp-only Basic link strategy can't form for two symmetric masquerade NATs, and the punch client sets no relay hint, so nat.node_punch/peerSupportsNAT route over a Tor link -> configure-nat-tor is required): - punch-nat: host-driven. Resolves both node identities, host-brokers mutual onion knowledge (nodes.add_endpoint tor:<onion>, not trusting auto-sync), ensures a live Tor signaling link, then triggers `nodes.new_link -strategies nat` on node1 (drives NATLinkStrategy end-to-end; NOT nat.punch which yields no kcp link). Confirms a durable kcp link on BOTH peers; diagnosis dump on failure (nodes.links / nat.list_holes / conntrack / ss / journal). - verify.py: blind kcp assertion on both peers -- a kcp link to the sibling (unique punch signal), and NEGATIVE: no direct/10.77 tcp link to the sibling. - nat-punch.story: two-nodes -> enable-tor -> enter-nat -> add-reflector -> configure-nat-tor -> punch-nat -> two-nodes-nat (sibling of tor-link). All astral-query targeting a NAT'd node runs inside its netns (ip netns exec priv). Milestone (enter-nat + add-reflector -> nat armed on both) is now live-GREEN. Syntax-validated; the full punch run is next.
Two defects blocked the punch from promoting to a kcp link; both fixed and
confirmed live (kcp link on both peers; full nat-punch story green end to end).
enter-nat: add an inbound DNAT of the public alias into the netns
(iif <lan> ip daddr <pub> dnat to 192.168.99.2). The alias is a LOCAL address
on the collapsed NAT+endpoint VM, so with SNAT only, inbound punch packets hit
the local INPUT (no listener) and never reach the netns puncher — the
conntrack-reply reverse-SNAT that should forward them fails under a source-port
realloc clash. A 3-point packet trace (netns veth / root veth / lan0) showed
both punchers emit and packets reach each peer's lan0, but inbound is never
delivered inward. The DNAT makes the box a real cone-NAT gateway (inbound and
outbound become one conntrack flow). This makes the NAT full-cone — the
permissive punchable type; "symmetric" was always a misnomer since astrald's
cone puncher cannot traverse a symmetric NAT.
nat-punch.story: run configure-nat-tor BEFORE add-reflector. add-reflector arms
nat via an in-memory reflected endpoint; configure-nat-tor restarts astrald,
which wiped it, so both peers were disarmed at punch time ("does not support NAT
traversal"). Arm last, after the final restart.
Move netsim/stories/<name>.story -> netsim/scenarios/<name>/<name>.story and add a short, high-level README.md per scenario: one-line summary, kind (fixture vs scenario) + family, start->save chain, task steps, the run command, and a plain- words explanation. Mirrors the task convention (a directory holding the recipe + its README). Update netsim/README.md tree and invocation examples to the new paths; nat-punch README notes the current full-cone NAT caveat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.