From 5cccf2d6ea9d2ad6d78b7d9538972f62f23da34d Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Fri, 31 Jul 2026 15:10:27 +0000 Subject: [PATCH 1/3] docs(qa): add release-review skill and six A/B regression scenarios Captures a repeatable nightly/weekly QA procedure (verified changelog, scoped code review, true old-vs-new binary GUI regression pass, visual changelog) as a reusable, environment-independent skill, plus the version-agnostic GUI scenarios it authored for the current diff-touched risk areas (identity key recovery, wallet Max-send/asset-lock, DPNS registration, error banners/Identity Home actions, shielded availability after sync, and DAPI budget resilience across repeated resyncs). Co-Authored-By: Claude Sonnet 5 --- .claude/skills/release-review/SKILL.md | 373 ++++++++++++++++++ docs/gui-testing/README.md | 14 +- .../dapi-budget-resilience-after-resync.md | 119 ++++++ .../scenarios/dpns-registration-flow.md | 119 ++++++ .../error-banners-identity-home-actions.md | 116 ++++++ .../identity-key-recovery-migration.md | 135 +++++++ ...atform-shielded-availability-after-sync.md | 93 +++++ .../scenarios/wallet-max-send-asset-lock.md | 126 ++++++ 8 files changed, 1094 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/release-review/SKILL.md create mode 100644 docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md create mode 100644 docs/gui-testing/scenarios/dpns-registration-flow.md create mode 100644 docs/gui-testing/scenarios/error-banners-identity-home-actions.md create mode 100644 docs/gui-testing/scenarios/identity-key-recovery-migration.md create mode 100644 docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md create mode 100644 docs/gui-testing/scenarios/wallet-max-send-asset-lock.md diff --git a/.claude/skills/release-review/SKILL.md b/.claude/skills/release-review/SKILL.md new file mode 100644 index 000000000..901f534d6 --- /dev/null +++ b/.claude/skills/release-review/SKILL.md @@ -0,0 +1,373 @@ +--- +name: release-review +description: "This skill should be used when preparing to QA a dash-evo-tool nightly/weekly build before it ships — comparing the last published pre-release against the current development-branch HEAD via a verified changelog, a scoped code review, and a true A/B GUI regression pass with a visual changelog." +--- + +# Release Review + +End-to-end nightly-release QA campaign for dash-evo-tool: find what actually changed since the +last shipped pre-release, run a scoped code review, and prove the GUI isn't worse than before in +the happy flow. Produces a triageable findings report plus a visual (screenshot) changelog a human +can eyeball in minutes. + +This procedure assumes nothing about the machine it runs on beyond a working Rust toolchain, `git`, +and the `gh` CLI, and nothing about which Claude Code capabilities are installed beyond the +built-in ones (Bash, file tools, and — if available — the ability to spawn sub-agents for +independent work). Anywhere a specialized skill/plugin would make a step faster or more rigorous, +it's named as an optional example in parentheses, never as a hard requirement — read "if you have +access to X" as "otherwise do the equivalent by hand." + +**Pick a scratch root before starting** and reuse it throughout — a directory outside the repo's +own working tree that this run can write to freely (a fresh `mktemp -d`, or a fixed directory you +control). Every path below written as `/...` means "somewhere under that root," not a +literal path — don't hardcode one run's actual path into anything durable (docs, other agents' +instructions, the final report). + +## Rerunning after a fix lands (or a pause) + +The trigger is usually "a fix/PR landed, redo the campaign against the new HEAD" or resuming after +a § Handling a Fixture/Infra Blocker pause. Don't rebuild from scratch: + +1. **Re-verify the baseline is still current** (`gh release list` — a newer pre-release may have + shipped since the prior run). If unchanged, the prior run's worktree/binary for it is still + valid — check for an existing `git worktree` and a previously-built binary under your scratch + root before rebuilding anything. +2. **Only rebuild what moved.** If the baseline is unchanged and only HEAD advanced, reuse the + baseline binary as-is; `git fetch` + `git checkout ` the HEAD worktree in place and + rebuild only that binary. This turns a two-cold-build Phase 2b into one incremental build. +3. **Archive, don't overwrite, the prior run's output.** Before writing fresh Phase 1-5 artifacts, + move the previous run's files into a subfolder (e.g. `run-vs-/`) under the + same artifacts location. Both runs stay comparable/auditable, and nothing silently clobbers a + finding a human might still want (e.g. a confirmed-fixed crash repro). +4. **Reuse scenario files and prior findings as a starting point, not from scratch.** A changelog + pass can diff the prior run's verified commit list against the new range instead of re-deriving + the whole thing; a GUI pass can decide per-scenario whether the flow is unchanged enough to reuse + the prior run's OLD-side (baseline) screenshots rather than recapturing them — an unchanged + baseline binary means its screenshots are still valid evidence. Always capture fresh NEW-side + screenshots (HEAD moved), and log explicitly which screenshots were reused vs freshly captured so + it's auditable rather than assumed. +5. If any part of this run is delegated (to a sub-agent or a fresh session), give it the prior run's + key facts (SHAs, decisions already made, artifact paths) explicitly — it has no memory of a prior + or paused run and will otherwise re-derive, and burn time on, things already verified. + +## Phase 0 — Fixture preflight (do this FIRST, cheaply) + +Before committing to a real GUI run, verify the end-to-end test fixtures are actually usable *now* +— testnet masternodes deregister and test wallets drain between runs. Check the repo's `.env.example` +for the exact variable names in current use; as of this writing they include: + +- `E2E_MN_PROTX_HASH`: confirm it's a currently-registered masternode/evonode. A stale hash surfaces + as a typed `MasternodeNotFound` error the first time something touches it — don't wait for that; + check via an SDK/CLI lookup or a quick platform-explorer query first. +- `E2E_WALLET_MNEMONIC`: confirm the wallet is actually funded on testnet *right now* (a live balance + check, not "it was funded last time"). If empty, get a faucet drop (a testnet faucet — web, CLI, + or whatever this project's e2e docs point at) or a fresh funded fixture BEFORE starting Phase 4 — + don't discover this after a long SPV sync. + +If either fixture is broken and can't be fixed quickly: stop here, log it (see § Handling a +Fixture/Infra Blocker below), and don't burn hours running non-fund-dependent checks only — check +with the user whether a partial (navigation-only) pass is worth doing now or the whole campaign +should wait. + +**Confirmed false-alarm trap**: before concluding a fixture is dead, verify the app's **network +selector** is actually set to Testnet. The dropdown has no explicit default after seeding a fresh +data directory from `.env.example` — a prior run spent real time diagnosing a "dead masternode" and +"unfunded wallet" that were both fine; the app was just sitting on Mainnet the whole time. Check the +network selector FIRST, before treating a zero balance or a `MasternodeNotFound` as a real fixture +problem. + +## Phase 1 — Baseline identification + verified changelog + +1. **Find the baseline release**: `gh release list --repo dashpay/dash-evo-tool` filtered to + `draft:false && prerelease:true`, most recent. Automated builds are commonly cut as drafts first + and can be deleted before undrafting (check `.github/workflows/` for the exact release-cutting + workflow and tag-naming scheme currently in use) — a newer *local* tag with no matching GitHub + release is not a valid baseline. +2. **Force-sync the tag from origin before trusting it**: + `git fetch origin "+refs/tags/:refs/tags/"`, then `git rev-parse ^{commit}`. + **Confirmed gotcha**: a local tag ref can silently diverge from origin (stale from an earlier + fetch/session) and point at an *ancestor* commit, producing a diff range that's wrong (larger than + reality) without any error. Always re-verify against `git ls-remote --tags origin ` before + computing any diff range from a tag. +3. **Determine the active development branch** — check the repo's own contribution docs (e.g. + `CLAUDE.md`, `CONTRIBUTING.md`) rather than assuming `main`/`master`; this project may use a + dedicated long-lived dev branch instead. Compute the range against `origin/` (never + a possibly-stale local branch ref, and never the invoking session's own feature branch): + `git log ..origin/ --oneline`. +4. **Don't trust `CHANGELOG.md` alone.** Its unreleased section can contain stale/premature entries + or miss real changes. For every commit in range, resolve its PR number and read the actual PR body + (`gh pr view `) — not just the commit subject, which can undersell what a PR really changed + (confirmed: one PR's CHANGELOG entry omitted two of its own later review-round fixes). Cross-check + both directions: CHANGELOG claims with no backing PR, and merged PRs with no CHANGELOG entry. +5. **Confirm things are actually in-range, not pre-existing**: `git merge-base --is-ancestor + ` for anything you suspect might already be shipped — don't regression-test + pre-existing behavior as if it were new. +6. Produce a verified changelog grouped by theme, each entry citing commit SHA + PR number, tagged + user-visible (GUI-relevant) vs internal-only (tests/CI/deps), plus a regression risk map by + functional area. Save under your artifacts location for this run (see § Artifact Conventions). + +## Phase 2 — Scoped code review + +Review the diff `...origin/` (three-dot range) thoroughly: security, +project/structural consistency, code quality, dependency risk, documentation accuracy. Run it from +an isolated worktree, never the invoking session's active checkout — a plain `git diff`/`git log` +between two refs doesn't need a checkout of either at all, but reviewing from a clean worktree +avoids any risk of touching the human's working tree. (If a multi-agent code-review skill/plugin is +available — e.g. one that fans reviewers out by dimension and file group — use it; otherwise cover +the same dimensions yourself, splitting the work across multiple passes for a large diff rather than +one shot.) Archive the resulting findings report to your artifacts location immediately if it was +produced somewhere scratch/ephemeral. + +**Scale for the actual diff size.** A genuinely large range (many dozens of files, tens of thousands +of changed lines) deserves a proportionally wider review — more reviewers/passes across more file +groups, not the default handful. If findings are emitted with a file:line `location`, make sure it's +**repo-relative, not an absolute worktree path** — an absolute path like +`/some/scratch/worktree/src/foo.rs:12` breaks any downstream permalink generation and is hard to +read out of context. Also check the diff for review-comment IDs (from a *previous* review round) +that leaked into committed code or test comments — a real, recurring defect class: a prior review's +transient ID (e.g. referencing an old finding number) ending up quoted in a doc comment as if it were +permanent documentation. + +**Severity scores are not negotiable by request.** If you ask a reviewer (human or agent) to +reclassify or re-score a finding, that request is not itself evidence. A reviewer that declines to +inflate a severity score to match your expectation and instead re-derives it honestly — even landing +lower than you expected — is doing its job correctly. Don't push back without a substantive +argument for the new score. + +## Phase 2b — Build both binaries for A/B comparison + +Create two isolated `git worktree`s under your scratch root — one at the baseline tag's commit, one +at HEAD — never inside the repo's own tracked working tree, and never let a spawned sub-agent create +worktrees on its own initiative if you're coordinating multiple agents (create them yourself first, +hand the paths out). Build each with: + +``` +CARGO_TARGET_DIR=/-target cargo build --bin dash-evo-tool --manifest-path /Cargo.toml +``` + +Distinct `CARGO_TARGET_DIR`s are mandatory — both binaries must coexist afterward for the A/B run; +sharing a target dir means whichever built last silently overwrites the other's binary at the same +output path. Always pass `--manifest-path` explicitly (worktree auto-discovery can resolve to the +wrong checkout). Build sequentially if the machine is RAM-constrained (two full dependency-graph +cold builds concurrently risks OOM/thrashing) — check available memory / existing load first. + +**Confirmed gotcha, background builds only, both Linux and macOS**: a `cargo build` backgrounded +(`... &`, `nohup`, or similar) from an automated shell can fail with a "cargo: No such file or +directory"-style error even though `cargo` works fine in a normal foreground call — a backgrounded +subshell doesn't always inherit the interactive profile that puts `cargo` on `PATH` (common when +installed via `rustup` under `~/.cargo/bin`). Capture the resolved path with `command -v cargo` (or +`rustup which cargo`) *while it's confirmed working*, and use that absolute path for any backgrounded +build invocation rather than the bare `cargo` name. + +## Phase 3 — Author GUI regression scenarios + +Informed by Phase 1's risk map, write **version-agnostic** scenario files under +`docs/gui-testing/scenarios/` (from that directory's `TEMPLATE.md`) — the same procedure runs +against both binaries, so never phrase a step as "confirm the fix works." Bake the blocker rule (§ +below) directly into each scenario's "Expected outcome" section. Update the "Scenario index" table +in `docs/gui-testing/README.md`. + +Also produce a risk-prioritized `docs/user-stories.md` checklist, two tiers: +- **Deep tier**: full acceptance-criteria comparison (old vs new) for every story plausibly touched + by the diff. +- **Smoke tier**: everything else — screen opens, no crash, no error in logs — toured in one + continuous session per binary rather than relaunching per story ID. + +Save the checklist under your artifacts location (a QA-campaign planning doc, not permanent product +documentation — don't fold it into the tracked repo). + +If a story text is found stale against shipped behavior while cross-referencing (confirmed: found +one describing a pre-redesign button layout weeks after the redesign shipped), flag it as a doc +defect for the final report — don't silently fix it mid-campaign, that's scope creep. + +## Phase 4 — GUI A/B run + +Drive the actual compiled binary through a real rendered window, twice (once per build) — how you +reach a real window differs by OS, so confirm before assuming: + +- **Any headless/server Linux environment**: check whether `$DISPLAY` (or `$WAYLAND_DISPLAY`) is + already set to something — if the session was reached over SSH with X11 forwarding, that ambient + value points at the *human's own screen*, not a safe throwaway one. **Verify this before every GUI + launch** — launching against the wrong display pops the app window on someone's real desktop. If + no safe display exists, stand up a virtual one (e.g. `Xvfb`) and point `DISPLAY` at it explicitly + for every launch — don't rely on an inherited/ambient value. +- **macOS**: normally has a real display attached already; if running headless/remote (SSH, CI + runner), GUI automation typically needs Screen Recording/Accessibility permission granted to the + driving process, or a remote-desktop/VNC session to attach to. There is no X-server equivalent to + spin up standalone the way `Xvfb` works on Linux. +- Whatever the mechanism, **isolate a scratch data directory per launch** (`mktemp -d`, portable on + both Linux and macOS) and point the app's data-dir env var at it — never reuse or mutate a real + user's data directory. + +For each scenario/story check: confirm no conflicting instance of the app is already running +(process list, e.g. `pgrep -f dash-evo-tool` or `ps aux | grep dash-evo-tool` if `pgrep` isn't +available) before launching. Launch **old**, perform the flow, screenshot, fully close (confirm +process exit); launch **new**, identical flow, screenshot, close. Never run both simultaneously — +one real window/display target at a time, and it keeps screenshot provenance unambiguous. Check the +app's log output for both runs (with an isolated data directory, verify where logs actually landed +for this run rather than assuming a fixed default path). + +**Blocker rule** (confirm with the user for each run — the default below is a starting point, not a +universal constant): +- **Blocking**: new version worse than old from the user's perspective **in the happy flow**, or a + **data-loss** scenario. +- **Not blocking**: concurrency/timing glitches; anything reproduced identically on **both** + builds (pre-existing — note it, don't flag as a regression). +- Reproduce anything about to be marked blocking at least twice before confirming. +- If a repro attempt genuinely can't be reproduced after several tries, don't leave it open + indefinitely — write up the negative evidence, downgrade to backlog, and say explicitly that repro + was attempted and failed, rather than silently dropping it or mandating another attempt on every + future rerun. + +**Blocking classification must say pre-existing vs diff-introduced, not just "blocking."** A defect +can be blocking-worthy on its own severity (real data loss) while NOT being a regression this diff +caused. Before asserting "this release introduced X," check: `git log .. -- +` — zero commits touching the file means it's pre-existing, not diff-introduced. Frame +the finding as "we are choosing to hold/flag a pre-existing defect," not "this release broke X" — the +second is a factual claim that can be wrong and, if wrong, undermines the whole report's credibility. + +Emit confirmed findings in whatever findings format the rest of this campaign's report uses (see +Phase 2/5) — at minimum: title, severity, a location reference (a file:line if one applies, otherwise +a synthetic description like "GUI: `` > ``"), description, recommendation, and +whether it's blocking per the rule above. + +**If findings get consolidated/renumbered by any downstream tooling, treat IDs as provisional and +don't embed them anywhere durable.** A findings pipeline may reassign IDs when merging sections +(confirmed: GUI-section IDs were renumbered twice across two merge passes in one run). Never +reference a provisional or even a "final" ID in a scenario file, the visual changelog, or an +inter-agent message meant to outlive one exchange — reference findings by **title** or **scenario +name** instead, which survive renumbering. Warn whoever's writing durable artifacts about this +explicitly before they start, not after — catching and fixing already-embedded IDs later is possible +but wasteful. + +## Escalating a claim that can't be settled from the repo alone + +A review pass occasionally surfaces something it explicitly flags as unresolved from static analysis +— e.g. "does the live network actually behave the way I think the code implies?" Don't ship that as +a confirmed blocking finding, and don't dismiss it either: investigate it directly and narrowly +rather than waiting for a human to resolve it, if it's the kind of thing investigable from evidence +(read the actual pinned dependency's source, trace the real call order, and — if it's the only way to +get ground truth — make a strictly read-only live check, e.g. a version/status query against the +live network, never a transaction). Confirmed valuable: a claim about a possible mainnet outage was +fully resolved via a few read-only queries against live mainnet nodes in well under an hour — cheap +compared to either shipping a false alarm or silently dropping a real risk. When the verdict lands, +fold the correction into the report explicitly — state that an earlier pass got something wrong and +why, rather than quietly replacing it. A report that visibly corrects itself is more trustworthy than +one that never admits an error. + +## Provenance: unrelated concurrent work on a shared environment + +If this campaign runs on a machine or account that may have other unrelated work in flight (other +worktrees, other sessions, other automation touching the same repo), a QA pass may stumble on +evidence of a *different*, unaffiliated effort — a report file, a branch, a running build — that +looks relevant to something this campaign is investigating. Before treating it as this campaign's +own finding: +1. **Verify it's actually external**, don't assume: check for a still-live process tied to it, a + moving `git log` HEAD in its worktree, a different session/transcript identity, or simply that + nothing in this campaign's own roster claims authorship. +2. **If external, exclude it entirely from the formal report** — don't cite, reference, or fold in + findings/fixes from work with no chain of custody to this campaign, even if independently verified + as genuine. It may be unstable (still evolving), unauthorized for this release, or owned by + someone who hasn't consented to having their in-flight work published in your report. +3. **Do surface its existence to the user as a side note** when reporting results — it's relevant + context for a ship decision even though it's not your finding. +4. **Leak-scan the final artifacts** before declaring done: search the produced report/HTML for any + branch name, commit hash, worktree path, or distinctive identifier from the external work, to + confirm the exclusion actually held rather than assuming it did. + +If you're confident this run is the only thing touching the repo/environment, this section doesn't +apply — but check rather than assume on a shared machine. + +## Security incidents during GUI testing + +If GUI testing handles real secrets (a testnet mnemonic, a password fixture) and something goes +wrong — a screenshot with the secret visible, a secret typed into a visible log/transcript — the +self-report is not the end of it: +1. **Independently verify remediation** — don't just trust "I deleted it." Check the files are + actually gone, check adjacent files/screenshots for spillover, confirm no other copy exists. +2. **Write an incident note** into this run's artifacts (what happened, exposure window, what was + verified, actual risk given the context — testnet vs mainnet, network-restricted vs public, + closed window vs ongoing). +3. **Don't unilaterally rotate a shared fixture.** A fixture used beyond just this campaign (e.g. by + other test suites in the repo) shouldn't be silently burned and replaced — queue the + rotate-or-accept decision for the user. +4. **Never screenshot a populated secret-entry field going forward** — capture before typing or + after the field is cleared/masked, for the rest of the run. +5. Log the incident somewhere that survives this session ending (an issue tracker entry, a durable + notes/memory mechanism if one is available, or at minimum a clearly-named file in this run's + artifacts) so it isn't lost if the user doesn't act on it immediately. + +## Phase 5 — Visual changelog + consolidated report + +Build a self-contained HTML page (`visual-changelog.html`) presenting old/new screenshot pairs side +by side per scenario/story, with a verdict badge and a one-line rationale each — inline the images +(e.g. base64) so the page has no external file dependencies. Save it to your artifacts location. +Merge Phase 4's findings into Phase 2's report as an additional section, re-validate the combined +report against whatever schema/format it uses, and re-render any human-readable version. The +combined report is then ready to hand to the user for triage — code-review and GUI-regression +findings reviewed together in one pass. (If an interactive triage tool is available, use it; +otherwise a written summary the user can respond to works fine.) + +## Handling a fixture/infra blocker + +If Phase 4 hits a real environment blocker (dead fixture, unreachable network, broken CI runner — +not an app bug), don't push through with silent scope cuts: +1. Report the exact blocker (the actual error, what was verified, what's still needed) to the user + and ask how to proceed — a scope trim is a real thoroughness-vs-speed tradeoff, not something to + decide alone. +2. If the user says to pause: cleanly close any running app instances, save interim findings under + this run's artifacts location, and log what's blocked, what already completed and is reusable on + a rerun, and what a fresh run needs from scratch (baseline/HEAD will likely have moved on by + then) — durably enough to survive the current session ending. +3. Updating this skill file with anything newly learned doesn't depend on Phase 4/5 completing — do + it whenever asked, even mid-block. + +## Artifact conventions + +Pick one consistent, git-ignored location outside the tracked repo for everything durable in a given +run (a dated directory under your scratch root, or wherever your own setup keeps generated reports) +and use it for: the verified changelog, the code-review report (JSON/Markdown or whatever format is +in use), GUI findings, the GUI test log, `visual-changelog.html`, screenshots, and the user-stories +checklist. If your environment has a way to share generated files with the user (a hosting mechanism, +an upload/artifact tool, or simply pointing at local paths they can open), use whatever's actually +available — don't assume a specific serving mechanism exists. + +Keep a small `qa-run-state.md` at the top of that location as a durable scratchpad: verified SHAs, +worktree/binary paths, a phase checklist, and an explicit "open items for the user" list. That's what +lets a rerun (or a resumed session after a context loss) pick up cleanly instead of re-deriving +everything. On a same-day rerun, archive the prior run's files into a subfolder first (see § +Rerunning above) rather than overwriting them. + +Don't bake one run's actual findings, SHAs, or open items into *this* skill file — they belong in +that run's own artifacts. When updating this file after a run, fold in newly-learned *procedure* +(a gotcha, a missing step, a better ordering) and leave the specific bug numbers/dates/commits out. + +## Running this unattended (user stepped away) + +This campaign is long enough that the user often steps away mid-run. If so: +- **Corroborate status against direct evidence, not just self-reports.** A status message saying + "still working" is not itself proof of progress — check actual file timestamps in the scratch + area and running processes before concluding anything is stuck, or conversely before assuming + silence means done. +- **Queue decisions instead of fabricating approval.** Anything that's normally the user's call — + merging/pushing anything, rotating a shared fixture, a scope tradeoff on a fixture blocker — gets + written up and queued, never assumed. Keep working on whatever doesn't need that decision rather + than stalling the whole campaign on one open question. +- **Notify once, at the end**, with the full queue: ship read, report location, and every item that + needs a decision — not a running commentary as each phase completes. Use whatever notification + mechanism your environment provides (a desktop/mobile notification tool, a chat message); if none + is available, just deliver the full summary in your final response. + +## Skills used (optional accelerators, not requirements) + +Every step above works performed directly with standard tools (`git`, `gh`, `cargo`, a real GUI +session, plain file writes). If your environment happens to have compatible skills/plugins +installed, they can speed things up — for example: a multi-agent orchestration capability for +running independent phases in parallel and tracking a long multi-phase run durably; a multi-agent +code-review skill (e.g. something like `grumpy-review`) instead of a single-pass manual review; a +structured findings-report format/schema and severity-scoring convention instead of a plain +Markdown table; an interactive browser-based triage tool instead of a written summary; a GUI-driving +skill that already handles the headless-display setup for you; a testnet-faucet helper instead of a +manual funding step; and an end-of-run notification skill for alerting a user who's stepped away. +None of these are assumed to exist — treat every mention above as "if available," never as a +dependency this procedure requires. diff --git a/docs/gui-testing/README.md b/docs/gui-testing/README.md index 7da93a2b9..2c3a62548 100644 --- a/docs/gui-testing/README.md +++ b/docs/gui-testing/README.md @@ -253,8 +253,20 @@ one scenario silently invalidating another: ## Scenario index +These six scenarios are written **version-agnostic**: the same procedure +runs unmodified against both the baseline release build and the current +`v1.0-dev` build for an A/B comparison, with the blocker rule (worse than +baseline in the happy flow, or data loss, is blocking; concurrency/glitches +and issues present on both builds are not) baked into each file's own +"Expected outcome" section. + | Scenario | What it verifies | |---|---| -| _(none yet — first one lands once the masternode-withdrawal-without-a-wallet run is verified)_ | | +| [`identity-key-recovery-migration.md`](scenarios/identity-key-recovery-migration.md) | Key-placement resolution and legacy-key recovery (#941, #945, #946, #948): keys found/signed/restored/removed correctly regardless of internal filing convention; Keys screen reachable from every interface level; same-numbered-key collisions never cross-contaminate | +| [`wallet-max-send-asset-lock.md`](scenarios/wallet-max-send-asset-lock.md) | "Max" agrees with what the wallet can build/send across Shield/Fund-Platform-Address/Create-Identity/Top-Up/Transfer/Withdraw (#937, #927), including dust-wallet and rapid-UTXO-churn variants | +| [`dpns-registration-flow.md`](scenarios/dpns-registration-flow.md) | Contested-vs-registered DPNS messaging, pending-registration indicator/tooltip, onboarding checklist, and DashPay social-profile save feedback (#918) | +| [`error-banners-identity-home-actions.md`](scenarios/error-banners-identity-home-actions.md) | Error-banner wording, validation consistency, identity-removal responsiveness, Identity Home's action row, already-consumed-deposit messaging, deposit-verification crash fix (#927, #934) | +| [`platform-shielded-availability-after-sync.md`](scenarios/platform-shielded-availability-after-sync.md) | Shielded features correctly activate right after SPV/platform sync instead of silently staying disabled (#936, #938) | +| [`dapi-budget-resilience-after-resync.md`](scenarios/dapi-budget-resilience-after-resync.md) | Repeated SPV `Syncing`↔`Synced` transitions no longer exhaust the shared DAPI request budget and break unrelated actions like identity top-up (#950) | 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md new file mode 100644 index 000000000..6d686e68b --- /dev/null +++ b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md @@ -0,0 +1,119 @@ +# Scenario: Connection budget survives repeated resync cycles + +**Verifies:** Cycling the SPV connection state between `Syncing` and `Synced` +repeatedly does not exhaust the app's shared DAPI request budget, and does not +cause an unrelated Platform action (identity top-up) to fail with a generic +"servers are temporarily unreachable" banner. (Risk area: #950, building on +#936/#938.) + +**Tier justification:** The failure this scenario targets is a real, +live-network request-budget exhaustion that only manifests after several +actual `Syncing`→`Synced` transitions against a real DAPI connection — it +cannot be simulated in `kittest` (no network) or isolated to a single +`backend_task` call (the bug is about a background task's cumulative effect +across transitions, not any single call's return value). + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet + - `E2E_IDENTITY_ID` (or an identity created ad hoc) — an identity to top up +- Verify exact banner/error wording during execution rather than assuming the + text below is literal + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/dapi-budget-resilience.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Baseline Platform Info check.** Open Platform Info (Developer/Advanced + view). Record the exact "Current Epoch Information" text, in particular + whether the fee-multiplier line reads "unavailable" or states it is "a + fixed value, not read from the network" (or neither, if built against a + version predating both #936 and #950). +2. **Force repeated `Syncing`↔`Synced` transitions.** With the app running + and initial sync complete, force several resync cycles in the same + session — e.g. toggle network connectivity briefly (airplane-mode-style + interruption, or a short `iptables`/firewall block on the DAPI/Core + ports if available) 4–6 times over a few minutes, or use "Clear cached + SPV data to force a resync" (NET-020) repeatedly. Each cycle should visibly + pass back through a `Syncing` state before returning to `Synced`. +3. **Attempt identity top-up immediately after the resync cycles.** Top up + an identity's credits (any small amount) right after step 2's last + transition back to `Synced`. Record whether it succeeds, and if it fails, + capture the exact banner text. +4. **Repeat the top-up attempt once more** a short time later (a minute or + two) without additional resync cycling, to distinguish "budget recovers + on its own" from "budget stays exhausted." +5. **Check the logs.** Search `det.log`/`det-stderr.log` for + `DapiAllAddressesExhausted` or similar rate-limit/address-pool exhaustion + markers appearing near the resync cycles from step 2, independent of + whether the UI showed an error. +6. **Re-check Platform Info.** Reopen Platform Info after the resync cycling + and top-up attempts; record whether the epoch/fee-multiplier text is + still consistent with step 1's observation (no crash, no stuck "loading" + state). + +## Safety constraints specific to this scenario + +- Step 3's top-up moves real (small) testnet funds — keep amounts minimal, + per the standing fund-movement cap. +- If simulating a network interruption via firewall rules, restore + connectivity immediately after each toggle and confirm via `pgrep`/process + liveness that the app itself never needed restarting. + +## Expected outcome / pass criteria + +Record the observed behavior for each build at every step, then apply this +rule: + +- **BLOCKING** only if HEAD is worse than baseline in the happy flow — e.g. + the identity top-up in step 3 succeeds on baseline after repeated resync + cycling but fails with a generic "servers are temporarily unreachable" + banner on HEAD, or `DapiAllAddressesExhausted` (or equivalent) appears in + HEAD's logs where it did not on baseline — or a **data-loss** outcome on + either build. +- **NOT blocking**: the reverse direction is the *expected improvement* this + PR ships — if baseline's top-up fails after repeated resync cycling (the + bug #950 fixes) and HEAD's succeeds, that's the intended fix working, not + a baseline "failure" to flag beyond noting it factually. A wording-only + difference in the Platform Info fee-multiplier line (`unavailable` vs `a + fixed value, not read from the network`) is expected and not itself a + defect on either build. +- If neither build reproduces an exhaustion-triggered top-up failure within + the attempted resync-cycle budget, record that plainly as "not reproduced + this run" rather than asserting the fix is confirmed — a live-network + race like this can be timing-sensitive. + +## Known gotchas + +- Simulating repeated `Syncing`→`Synced` transitions without real network + interruption tooling may require several minutes of patience — SPV + reconnect timing is not instantaneous. Budget real wall-clock time for + this scenario rather than expecting each cycle to complete in seconds. +- The DAPI request-budget exhaustion this scenario targets is a *shared, + per-client* resource — if another tool/process is also hitting the same + SDK client concurrently (e.g. an `det-cli` smoke test running in + parallel), the observed exhaustion timing may differ from a clean, + single-consumer session. Run this scenario in isolation from other DAPI + traffic against the same data directory. +- This is the only scenario in the library that deliberately drives + multiple resync cycles in one session — if a *different*, unrelated + defect surfaces only under repeated resync (not specific to #950), note + it here rather than assuming it belongs to another scenario file. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/dpns-registration-flow.md b/docs/gui-testing/scenarios/dpns-registration-flow.md new file mode 100644 index 000000000..cefc25e25 --- /dev/null +++ b/docs/gui-testing/scenarios/dpns-registration-flow.md @@ -0,0 +1,119 @@ +# Scenario: DPNS registration messaging and social-profile save feedback + +**Verifies:** DPNS registration correctly distinguishes "registered outright" +from "submitted for a community vote" (contested name), a pending +registration shows a clear indicator/tooltip instead of looking unrequested, +the onboarding checklist counts a pending request toward "Pick a username," +and — from the same underlying PR — the DashPay social-profile save flow +gives real progress/success/error feedback without a stuck progress banner +or a stale result misattributed after switching identities mid-save. (Risk +area: #918.) + +**Tier justification:** Needs a real contested-name registration against a +live testnet name-contest window, real onboarding-checklist state derived +from actual identity data, and real async save timing (a save in flight +when the user switches identity) — none of which a no-display harness can +drive with a live contest clock and real background-task scheduling. + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet +- A DPNS name likely to be contested (short, generic, or currently popular) + to register during an active contest window +- At least two identities on the same wallet (for the identity-switch- + mid-save step) +- Verify exact tooltip/button text during execution rather than assuming + the wording below is literal + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/dpns-registration.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Register a contested name.** On an identity with no username yet, + register a name likely to be contested. Record the exact completion + message shown — does it say "registered" or does it indicate a pending + community vote? +2. **Identities list indicator.** Record whether/how the Identities list + shows this identity's pending request differently from "no username." +3. **Identity Home indicator.** Record what the Identity Home hero card + shows for this identity — the requested name, a status indicator, or the + generic "pick a username" prompt. +4. **Indicator tooltip.** If a status indicator/pill is present, open its + tooltip and record what it says (who decides, any estimated timing). +5. **Onboarding checklist.** Record whether the onboarding checklist treats + the submitted request as satisfying "Pick a username," and what wording + it uses. +6. **Uncontested name, for comparison.** On a second identity, register a + name unlikely to be contested and record the completion messaging — + confirm it's still distinguishable from the contested case. +7. **Contest resolves.** If the contest window is short enough to wait out, + revisit the identity after resolution and record what the indicator does + (disappears / name displays normally on a win; reverts to "pick a + username" on a loss). +8. **Social profile save, normal case.** Open Contacts → set up/edit social + profile, change the display name or bio, save. Record what feedback + appears during and after the save (progress indicator, then a + success/error banner) and whether the progress indicator clears when the + outcome appears. +9. **Social profile save, forced failure.** If you can force a failure + (briefly disconnect network mid-save), record the same as step 8 for the + failure path. +10. **Switch identity mid-save.** Start a profile save on identity A, and + before it completes, switch to identity B. Once the save's result + arrives in the background, record which identity (if any) shows a + result banner, and whether it's ever attributed to the wrong + (now-active) identity. +11. **Contacts setup CTA.** On an identity with no DashPay profile, open + the Contacts tab and record the setup card's CTA wording and whether any + "Why?"/explanation control actually opens an explanation. + +## Safety constraints specific to this scenario + +- None beyond the standing rules — DPNS registration fees are small and + testnet-only here. + +## Expected outcome / pass criteria + +Record the observed behavior for each build at every step, then apply this +rule: + +- **BLOCKING** only if HEAD is worse than baseline in the happy flow (e.g. + a contested registration falsely claims "Registered!" on HEAD when + baseline correctly said "pending"; a save's progress banner gets + permanently stuck on HEAD where baseline cleared it; a save result gets + attributed to the wrong identity on HEAD where baseline didn't), or a + **data-loss** outcome (a profile edit silently lost) on either build. +- **NOT blocking**: a timing-sensitive race that needs several attempts to + reproduce and only sometimes shows on either build; wording differences + with no functional consequence; an issue reproducing identically on both + builds. +- If baseline doesn't have a pending-registration indicator at all (a new + feature on HEAD), record that as a feature-presence difference, not a + step failure on baseline. + +## Known gotchas + +- Contest resolution timing depends on the live masternode voting schedule + on testnet — step 7 may need to be deferred to a later session rather than + blocking the rest of this scenario. +- The identity-switch race (step 10) is timing-sensitive; several attempts + with the save action and the identity switch performed in quick succession + may be needed to land inside the race window on either build. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/error-banners-identity-home-actions.md b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md new file mode 100644 index 000000000..757f1e3b9 --- /dev/null +++ b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md @@ -0,0 +1,116 @@ +# Scenario: Error-banner wording, validation consistency, and Identity Home actions + +**Verifies:** Error banners show plain, actionable text rather than raw +upstream error strings; format-validation rules (DPNS name/memo/keyword/ +label length) behave the same everywhere they're triggered; identity removal +doesn't block the UI thread; Identity Home shows one non-redundant row of +action buttons instead of duplicate destinations; `Send to wallet` is +disabled with an explanation when no usable withdrawal key is held; an +already-consumed deposit says so plainly instead of suggesting a retry; and +the app doesn't crash during deposit verification. (Risk area: #927, #934.) + +**Tier justification:** Needs a real deposit broadcast/verification cycle to +exercise the worker-thread stack-size fix, and real visual confirmation of +button count/labels/layout that a no-display harness can assert on logic but +not on what actually renders. + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet +- One identity with a locally-held TRANSFER/OWNER key, and ideally a second + identity with only an on-chain authentication key and no locally-held + signing key (for the `Send to wallet` disabled-state comparison) +- Verify exact button labels/wording during execution rather than assuming + the text below is literal + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/identity-home-actions.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Identity Home button row.** Open Identity Home. Record how many action + buttons are present, their labels, and whether any two open the same + destination screen. +2. **`Send to wallet` gating, enabled case.** On the identity with a + locally-held signing key, record whether "Send to wallet" (or whatever + the withdrawal-initiating control is called) is enabled and whether it + opens a working withdrawal flow. +3. **`Send to wallet` gating, disabled case.** On the identity without a + locally-held signing key, record whether the same control is disabled, + and if so, what explanation (if any) is shown. +4. **Funding wording consistency.** Walk through the identity funding + wizard (breadcrumb, heading, sub-heading, CTA buttons) and record whether + the terminology is consistent throughout (vs. mixing two different terms + for the same action). +5. **Already-consumed deposit.** Attempt to fund using a deposit that has + already been consumed by another operation (re-use a completed one). + Record the exact message shown — does it say the deposit cannot be + reused and to choose/start a different one, or does it suggest retrying? +6. **Deposit verification stability.** Perform several deposit verifications + in a row (fund an identity via deposit two or three times). Record + whether the app crashes, and check `det-stderr.log`/`det.log` for a + panic marker (`location=...`) even if the UI appeared fine. +7. **Format-validation consistency.** Trigger a format-validation error the + same way in two different places that should share the same rule (e.g. + an overlong DPNS name in registration, and the same length rule via a + keyword/label field elsewhere). Record whether both reject at the same + limit. +8. **Error banner wording.** Trigger a database/parsing-style error if + possible (e.g. an interrupted operation). Record whether the banner text + is plain and actionable, or contains raw error strings/stack + traces/internals. +9. **Identity removal responsiveness.** Remove a local identity and record + whether the UI freezes/stutters during the removal, and how the result is + reported. + +## Safety constraints specific to this scenario + +- Steps 5–6 move real (small) testnet funds — keep amounts minimal. + +## Expected outcome / pass criteria + +Record the observed behavior for each build at every step, then apply this +rule: + +- **BLOCKING** only if HEAD is worse than baseline in the happy flow (e.g. + HEAD crashes during deposit verification where baseline didn't; HEAD's + already-consumed-deposit message is more misleading than baseline's; a + button that worked correctly on baseline is now broken/duplicated on + HEAD), or a **data-loss** outcome on either build. +- **NOT blocking**: a UI freeze/stutter that's cosmetic and self-resolves; + wording differences with no functional consequence; an issue reproducing + identically on both builds (e.g. if baseline also shows 6 duplicate + buttons and HEAD also does — note it, don't block). +- If baseline's button row differs in count/labels from HEAD's by design + (this PR's whole purpose is to collapse duplicates), record both + observations plainly rather than treating either as automatically wrong — + apply the blocker rule based on functional correctness (do all buttons + work, are destinations distinguishable), not on matching layouts. + +## Known gotchas + +- If `Send to wallet` behaves inconsistently, check whether you're looking + at Identity Home, the identities-list popup, or the Withdrawal screen + directly — these three surfaces historically used different gating + criteria; note which surface any discrepancy is on. +- This PR range touches many files across validation, fee estimation, error + banners, and identity removal — if a defect surfaces, compare directly + against the baseline build's behavior for the same step before concluding + it's new. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/identity-key-recovery-migration.md b/docs/gui-testing/scenarios/identity-key-recovery-migration.md new file mode 100644 index 000000000..3695fa546 --- /dev/null +++ b/docs/gui-testing/scenarios/identity-key-recovery-migration.md @@ -0,0 +1,135 @@ +# Scenario: Identity key recovery, migration, and key-placement consistency + +**Verifies:** A private key is found, signed with, restored, and removed +consistently regardless of which internal convention filed it; a legacy +(pre-1.0) install's stranded keys can be recovered without re-entering +secrets; the Key Info / "Manage keys" screen is reachable from the standard +identity view and from a masternode's own detail page, and the two never +disagree about a key's role or held-state. (Risk area: #941, #945, #946, +#948 — key-placement-resolution rework, issue #889.) + +**Tier justification:** Needs a real, previously-migrated v0.9.3-shaped +identity/masternode fixture, real password-prompt cancel timing, and +cross-screen navigation (masternode detail page ↔ Key Info ↔ identities keys +list) that a no-display harness can't drive end-to-end. + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`) — do not skip or +alter steps between runs. Record what actually happens on each build; do not +presuppose which behaviors are "the fix." + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet + - `E2E_MN_PROTX_HASH` — (optional) a testnet masternode ProTxHash this + wallet controls, for the masternode-side half of this scenario +- A password-protected standalone identity (create one via the identity's + key-protection settings if none exists) — needed for the password-cancel + step +- Ideally, an identity/masternode with data left over from a pre-1.0 + install (or one seeded to resemble that state) — if unavailable, note the + gap rather than fabricating one +- Verify exact widget names during execution — the labels below (e.g. + "Manage keys", "Restore") describe intent, not a confirmed literal string + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/identity-key-recovery.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Keys screen reachability.** Open an identity in the default (least + advanced) interface level. Look for a way to reach a screen listing the + identity's keys (e.g. under Settings) without switching to a more + advanced level and without starting a payment. Record whether it's + reachable, and from where. +2. **Masternode key screen.** Open a masternode's detail page and look for + an equivalent "manage keys" entry point. Record what it opens. +3. **Role-label consistency.** Pick a key visible on both the identity's own + keys list and (if applicable) the masternode detail page. Record the + role name shown in each place — are they the same wording? +4. **Held-but-unpublished key.** If a key is saved on this device but not + on any on-chain key list for the identity, look for where the app + surfaces it. Record whether it's visible/reachable anywhere, and what + options are offered (view, remove). +5. **Restore stranded keys.** On an identity/masternode with legacy data + holding keys not yet in the current store, look for a "restore" affordance + on both the masternode detail page and the identity's own keys screen. + Trigger it, restore one key, and record: does it now show as held? Does + the offer narrow or disappear once nothing is left? Run it again and + record whether it's a safe no-op. +6. **Password-protected restore, cancel path.** On the password-protected + identity, start a restore and cancel/dismiss the password prompt. + Record the state of the Restore control afterward — does it look ready + to try again, or does it look stuck/already-started? +7. **Password prompt necessity.** For a key held in the clear in one place + and password-protected in another (if your fixture has this shape), + use/sign with it and record whether a password prompt appears. +8. **Same-numbered-key collision.** If reachable in your fixture (a + masternode's own record and its linked voting identity sharing a key + number), enter/paste a private key for one of the two same-numbered keys. + Record what happens to *both* keys afterward — check each one's own page, + not just the confirmation banner. +9. **Key removal.** Remove a saved private key. Record: does the key's page + immediately reflect the removal, or does it require leaving and + returning? Can the removed key still be revealed/signed with afterward? +10. **Live update while a key's page is open.** With a key's page open, + trigger a restore or edit for the same identity from elsewhere (a + second window on the same data dir, or via Back/forward navigation), + then return to the open key's page and make an edit. Record whether the + other change survives. + +## Safety constraints specific to this scenario + +- Use a real WIF/private key only for a throwaway testnet + identity/masternode fixture — never a mainnet key, never a value pasted + into a scenario file, report, or commit. +- Step 8 can destroy a key's private half if something goes wrong — rehearse + first with a fixture that has a recovery path (mnemonic-derived), not a + hand-imported unique WIF with no backup. + +## Expected outcome / pass criteria + +Record the observed behavior for each build (baseline, then HEAD) at every +step above, then apply this rule: + +- **BLOCKING** only if HEAD behaves *worse than baseline* in the happy path + described above (e.g. a key that was findable/usable on baseline is no + longer findable/usable on HEAD; a restore that worked cleanly on baseline + now corrupts or loses a key on HEAD), or if any step reveals a **data-loss** + outcome (a key's private half becoming permanently unrecoverable) on + either build. +- **NOT blocking**: a timing-dependent race that only sometimes reproduces; + a UI glitch with no functional consequence; or an issue that reproduces + identically on both builds (pre-existing — note it, don't block on it). +- If a step's affordance doesn't exist at all on baseline (e.g. the Keys + screen or the restore offer are new features introduced within this + diff), that's expected — record it as "not present on baseline, present + on HEAD" rather than a failure of the baseline run. + +## Known gotchas + +- Verifying a restored voting key against its actual on-chain address is an + explicitly deferred follow-up (issue #942) on both builds where the + feature exists at all — a voting key on an unlinked separate voting + identity correctly being listed as "cannot be restored automatically" is + expected, not a defect. +- A voting key stored directly on the identity's own record (rather than a + separate voting identity) has a documented residual on HEAD: saving/ + removing it by hand can affect a same-numbered voting key on a linked + voting identity. After doing so, re-open the keys list and check every key + still reads as expected before treating anything odd here as a new + regression. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md new file mode 100644 index 000000000..f2736eb6d --- /dev/null +++ b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md @@ -0,0 +1,93 @@ +# Scenario: Shielded feature availability right after platform sync + +**Verifies:** The app detects the connected network's protocol version +successfully after SPV/platform sync completes, so shielded send/receive/ +shield/unshield become available as soon as the network supports them, +without a spurious generic error banner as a side effect of that detection. +(Risk area: #936, #938.) + +**Tier justification:** This is specifically about a real live-network +proof-verification call succeeding or failing against the actual connected +testnet — it cannot be simulated without a real SPV sync and a real +protocol-version-detection round trip. + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet +- Verify exact banner/error wording during execution rather than assuming + the text below is literal + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/shielded-availability.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Cold start.** Launch the app fresh (isolated data dir, no prior sync + state) and wait for SPV sync to complete. Record whether any generic + error banner (e.g. "An unexpected error occurred") appears purely as a + side effect of sync completing. +2. **Shielded action availability.** Once sync completes, record whether + Shield, Unshield, Send Privately (shielded), and shielded Receive are + reachable/enabled. +3. **Send-fee estimate.** Open a Send flow and record whether a fee + estimate is shown, and whether it appears to update or stays fixed at a + default value. +4. **Network interruption during detection.** If you can simulate a brief + network outage right at startup, record whether the app keeps retrying + detection (shielded stays unavailable but the app doesn't misbehave) or + assumes a version without confirmation. +5. **Restart after successful detection.** Restart the app once shielded + features were confirmed available, and record whether they remain + available immediately on the next launch. + +## Safety constraints specific to this scenario + +- None beyond the standing rules. + +## Expected outcome / pass criteria + +Record the observed behavior for each build at every step, then apply this +rule: + +- **BLOCKING** only if HEAD is worse than baseline in the happy flow (e.g. + shielded features activate correctly after sync on baseline but stay + silently disabled on HEAD, or vice versa in a way that matters for this + release; a spurious error banner appears on HEAD where baseline was + clean), or a **data-loss** outcome (unlikely for this scenario, but note + if found) on either build. +- **NOT blocking**: a difference in exactly which internal call detects the + protocol version, as long as the user-visible outcome (shielded features + available when the network supports them, generic error banner absent) is + the same or better on HEAD; an issue reproducing identically on both + builds. +- If baseline's shielded features never activate at all after sync (the bug + this PR fixes), and HEAD's do — that's the expected direction of + improvement, not something to flag as a baseline "failure" beyond noting + it factually. + +## Known gotchas + +- The fee estimate may legitimately keep using a "last known rate" rather + than a freshly fetched one on HEAD — this is a documented temporary + limitation of the fix, not itself a defect, as long as it's not worse than + baseline's own fee-estimate behavior. +- If testing via MCP/CLI tools rather than the GUI, the same underlying + protocol-version-caching code path applies — a caching bug here previously + affected `mcp::resolve`-backed tools too. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md new file mode 100644 index 000000000..be051188d --- /dev/null +++ b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md @@ -0,0 +1,126 @@ +# Scenario: "Max" across asset-lock funding and Transfer/Withdraw flows + +**Verifies:** "Max" (and its dispatch-time validation) reflects what the +wallet can actually build/send, across every flow that offers a Max button: +Shield DASH, Fund Platform Address (Simple builder-driven form), Send +directly to an identity, Create Identity / Top Up Identity from wallet +balance or a received deposit, and identity Transfer/Withdraw. Must include +a near-empty ("dust") wallet variant and a rapid-UTXO-composition-change +variant specifically — these are real edge cases in this diff and are not +reliably described by the CHANGELOG text. (Risk area: #937, #927.) + +**Tier justification:** Needs a live, synced SPV wallet with real UTXOs +(confirmed, unconfirmed, dust-only) and a real broadcast to prove Max +actually sends on the first try — the ceiling query and the fee-reserve +estimator both depend on genuine wallet/network state a no-display harness +can't fake. + +**Run against BOTH builds with this identical procedure** (baseline +`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). Describe what you +observe on each build without assuming which one is "correct." + +## Prerequisites + +- Network: testnet +- Environment variables (names only): + - `E2E_WALLET_MNEMONIC` — funded testnet wallet +- A wallet with an unusually large number of small UTXOs, if available + (otherwise send several small amounts to the test wallet first) +- A second, separate near-empty wallet (drained close to the network dust + threshold) for the dust-wallet variant +- A registered, funded identity (for the Transfer/Withdraw half) +- Verify exact widget/button labels during execution before relying on the + names used below + +## Setup + +```bash +DATADIR=$(mktemp -d) +cp .env.example "$DATADIR/.env" +pgrep -af dash-evo-tool + +BIN= +test -x "$BIN" +LOG="$DATADIR/wallet-max-send.log" +DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & +``` + +## Procedure + +1. **Checking → Max → send, each asset-lock flow.** For each of: Shield + DASH, Fund Platform Address (Simple form), Send to identity, Create + Identity (fund from wallet balance), Top Up Identity (fund from wallet + balance) — open the flow, note whether/how the amount field indicates a + check is in progress, press Max, then send. Record whether the send + succeeds on the first attempt. +2. **Received-deposit ceiling.** On Create Identity or Top Up Identity, use + "fund from a received deposit." Record what ceiling Max offers relative + to what actually arrived at that specific deposit address versus the + wallet's total balance elsewhere. +3. **Advanced manual-input path.** On the Advanced manual-input + Platform-address flow, record what Max/validation is governed by (the + Core inputs selected, or something else). +4. **Stale-quote behavior.** Start a Max check on any flow above, then — + before sending — change the wallet's spendable funds from elsewhere + (send some of the same wallet's funds out, or receive a new deposit). + Record what the amount field does, and what the eventual send validates + against. +5. **Check-failure UX.** If you can force the availability check to fail + (e.g. briefly interrupt network during the check), record what the + amount field shows and whether a retry path is offered, and whether you + can still switch to a different funding method. +6. **Dust wallet.** Using the near-empty/dust wallet, open any flow above + and press Max. Record the result and whether it settles or stays + perpetually "checking." +7. **Rapid UTXO churn.** With a wallet whose balance is actively changing + (mid-sync, or receiving several small deposits in quick succession), open + a funding flow and watch the amount field for at least 30 seconds. Record + how many times it re-checks automatically and whether it settles or + keeps re-dispatching. +8. **Identity Transfer, Max.** Open Transfer credits between identities, + press Max, confirm and send. Record success/failure and the reserved fee + amount shown. +9. **Identity Withdraw, Max.** Open Withdraw to Core address, press Max, + confirm and send. Same recording. +10. **Reserve accuracy.** After steps 8–9, compare the amount actually + deducted (visible in the transaction/activity detail) to what Max + reserved. Record whether they're a close, sane match or wildly + different. + +## Safety constraints specific to this scenario + +- Cap every real send to the smallest amount that proves the behavior. +- Step 7 needs to run long enough (≥30s) to distinguish "one legitimate + automatic retry" from "runs away re-dispatching" — don't conclude early. + +## Expected outcome / pass criteria + +Record the observed behavior for each build at every step, then apply this +rule: + +- **BLOCKING** only if HEAD is worse than baseline in the happy flow (e.g. + Max sends successfully on baseline but fails/rejects on HEAD for the same + wallet state; HEAD's dust-wallet or rapid-churn case gets permanently + stuck where baseline at least settled; HEAD's fee reserve is wildly less + accurate than baseline's), or if a **data-loss** outcome appears on either + build. +- **NOT blocking**: a check that legitimately runs longer on one build than + the other without ever getting stuck; a UI wording difference with no + functional consequence; an issue reproducing identically on both builds. +- If baseline doesn't have a Max button on a flow at all where HEAD does (or + vice versa), record that as a feature-presence difference, not a + pass/fail on that step. + +## Known gotchas + +- The probe can hold the wallet-manager write lock for up to ~5 seconds per + call in a pathological case on HEAD — a real send attempted at the exact + same moment may appear to briefly stall rather than fail; wait a few + seconds before treating that as a hang. +- Issue #909 / `rust-dashcore#911` (Send Core→Core Max fails with + `InsufficientFunds` when change would be zero) is a distinct, real, + already-tracked, **still-open** upstream bug unrelated to this scenario's + asset-lock/Transfer/Withdraw Max — if a plain Core-to-Core "Max send" + reproduces it on both builds, that's pre-existing, not new. + +🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent From d327b55782d8ba6edee712b1706a9eba83518ce7 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Fri, 31 Jul 2026 17:12:19 +0000 Subject: [PATCH 2/3] fix(qa): resolve canonical remote/SHA, real fixture provisioning, and shared-state gaps in release-review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses review feedback on PR #952: - Never assume `origin` is the canonical repo — resolve the remote whose fetch URL actually matches it, since a fork checkout's `origin` can point elsewhere (confirmed: it silently diverged in one checkout). - Resolve the development branch to a commit SHA once in Phase 1 and use that SHA everywhere downstream; Phase 2b no longer builds a bare `HEAD`, which was ambiguous (and wrong) when this procedure runs from a feature checkout rather than the development branch itself. - Phase 0's fixture section no longer claims `.env.example` defines the E2E_* variable names (it doesn't) or that the GUI reads them (it doesn't either — confirmed via `AppSettings::default()` == Mainnet, and no `E2E_` references anywhere in `src/`). Reframed as a scenario-doc naming convention plus an explicit GUI provisioning sequence (network selection, wallet import, identity discovery) that a fresh data directory needs. - Phase 4 now calls out that isolated data directories don't give the two builds equivalent live-network starting state — fund-moving/name- registering steps need independently-funded fixtures or recorded before/after state, not a shared wallet run sequentially. - Softened the pre-existing-vs-diff-introduced check: an untouched file proves only that it wasn't directly edited, not that nothing upstream of it changed — old-vs-new reproduction is now framed as the primary evidence, the git-log check as corroboration only. - Removed hardcoded baseline-tag/branch references from all six scenario files, which contradicted their own "version-agnostic" framing and would go stale after the next release. - Fixed a scenario step that implied a DPNS name, a contract keyword, and a DashPay account label share one length limit — they're 63/50/100 chars respectively by design; the step now compares one field type at a time. Co-Authored-By: Claude Sonnet 5 --- .claude/skills/release-review/SKILL.md | 126 +++++++++++++----- .../dapi-budget-resilience-after-resync.md | 7 +- .../scenarios/dpns-registration-flow.md | 7 +- .../error-banners-identity-home-actions.md | 21 +-- .../identity-key-recovery-migration.md | 9 +- ...atform-shielded-availability-after-sync.md | 7 +- .../scenarios/wallet-max-send-asset-lock.md | 9 +- 7 files changed, 126 insertions(+), 60 deletions(-) diff --git a/.claude/skills/release-review/SKILL.md b/.claude/skills/release-review/SKILL.md index 901f534d6..04538ba34 100644 --- a/.claude/skills/release-review/SKILL.md +++ b/.claude/skills/release-review/SKILL.md @@ -53,12 +53,21 @@ a § Handling a Fixture/Infra Blocker pause. Don't rebuild from scratch: ## Phase 0 — Fixture preflight (do this FIRST, cheaply) Before committing to a real GUI run, verify the end-to-end test fixtures are actually usable *now* -— testnet masternodes deregister and test wallets drain between runs. Check the repo's `.env.example` -for the exact variable names in current use; as of this writing they include: - -- `E2E_MN_PROTX_HASH`: confirm it's a currently-registered masternode/evonode. A stale hash surfaces - as a typed `MasternodeNotFound` error the first time something touches it — don't wait for that; - check via an SDK/CLI lookup or a quick platform-explorer query first. +— testnet masternodes deregister and test wallets drain between runs. + +**These fixture variable names are a scenario-doc naming convention, not app config the GUI reads +automatically.** `.env.example` does not define them, and the GUI binary itself doesn't consume any +of them directly — confirm this hasn't changed before relying on it (`grep -rn "E2E_" src/` from a +repo checkout; as of this writing it returns nothing). They exist purely so scenario files and +operators refer to the same fixture by the same name, matching the naming already used by +`tests/backend-e2e/` (which *does* read `E2E_WALLET_MNEMONIC` for its own harness, but that's a +separate test binary, not this GUI). As of this writing, the scenarios under +`docs/gui-testing/scenarios/` use: + +- `E2E_MN_PROTX_HASH`: confirm the value you intend to use is a currently-registered + masternode/evonode before relying on it in a scenario — a stale hash surfaces as a typed + `MasternodeNotFound` error the first time something touches it in the GUI. Check via an SDK/CLI + lookup or a quick platform-explorer query first, since nothing enforces this automatically. - `E2E_WALLET_MNEMONIC`: confirm the wallet is actually funded on testnet *right now* (a live balance check, not "it was funded last time"). If empty, get a faucet drop (a testnet faucet — web, CLI, or whatever this project's e2e docs point at) or a fresh funded fixture BEFORE starting Phase 4 — @@ -69,46 +78,72 @@ Fixture/Infra Blocker below), and don't burn hours running non-fund-dependent ch with the user whether a partial (navigation-only) pass is worth doing now or the whole campaign should wait. -**Confirmed false-alarm trap**: before concluding a fixture is dead, verify the app's **network -selector** is actually set to Testnet. The dropdown has no explicit default after seeding a fresh -data directory from `.env.example` — a prior run spent real time diagnosing a "dead masternode" and -"unfunded wallet" that were both fine; the app was just sitting on Mainnet the whole time. Check the -network selector FIRST, before treating a zero balance or a `MasternodeNotFound` as a real fixture -problem. +**A fresh data directory does not start in a usable state for these scenarios — provisioning it is a +real, non-optional GUI setup sequence, not a formality.** Confirmed: a fresh install's default +settings select **Mainnet**, not Testnet, and a fresh data directory has no imported wallet, no +identities, and no masternode linkage. Before running any scenario against a freshly-seeded data +directory, drive the GUI itself through: selecting Testnet in the network selector, importing/ +restoring the wallet from the mnemonic fixture, discovering or importing the identities the scenario +needs, and — if the scenario needs it — linking a masternode or seeding legacy-migration state. Do +this explicitly and record it as setup, not as part of the scenario's own measured procedure. + +**Confirmed false-alarm trap**: precisely because the network selector has no reliable default, +double-check it's actually on Testnet before concluding a fixture is dead — a prior run spent real +time diagnosing a "dead masternode" and "unfunded wallet" that were both fine; the app was just +sitting on Mainnet the whole time. Check the network selector FIRST, before treating a zero balance +or a `MasternodeNotFound` as a real fixture problem. ## Phase 1 — Baseline identification + verified changelog -1. **Find the baseline release**: `gh release list --repo dashpay/dash-evo-tool` filtered to +1. **Resolve the canonical remote before running any remote-qualified git command.** In a fork-based + checkout, `origin` commonly points at a contributor's fork, not the canonical repo — using it + unconditionally can silently sync tags/branches from the wrong place with no error (confirmed: in + one checkout, `origin` resolved the development branch to a materially different commit than the + canonical repo did). Find the remote whose fetch URL actually matches the canonical repo, e.g. + `git remote -v | grep dashpay/dash-evo-tool`; if none exists, add one + (`git remote add upstream https://github.com/dashpay/dash-evo-tool.git`) or fetch directly by URL. + Call this `` below and use it consistently for every step in this skill — never + assume `origin` is it, and don't re-derive it more than once per run. +2. **Find the baseline release**: `gh release list --repo dashpay/dash-evo-tool` filtered to `draft:false && prerelease:true`, most recent. Automated builds are commonly cut as drafts first and can be deleted before undrafting (check `.github/workflows/` for the exact release-cutting workflow and tag-naming scheme currently in use) — a newer *local* tag with no matching GitHub release is not a valid baseline. -2. **Force-sync the tag from origin before trusting it**: - `git fetch origin "+refs/tags/:refs/tags/"`, then `git rev-parse ^{commit}`. - **Confirmed gotcha**: a local tag ref can silently diverge from origin (stale from an earlier - fetch/session) and point at an *ancestor* commit, producing a diff range that's wrong (larger than - reality) without any error. Always re-verify against `git ls-remote --tags origin ` before - computing any diff range from a tag. -3. **Determine the active development branch** — check the repo's own contribution docs (e.g. +3. **Force-sync the tag from `` before trusting it**: + `git fetch "+refs/tags/:refs/tags/"`, then + `git rev-parse ^{commit}`. **Confirmed gotcha**: a local tag ref can silently diverge from the + remote (stale from an earlier fetch/session) and point at an *ancestor* commit, producing a diff + range that's wrong (larger than reality) without any error. Always re-verify against + `git ls-remote --tags ` before computing any diff range from a tag. +4. **Determine the active development branch and resolve it to a commit SHA — record that SHA, don't + carry a branch name or bare `HEAD` forward.** Check the repo's own contribution docs (e.g. `CLAUDE.md`, `CONTRIBUTING.md`) rather than assuming `main`/`master`; this project may use a - dedicated long-lived dev branch instead. Compute the range against `origin/` (never - a possibly-stale local branch ref, and never the invoking session's own feature branch): - `git log ..origin/ --oneline`. -4. **Don't trust `CHANGELOG.md` alone.** Its unreleased section can contain stale/premature entries + dedicated long-lived dev branch instead. Resolve it once: + `git rev-parse /`, and treat that output as **the development SHA** + for the rest of this run — every later phase (the diff range, Phase 2b's second worktree, + screenshots, the final report) uses this exact SHA. Never substitute a bare `HEAD` for it (that's + the invoking checkout's HEAD, not necessarily the development branch's — wrong whenever this + procedure is run from a feature/PR checkout, which is common) and never re-resolve the branch ref + later in the run, since it can move while you're still working. Compute the range: + `git log .. --oneline`. +5. **Don't trust `CHANGELOG.md` alone.** Its unreleased section can contain stale/premature entries or miss real changes. For every commit in range, resolve its PR number and read the actual PR body (`gh pr view `) — not just the commit subject, which can undersell what a PR really changed (confirmed: one PR's CHANGELOG entry omitted two of its own later review-round fixes). Cross-check both directions: CHANGELOG claims with no backing PR, and merged PRs with no CHANGELOG entry. -5. **Confirm things are actually in-range, not pre-existing**: `git merge-base --is-ancestor +6. **Confirm things are actually in-range, not pre-existing**: `git merge-base --is-ancestor ` for anything you suspect might already be shipped — don't regression-test - pre-existing behavior as if it were new. -6. Produce a verified changelog grouped by theme, each entry citing commit SHA + PR number, tagged + pre-existing behavior as if it were new. This one check is necessary but not sufficient on its + own — see the fuller treatment in the blocker-classification section of Phase 4, which this same + caveat also applies to. +7. Produce a verified changelog grouped by theme, each entry citing commit SHA + PR number, tagged user-visible (GUI-relevant) vs internal-only (tests/CI/deps), plus a regression risk map by functional area. Save under your artifacts location for this run (see § Artifact Conventions). ## Phase 2 — Scoped code review -Review the diff `...origin/` (three-dot range) thoroughly: security, +Review the diff `...` (three-dot range, using the exact SHA resolved +in Phase 1) thoroughly: security, project/structural consistency, code quality, dependency risk, documentation accuracy. Run it from an isolated worktree, never the invoking session's active checkout — a plain `git diff`/`git log` between two refs doesn't need a checkout of either at all, but reviewing from a clean worktree @@ -137,9 +172,11 @@ argument for the new score. ## Phase 2b — Build both binaries for A/B comparison Create two isolated `git worktree`s under your scratch root — one at the baseline tag's commit, one -at HEAD — never inside the repo's own tracked working tree, and never let a spawned sub-agent create -worktrees on its own initiative if you're coordinating multiple agents (create them yourself first, -hand the paths out). Build each with: +at **the development SHA resolved in Phase 1** (never a bare `HEAD` — if this procedure is running +from a feature/PR checkout, `HEAD` there is that checkout, not the development branch you diffed and +wrote the changelog against) — never inside the repo's own tracked working tree, and never let a +spawned sub-agent create worktrees on its own initiative if you're coordinating multiple agents +(create them yourself first, hand the paths out). Build each with: ``` CARGO_TARGET_DIR=/-target cargo build --bin dash-evo-tool --manifest-path /Cargo.toml @@ -207,6 +244,19 @@ one real window/display target at a time, and it keeps screenshot provenance una app's log output for both runs (with an isolated data directory, verify where logs actually landed for this run rather than assuming a fixed default path). +**Isolated data directories give the two builds separate local state, but not separate live-network +state — plan for that separately.** A scenario that spends a UTXO, consumes a deposit, registers a +DPNS name, or moves credits mutates shared on-chain/live-network fixture state; running the baseline +binary through it and then the development binary through the identical flow means the second run +starts from state the first run already changed (an already-consumed deposit, a spent UTXO set, a +name that's no longer available), which can produce a false regression or a false pass. For any +scenario step that mutates fund-moving or name-registering state, do one of: give each build its +own independently-funded/equivalent fixture (separate wallet or identity per side) rather than +sharing one; or explicitly record each side's starting balances/UTXOs/deposits/identities/DPNS names +before that step and account for the difference when judging the result. Steps that only read state +(navigation, display checks) aren't affected — reserve this for anything that actually spends or +registers something. + **Blocker rule** (confirm with the user for each run — the default below is a starting point, not a universal constant): - **Blocking**: new version worse than old from the user's perspective **in the happy flow**, or a @@ -221,10 +271,16 @@ universal constant): **Blocking classification must say pre-existing vs diff-introduced, not just "blocking."** A defect can be blocking-worthy on its own severity (real data loss) while NOT being a regression this diff -caused. Before asserting "this release introduced X," check: `git log .. -- -` — zero commits touching the file means it's pre-existing, not diff-introduced. Frame -the finding as "we are choosing to hold/flag a pre-existing defect," not "this release broke X" — the -second is a factual claim that can be wrong and, if wrong, undermines the whole report's credibility. +caused. But don't let a single check decide this either way: `git log .. +-- ` returning zero commits proves only that the file wasn't *directly* edited — a +changed caller, a shared model type, a persisted-data format change, a feature gate, or a bumped +dependency can still cause a regression that surfaces in an otherwise-untouched file. Treat old-vs-new +reproduction as the primary evidence (does the exact same trigger actually behave differently on the +two builds?), and use the git-log check as one corroborating signal, not proof by itself — if the +file is untouched but you suspect the behavior still changed, trace the transitive callers/ +dependencies of the code path before concluding "pre-existing." Frame the finding as "we are choosing +to hold/flag a pre-existing defect," not "this release broke X" — the second is a factual claim that +can be wrong and, if wrong, undermines the whole report's credibility. Emit confirmed findings in whatever findings format the rest of this campaign's report uses (see Phase 2/5) — at minimum: title, severity, a location reference (a file:line if one applies, otherwise diff --git a/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md index 6d686e68b..07764b2fb 100644 --- a/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md +++ b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md @@ -13,8 +13,9 @@ cannot be simulated in `kittest` (no network) or isolated to a single `backend_task` call (the bug is about a background task's cumulative effect across transitions, not any single call's return value). -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here). ## Prerequisites @@ -32,7 +33,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/dapi-budget-resilience.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & diff --git a/docs/gui-testing/scenarios/dpns-registration-flow.md b/docs/gui-testing/scenarios/dpns-registration-flow.md index cefc25e25..36ec8e179 100644 --- a/docs/gui-testing/scenarios/dpns-registration-flow.md +++ b/docs/gui-testing/scenarios/dpns-registration-flow.md @@ -15,8 +15,9 @@ from actual identity data, and real async save timing (a save in flight when the user switches identity) — none of which a no-display harness can drive with a live contest clock and real background-task scheduling. -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here). ## Prerequisites @@ -37,7 +38,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/dpns-registration.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & diff --git a/docs/gui-testing/scenarios/error-banners-identity-home-actions.md b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md index 757f1e3b9..c2ceaeafc 100644 --- a/docs/gui-testing/scenarios/error-banners-identity-home-actions.md +++ b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md @@ -14,8 +14,9 @@ exercise the worker-thread stack-size fix, and real visual confirmation of button count/labels/layout that a no-display harness can assert on logic but not on what actually renders. -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here). ## Prerequisites @@ -35,7 +36,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/identity-home-actions.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & @@ -65,11 +66,15 @@ DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & in a row (fund an identity via deposit two or three times). Record whether the app crashes, and check `det-stderr.log`/`det.log` for a panic marker (`location=...`) even if the UI appeared fine. -7. **Format-validation consistency.** Trigger a format-validation error the - same way in two different places that should share the same rule (e.g. - an overlong DPNS name in registration, and the same length rule via a - keyword/label field elsewhere). Record whether both reject at the same - limit. +7. **Format-validation consistency.** DPNS names, contract keywords, and + DashPay account labels each have their own documented length limit (they + are different fields with different rules by design — do not expect them + to reject at the same length). Instead, pick ONE of these fields and + trigger its format-validation error through two different entry points + that both accept it (e.g. the same overlong DPNS name attempted through + registration and, if another screen also accepts a DPNS name, through + that screen too). Record whether both entry points reject at the same + limit with the same message, for that one field. 8. **Error banner wording.** Trigger a database/parsing-style error if possible (e.g. an interrupted operation). Record whether the banner text is plain and actionable, or contains raw error strings/stack diff --git a/docs/gui-testing/scenarios/identity-key-recovery-migration.md b/docs/gui-testing/scenarios/identity-key-recovery-migration.md index 3695fa546..c1e5fb52e 100644 --- a/docs/gui-testing/scenarios/identity-key-recovery-migration.md +++ b/docs/gui-testing/scenarios/identity-key-recovery-migration.md @@ -13,9 +13,10 @@ identity/masternode fixture, real password-prompt cancel timing, and cross-screen navigation (masternode detail page ↔ Key Info ↔ identities keys list) that a no-display harness can't drive end-to-end. -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`) — do not skip or -alter steps between runs. Record what actually happens on each build; do not +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here) — do not skip or alter +steps between runs. Record what actually happens on each build; do not presuppose which behaviors are "the fix." ## Prerequisites @@ -41,7 +42,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/identity-key-recovery.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & diff --git a/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md index f2736eb6d..0f6a118d3 100644 --- a/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md +++ b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md @@ -11,8 +11,9 @@ proof-verification call succeeding or failing against the actual connected testnet — it cannot be simulated without a real SPV sync and a real protocol-version-detection round trip. -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here). ## Prerequisites @@ -29,7 +30,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/shielded-availability.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & diff --git a/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md index be051188d..fa3aa14b8 100644 --- a/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md +++ b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md @@ -15,9 +15,10 @@ actually sends on the first try — the ceiling query and the fee-reserve estimator both depend on genuine wallet/network state a no-display harness can't fake. -**Run against BOTH builds with this identical procedure** (baseline -`v1.0.0-weekly.20260721`, then HEAD `origin/v1.0-dev`). Describe what you -observe on each build without assuming which one is "correct." +**Run against BOTH builds with this identical procedure** — the baseline and +development binaries selected for the current campaign (record their exact +SHAs in that campaign's own artifacts, not here). Describe what you observe +on each build without assuming which one is "correct." ## Prerequisites @@ -39,7 +40,7 @@ DATADIR=$(mktemp -d) cp .env.example "$DATADIR/.env" pgrep -af dash-evo-tool -BIN= +BIN= test -x "$BIN" LOG="$DATADIR/wallet-max-send.log" DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & From 4746e137723a0618dfdaea5c426c75b6d339c40d Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Mon, 31 Aug 2026 09:39:04 +0000 Subject: [PATCH 3/3] fix(qa): close second review round on release-review skill and scenarios Addresses 14 unresolved PR review comments (coderabbitai + thepastaclaw) verified against current HEAD, all confirmed still live before fixing: release-review/SKILL.md: - Fail closed on baseline reuse: compare the peeled remote tag SHA against the recorded baseline SHA, not just the release/tag name - Pin --repo dashpay/dash-evo-tool on every gh call; require an exact canonical-remote fetch-URL match instead of a loose substring grep - Compare peeled-to-peeled when verifying annotated release tags (the previous check mismatched on every correctly-signed tag in this repo) - Fetch the development branch before resolving its SHA, and require a named remote (drop the incompatible fetch-by-bare-URL fallback) - Add preventive secret controls (masked input, no secrets in argv/env dumps/logs, pre-embedding scan, restrictive permissions) ahead of the existing post-incident guidance; correct a misleading .gitignore claim - Narrow the "not blocking" race/timing exemption so a new regression can't hide behind "it's just a race" - Remove the record-and-account-for-the-difference escape hatch for mutating A/B fixtures; require an equivalent fixture or restored snapshot instead - Add E2E_IDENTITY_ID to the Phase 0 fixture preflight inventory - Tag the build-command fence as shell (MD040) docs/gui-testing/: define a single "A/B build comparison contract" in README.md (build-neutral roles, narrowed blocker rule, mandatory fixture equivalence) referenced from all six scenario files instead of duplicating it, then close each scenario's own specific gap: - Remove diff-relative/fix-direction language from 5 scenario files (identity-key-recovery-migration, dapi-budget-resilience-after-resync, error-banners-identity-home-actions, platform-shielded-availability- after-sync, dpns-registration-flow) - dpns-registration-flow: add an explicit precedence rule so intermittent wrong-identity attribution can't be waived under the timing-sensitive non-blocking clause - identity-key-recovery-migration: same-numbered key-collision data loss stays blocking regardless of build, never pre-waived as an expected HEAD residual - dapi-budget-resilience-after-resync + error-banners-identity-home- actions: log-check steps now target $LOG/$DATADIR explicitly and confirm the file exists before recording a negative result - dapi-budget-resilience-after-resync: rename "Baseline Platform Info check" to "Initial Platform Info check" (the step runs on both builds) 7 of 21 original threads were already resolved in the prior round and are unchanged by this commit. Co-Authored-By: Claude Sonnet 5 --- .claude/skills/release-review/SKILL.md | 140 +++++++++++++----- docs/gui-testing/README.md | 47 +++++- .../dapi-budget-resilience-after-resync.md | 42 ++++-- .../scenarios/dpns-registration-flow.md | 28 +++- .../error-banners-identity-home-actions.md | 31 ++-- .../identity-key-recovery-migration.md | 35 +++-- ...atform-shielded-availability-after-sync.md | 13 +- .../scenarios/wallet-max-send-asset-lock.md | 5 + 8 files changed, 249 insertions(+), 92 deletions(-) diff --git a/.claude/skills/release-review/SKILL.md b/.claude/skills/release-review/SKILL.md index 04538ba34..b7c78f7eb 100644 --- a/.claude/skills/release-review/SKILL.md +++ b/.claude/skills/release-review/SKILL.md @@ -28,13 +28,19 @@ instructions, the final report). The trigger is usually "a fix/PR landed, redo the campaign against the new HEAD" or resuming after a § Handling a Fixture/Infra Blocker pause. Don't rebuild from scratch: -1. **Re-verify the baseline is still current** (`gh release list` — a newer pre-release may have - shipped since the prior run). If unchanged, the prior run's worktree/binary for it is still - valid — check for an existing `git worktree` and a previously-built binary under your scratch - root before rebuilding anything. -2. **Only rebuild what moved.** If the baseline is unchanged and only HEAD advanced, reuse the - baseline binary as-is; `git fetch` + `git checkout ` the HEAD worktree in place and - rebuild only that binary. This turns a two-cold-build Phase 2b into one incremental build. +1. **Re-verify the baseline is still current** (`gh release list --repo dashpay/dash-evo-tool` — a + newer pre-release may have shipped since the prior run). A matching tag *name* is not enough: + re-resolve the canonical peeled tag commit + (`git ls-remote "refs/tags/^{}"`) and compare it against the baseline SHA + recorded in this run's `qa-run-state.md`. Only when both the tag name and that SHA are unchanged + is the prior run's worktree/binary for it still valid — check for an existing `git worktree` and a + previously-built binary under your scratch root before rebuilding anything. Fail closed on a + mismatch, and equally when no baseline SHA was recorded: re-establish the baseline per Phase 1 and + rebuild its binary rather than reusing an artifact you can't prove. +2. **Only rebuild what moved.** If the baseline is unchanged *by that SHA comparison* and only HEAD + advanced, reuse the baseline binary as-is; `git fetch` + `git checkout ` the HEAD + worktree in place and rebuild only that binary. This turns a two-cold-build Phase 2b into one + incremental build. 3. **Archive, don't overwrite, the prior run's output.** Before writing fresh Phase 1-5 artifacts, move the previous run's files into a subfolder (e.g. `run-vs-/`) under the same artifacts location. Both runs stay comparable/auditable, and nothing silently clobbers a @@ -42,10 +48,10 @@ a § Handling a Fixture/Infra Blocker pause. Don't rebuild from scratch: 4. **Reuse scenario files and prior findings as a starting point, not from scratch.** A changelog pass can diff the prior run's verified commit list against the new range instead of re-deriving the whole thing; a GUI pass can decide per-scenario whether the flow is unchanged enough to reuse - the prior run's OLD-side (baseline) screenshots rather than recapturing them — an unchanged - baseline binary means its screenshots are still valid evidence. Always capture fresh NEW-side - screenshots (HEAD moved), and log explicitly which screenshots were reused vs freshly captured so - it's auditable rather than assumed. + the prior run's OLD-side (baseline) screenshots rather than recapturing them — a baseline binary + unchanged by step 1's peeled-SHA comparison (not merely by tag name) still has valid screenshots. + Always capture fresh NEW-side screenshots (HEAD moved), and log explicitly which screenshots were + reused vs freshly captured so it's auditable rather than assumed. 5. If any part of this run is delegated (to a sub-agent or a fresh session), give it the prior run's key facts (SHAs, decisions already made, artifact paths) explicitly — it has no memory of a prior or paused run and will otherwise re-derive, and burn time on, things already verified. @@ -72,8 +78,13 @@ separate test binary, not this GUI). As of this writing, the scenarios under check, not "it was funded last time"). If empty, get a faucet drop (a testnet faucet — web, CLI, or whatever this project's e2e docs point at) or a fresh funded fixture BEFORE starting Phase 4 — don't discover this after a long SPV sync. +- `E2E_IDENTITY_ID`: the identity the top-up check in + `docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md` funds. Preflight it with a live + identity lookup (does the ID still resolve, and can it receive a top-up?), or decide explicitly and + up front that provisioning will create the identity ad hoc — that costs credits and GUI time, so + it's a decision to make before Phase 4, not to discover mid-scenario. -If either fixture is broken and can't be fixed quickly: stop here, log it (see § Handling a +If any of these fixtures is broken and can't be fixed quickly: stop here, log it (see § Handling a Fixture/Infra Blocker below), and don't burn hours running non-fund-dependent checks only — check with the user whether a partial (navigation-only) pass is worth doing now or the whole campaign should wait. @@ -99,9 +110,14 @@ or a `MasternodeNotFound` as a real fixture problem. checkout, `origin` commonly points at a contributor's fork, not the canonical repo — using it unconditionally can silently sync tags/branches from the wrong place with no error (confirmed: in one checkout, `origin` resolved the development branch to a materially different commit than the - canonical repo did). Find the remote whose fetch URL actually matches the canonical repo, e.g. - `git remote -v | grep dashpay/dash-evo-tool`; if none exists, add one - (`git remote add upstream https://github.com/dashpay/dash-evo-tool.git`) or fetch directly by URL. + canonical repo did). Find the remote whose *fetch URL* is exactly the canonical repo: read the URL + itself (`git remote get-url ` per remote, or the `(fetch)` line of `git remote -v`) and + require an exact match on `github.com/dashpay/dash-evo-tool` — HTTPS or SSH form, with or without + the `.git` suffix. A loose `git remote -v | grep dashpay/dash-evo-tool` is not that check: it also + matches a push-only line and any fork whose URL merely contains the string. If no remote matches, + add a **named** one (`git remote add upstream https://github.com/dashpay/dash-evo-tool.git`) and + fetch through that name — never fetch directly by bare URL, since step 4 resolves + `/` and a URL leaves no remote-tracking ref to rev-parse. Call this `` below and use it consistently for every step in this skill — never assume `origin` is it, and don't re-derive it more than once per run. 2. **Find the baseline release**: `gh release list --repo dashpay/dash-evo-tool` filtered to @@ -113,24 +129,38 @@ or a `MasternodeNotFound` as a real fixture problem. `git fetch "+refs/tags/:refs/tags/"`, then `git rev-parse ^{commit}`. **Confirmed gotcha**: a local tag ref can silently diverge from the remote (stale from an earlier fetch/session) and point at an *ancestor* commit, producing a diff - range that's wrong (larger than reality) without any error. Always re-verify against - `git ls-remote --tags ` before computing any diff range from a tag. + range that's wrong (larger than reality) without any error. Always re-verify against the remote + before computing any diff range from a tag — and compare peeled commit to peeled commit: + `git ls-remote "refs/tags/^{}"` returns the commit a tag points at, whereas + `git ls-remote --tags ` returns the *tag object* SHA for an annotated tag. + This project's release tags are annotated, so comparing the local peeled commit against the + unpeeled remote line mismatches on every healthy tag — that mismatch is an artifact of the wrong + comparison, not evidence of drift. (If the remote prints no `^{}` line the tag is lightweight and + its single SHA is already the commit. To check both forms explicitly, compare tag object against + `git rev-parse ` and peeled commit against `git rev-parse ^{commit}` as two separate + assertions.) 4. **Determine the active development branch and resolve it to a commit SHA — record that SHA, don't carry a branch name or bare `HEAD` forward.** Check the repo's own contribution docs (e.g. `CLAUDE.md`, `CONTRIBUTING.md`) rather than assuming `main`/`master`; this project may use a - dedicated long-lived dev branch instead. Resolve it once: - `git rev-parse /`, and treat that output as **the development SHA** - for the rest of this run — every later phase (the diff range, Phase 2b's second worktree, - screenshots, the final report) uses this exact SHA. Never substitute a bare `HEAD` for it (that's - the invoking checkout's HEAD, not necessarily the development branch's — wrong whenever this - procedure is run from a feature/PR checkout, which is common) and never re-resolve the branch ref - later in the run, since it can move while you're still working. Compute the range: + dedicated long-lived dev branch instead. Fetch that branch before resolving it — Phase 1's other + fetch syncs tags only, and a stale remote-tracking ref freezes a wrong development SHA into every + later phase with no error (confirmed: a checkout's `/` sat behind + the canonical head that `git ls-remote` reported). Resolve it once: + `git fetch "+refs/heads/:refs/remotes//"` + then `git rev-parse /`, and treat that output as + **the development SHA** for the rest of this run — every later phase (the diff range, Phase 2b's + second worktree, screenshots, the final report) uses this exact SHA. Never substitute a bare + `HEAD` for it (that's the invoking checkout's HEAD, not necessarily the development branch's — + wrong whenever this procedure is run from a feature/PR checkout, which is common) and never + re-resolve the branch ref later in the run, since it can move while you're still working. Compute + the range: `git log .. --oneline`. 5. **Don't trust `CHANGELOG.md` alone.** Its unreleased section can contain stale/premature entries or miss real changes. For every commit in range, resolve its PR number and read the actual PR body - (`gh pr view `) — not just the commit subject, which can undersell what a PR really changed - (confirmed: one PR's CHANGELOG entry omitted two of its own later review-round fixes). Cross-check - both directions: CHANGELOG claims with no backing PR, and merged PRs with no CHANGELOG entry. + (`gh pr view --repo dashpay/dash-evo-tool`) — not just the commit subject, which can undersell + what a PR really changed (confirmed: one PR's CHANGELOG entry omitted two of its own later + review-round fixes). Cross-check both directions: CHANGELOG claims with no backing PR, and merged + PRs with no CHANGELOG entry. 6. **Confirm things are actually in-range, not pre-existing**: `git merge-base --is-ancestor ` for anything you suspect might already be shipped — don't regression-test pre-existing behavior as if it were new. This one check is necessary but not sufficient on its @@ -178,7 +208,7 @@ wrote the changelog against) — never inside the repo's own tracked working tre spawned sub-agent create worktrees on its own initiative if you're coordinating multiple agents (create them yourself first, hand the paths out). Build each with: -``` +```shell CARGO_TARGET_DIR=/-target cargo build --bin dash-evo-tool --manifest-path /Cargo.toml ``` @@ -250,19 +280,24 @@ DPNS name, or moves credits mutates shared on-chain/live-network fixture state; binary through it and then the development binary through the identical flow means the second run starts from state the first run already changed (an already-consumed deposit, a spent UTXO set, a name that's no longer available), which can produce a false regression or a false pass. For any -scenario step that mutates fund-moving or name-registering state, do one of: give each build its -own independently-funded/equivalent fixture (separate wallet or identity per side) rather than -sharing one; or explicitly record each side's starting balances/UTXOs/deposits/identities/DPNS names -before that step and account for the difference when judging the result. Steps that only read state -(navigation, display checks) aren't affected — reserve this for anything that actually spends or -registers something. +scenario step that mutates fund-moving or name-registering state, give each build its own +independently-funded/equivalent fixture — a separate wallet, identity, or name per side, or a fixture +restored to an equivalent starting state between the two runs — rather than sharing one. Recording +each side's starting balances/UTXOs/deposits/identities/DPNS names and "accounting for the +difference" afterwards is not an alternative: you cannot account your way past a DPNS name that no +longer exists or a deposit the first run already consumed, because the second build can't execute the +flow at all. Steps that only read state (navigation, display checks) aren't affected — reserve this +for anything that actually spends or registers something. **Blocker rule** (confirm with the user for each run — the default below is a starting point, not a universal constant): - **Blocking**: new version worse than old from the user's perspective **in the happy flow**, or a **data-loss** scenario. -- **Not blocking**: concurrency/timing glitches; anything reproduced identically on **both** - builds (pre-existing — note it, don't flag as a regression). +- **Not blocking**: timing noise with no user-visible effect; anything reproduced identically on + **both** builds (pre-existing — note it, don't flag as a regression). Read that exemption + narrowly — a *new* intermittent race stays blocking whenever it breaks the happy flow, crashes the + app, or loses data. "It's a race" never downgrades those; intermittency only means you reproduce it + more times before confirming. - Reproduce anything about to be marked blocking at least twice before confirming. - If a repro attempt genuinely can't be reproduced after several tries, don't leave it open indefinitely — write up the negative evidence, downgrade to backlog, and say explicitly that repro @@ -336,6 +371,24 @@ apply — but check rather than assume on a shared machine. ## Security incidents during GUI testing +**Prevent first — every control below is cheaper than the incident response after it.** When a run +handles a real secret (a testnet mnemonic, a password fixture): +- Enter it through the GUI's own masked/protected input wherever one exists, rather than any path + that echoes the value back on screen. +- **Never pass a real mnemonic, password, or private key as a bare command-line argument** — it lands + in process listings, shell history, and this session's transcript, all of which outlive the run. + Pass it via an environment variable the app reads, or a file with restrictive permissions. +- Don't dump the environment (`env`, `printenv`, `set`) into a transcript, and redact fixture secrets + out of any log output before quoting it; check what a raised log level actually prints before + capturing it wholesale. +- **Check before saving or embedding, not after**: eyeball each screenshot for visible secret text, + and grep transcripts, logs, and generated artifacts for the fixture's own patterns (the mnemonic's + words, the password string, `xprv`/`tprv`/WIF prefixes) before they reach the artifacts location. + Repeat that check immediately before Phase 5's base64-inline step — inlining bakes the image into a + self-contained, trivially forwardable HTML file that is far harder to un-publish than a stray PNG. +- Restrict filesystem permissions where secrets and captures land (`chmod 700` the scratch/artifacts + directories, `chmod 600` any file holding a secret) — on the machine they actually land on. + If GUI testing handles real secrets (a testnet mnemonic, a password fixture) and something goes wrong — a screenshot with the secret visible, a secret typed into a visible log/transcript — the self-report is not the end of it: @@ -357,7 +410,9 @@ self-report is not the end of it: Build a self-contained HTML page (`visual-changelog.html`) presenting old/new screenshot pairs side by side per scenario/story, with a verdict badge and a one-line rationale each — inline the images -(e.g. base64) so the page has no external file dependencies. Save it to your artifacts location. +(e.g. base64) so the page has no external file dependencies. Run the pre-embedding secret check from +§ Security incidents during GUI testing over every image and caption first: once inlined, a leaked +secret travels with every copy of the page. Save it to your artifacts location. Merge Phase 4's findings into Phase 2's report as an additional section, re-validate the combined report against whatever schema/format it uses, and re-render any human-readable version. The combined report is then ready to hand to the user for triage — code-review and GUI-regression @@ -380,13 +435,16 @@ not an app bug), don't push through with silent scope cuts: ## Artifact conventions -Pick one consistent, git-ignored location outside the tracked repo for everything durable in a given -run (a dated directory under your scratch root, or wherever your own setup keeps generated reports) -and use it for: the verified changelog, the code-review report (JSON/Markdown or whatever format is +Pick one consistent location outside the tracked repo for everything durable in a given run (a dated +directory under your scratch root, or wherever your own setup keeps generated reports) and use it +for: the verified changelog, the code-review report (JSON/Markdown or whatever format is in use), GUI findings, the GUI test log, `visual-changelog.html`, screenshots, and the user-stories checklist. If your environment has a way to share generated files with the user (a hosting mechanism, an upload/artifact tool, or simply pointing at local paths they can open), use whatever's actually -available — don't assume a specific serving mechanism exists. +available — don't assume a specific serving mechanism exists. Note that "git-ignored" is not a safety +property here: `.gitignore` only governs paths inside the repository, so a directory outside it needs +its own access controls (restrictive permissions, and not sitting under anything published or synced) +rather than an ignore rule. Keep a small `qa-run-state.md` at the top of that location as a durable scratchpad: verified SHAs, worktree/binary paths, a phase checklist, and an explicit "open items for the user" list. That's what diff --git a/docs/gui-testing/README.md b/docs/gui-testing/README.md index 2c3a62548..f66a7b5bd 100644 --- a/docs/gui-testing/README.md +++ b/docs/gui-testing/README.md @@ -251,13 +251,50 @@ one scenario silently invalidating another: run spanning hours, set `CARGO_TARGET_DIR` to a private path before building and hash-verify (`sha256sum`) before each relaunch. +## A/B build comparison contract + +Every scenario in this library is run twice with the identical procedure: once +against the **baseline** binary and once against the **development** binary +selected for the current campaign. Both are roles, not specific versions — +record the two exact commit SHAs in that campaign's own artifacts, never in a +scenario file, so the library stays valid for the next campaign too. + +**Judge by observation, not by expectation.** Record what each build actually +does at every step before classifying anything. Never assume which build is +"correct", which behavior a given change was meant to introduce, or which +direction a difference is supposed to point. + +**Blocker rule** (a campaign may narrow this, never silently widen it): + +- **Blocking**: the development build is worse than the baseline build from the + user's perspective in the happy flow, or a **data-loss** outcome on either + build. +- **Not blocking**: a purely cosmetic difference, or behavior reproduced + identically on **both** builds (pre-existing — note it, don't flag it as a + regression). Intermittency is not an exemption: a race that reproduces on one + build and not the other is still a regression, and timing sensitivity only + changes how many attempts a repro needs. Reproduce anything about to be + marked blocking at least twice before confirming it. + +**Equivalent fixture state per build.** Isolated data directories give the two +builds separate *local* state, but not separate *live-network* state. Any step +that spends a UTXO, consumes a deposit, registers a DPNS name, or moves credits +mutates shared on-chain state, so the second build's run would otherwise start +from whatever the first run left behind — an already-consumed deposit, a spent +UTXO set, a name that is no longer available — producing a false regression or +a false pass. Before any such step, give each build either its own +independently-funded, equivalent fixture (a separate wallet/identity per side) +or a restored snapshot of the same starting state, and record each side's +starting balances/deposits/identities/names first. Recording the difference and +accounting for it afterwards is **not** sufficient. Steps that only read state +(navigation, display checks) are unaffected. + ## Scenario index -These six scenarios are written **version-agnostic**: the same procedure -runs unmodified against both the baseline release build and the current -`v1.0-dev` build for an A/B comparison, with the blocker rule (worse than -baseline in the happy flow, or data loss, is blocking; concurrency/glitches -and issues present on both builds are not) baked into each file's own +These six scenarios are written **build-neutral**: the same procedure runs +unmodified against either binary of an A/B campaign, with the blocker rule and +the fixture requirement from the [A/B build comparison +contract](#ab-build-comparison-contract) above applied in each file's own "Expected outcome" section. | Scenario | What it verifies | diff --git a/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md index 07764b2fb..bb9ff6880 100644 --- a/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md +++ b/docs/gui-testing/scenarios/dapi-budget-resilience-after-resync.md @@ -17,6 +17,11 @@ across transitions, not any single call's return value). development binaries selected for the current campaign (record their exact SHAs in that campaign's own artifacts, not here). +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet @@ -41,11 +46,11 @@ DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & ## Procedure -1. **Baseline Platform Info check.** Open Platform Info (Developer/Advanced +1. **Initial Platform Info check.** Open Platform Info (Developer/Advanced view). Record the exact "Current Epoch Information" text, in particular whether the fee-multiplier line reads "unavailable" or states it is "a - fixed value, not read from the network" (or neither, if built against a - version predating both #936 and #950). + fixed value, not read from the network" (or neither, if the build under + test predates that line entirely). 2. **Force repeated `Syncing`↔`Synced` transitions.** With the app running and initial sync complete, force several resync cycles in the same session — e.g. toggle network connectivity briefly (airplane-mode-style @@ -60,10 +65,19 @@ DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & 4. **Repeat the top-up attempt once more** a short time later (a minute or two) without additional resync cycling, to distinguish "budget recovers on its own" from "budget stays exhausted." -5. **Check the logs.** Search `det.log`/`det-stderr.log` for - `DapiAllAddressesExhausted` or similar rate-limit/address-pool exhaustion - markers appearing near the resync cycles from step 2, independent of - whether the UI showed an error. +5. **Check the logs.** Confirm the log targets exist first — `ls -l "$LOG"` + and `ls -l "$DATADIR"`, which is where the app writes `det.log` / + `det-stderr.log` — then search them: + + ```bash + grep -R 'DapiAllAddressesExhausted' "$LOG" "$DATADIR" + ``` + + Look for that or similar rate-limit/address-pool exhaustion markers + appearing near the resync cycles from step 2, independent of whether the UI + showed an error. Record "not reproduced" only after confirming the files + exist and are non-empty — a missing log file means the check never ran, not + that the marker is absent. 6. **Re-check Platform Info.** Reopen Platform Info after the resync cycling and top-up attempts; record whether the epoch/fee-multiplier text is still consistent with step 1's observation (no crash, no stuck "loading" @@ -88,13 +102,13 @@ rule: banner on HEAD, or `DapiAllAddressesExhausted` (or equivalent) appears in HEAD's logs where it did not on baseline — or a **data-loss** outcome on either build. -- **NOT blocking**: the reverse direction is the *expected improvement* this - PR ships — if baseline's top-up fails after repeated resync cycling (the - bug #950 fixes) and HEAD's succeeds, that's the intended fix working, not - a baseline "failure" to flag beyond noting it factually. A wording-only - difference in the Platform Info fee-multiplier line (`unavailable` vs `a - fixed value, not read from the network`) is expected and not itself a - defect on either build. +- **NOT blocking**: the reverse direction — the baseline build's top-up + failing after repeated resync cycling while the development build's + succeeds. Record it factually as an observed difference between the two + builds; do not assume it was the intended outcome, and do not treat it as a + baseline "failure" to flag. A wording-only difference in the Platform Info + fee-multiplier line (`unavailable` vs `a fixed value, not read from the + network`) is not itself a defect on either build. - If neither build reproduces an exhaustion-triggered top-up failure within the attempted resync-cycle budget, record that plainly as "not reproduced this run" rather than asserting the fix is confirmed — a live-network diff --git a/docs/gui-testing/scenarios/dpns-registration-flow.md b/docs/gui-testing/scenarios/dpns-registration-flow.md index 36ec8e179..f386fccc2 100644 --- a/docs/gui-testing/scenarios/dpns-registration-flow.md +++ b/docs/gui-testing/scenarios/dpns-registration-flow.md @@ -19,6 +19,11 @@ drive with a live contest clock and real background-task scheduling. development binaries selected for the current campaign (record their exact SHAs in that campaign's own artifacts, not here). +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet @@ -100,13 +105,19 @@ rule: permanently stuck on HEAD where baseline cleared it; a save result gets attributed to the wrong identity on HEAD where baseline didn't), or a **data-loss** outcome (a profile edit silently lost) on either build. -- **NOT blocking**: a timing-sensitive race that needs several attempts to - reproduce and only sometimes shows on either build; wording differences - with no functional consequence; an issue reproducing identically on both +- **NOT blocking**: wording differences with no functional consequence; an + issue reproducing identically on both builds; a timing-sensitive race that + needs several attempts to reproduce and only sometimes shows on **both** builds. -- If baseline doesn't have a pending-registration indicator at all (a new - feature on HEAD), record that as a feature-presence difference, not a - step failure on baseline. +- **Precedence: intermittency never downgrades a wrong-identity result.** If a + save result is attributed to the wrong identity (step 10) on one build and + never on the other, it stays BLOCKING however rarely it reproduces — that + race is exactly what this scenario exists to catch, so being hard to land + inside the race window is a reason to attempt it more times, not a reason to + waive it under the timing-sensitive clause above. +- If the pending-registration indicator exists on only one of the two selected + builds, record that as a feature-presence difference, not a step failure on + the build that lacks it. ## Known gotchas @@ -115,6 +126,9 @@ rule: blocking the rest of this scenario. - The identity-switch race (step 10) is timing-sensitive; several attempts with the save action and the identity switch performed in quick succession - may be needed to land inside the race window on either build. + may be needed to land inside the race window on either build. That + difficulty does not soften the result: per the precedence rule in "Expected + outcome", a wrong-identity attribution seen on one build only is blocking + even if it reproduced once in many tries. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/error-banners-identity-home-actions.md b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md index c2ceaeafc..389fa93e4 100644 --- a/docs/gui-testing/scenarios/error-banners-identity-home-actions.md +++ b/docs/gui-testing/scenarios/error-banners-identity-home-actions.md @@ -18,6 +18,11 @@ not on what actually renders. development binaries selected for the current campaign (record their exact SHAs in that campaign's own artifacts, not here). +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet @@ -63,9 +68,18 @@ DASH_EVO_DATA_DIR="$DATADIR" nohup "$BIN" >"$LOG" 2>&1 & Record the exact message shown — does it say the deposit cannot be reused and to choose/start a different one, or does it suggest retrying? 6. **Deposit verification stability.** Perform several deposit verifications - in a row (fund an identity via deposit two or three times). Record - whether the app crashes, and check `det-stderr.log`/`det.log` for a - panic marker (`location=...`) even if the UI appeared fine. + in a row (fund an identity via deposit two or three times). Record whether + the app crashes, then check the logs for a panic marker even if the UI + appeared fine. Confirm the log targets exist first — `ls -l "$LOG"` and + `ls -l "$DATADIR"`, which is where the app writes `det.log` / + `det-stderr.log` — then search them: + + ```bash + grep -R -e 'panicked' -e 'location=' "$LOG" "$DATADIR" + ``` + + Record "no panic" only after confirming the files exist and are non-empty — + a missing log file means the check never ran, not that the app was clean. 7. **Format-validation consistency.** DPNS names, contract keywords, and DashPay account labels each have their own documented length limit (they are different fields with different rules by design — do not expect them @@ -101,8 +115,7 @@ rule: wording differences with no functional consequence; an issue reproducing identically on both builds (e.g. if baseline also shows 6 duplicate buttons and HEAD also does — note it, don't block). -- If baseline's button row differs in count/labels from HEAD's by design - (this PR's whole purpose is to collapse duplicates), record both +- If the two builds' button rows differ in count or labels, record both observations plainly rather than treating either as automatically wrong — apply the blocker rule based on functional correctness (do all buttons work, are destinations distinguishable), not on matching layouts. @@ -113,9 +126,9 @@ rule: at Identity Home, the identities-list popup, or the Withdrawal screen directly — these three surfaces historically used different gating criteria; note which surface any discrepancy is on. -- This PR range touches many files across validation, fee estimation, error - banners, and identity removal — if a defect surfaces, compare directly - against the baseline build's behavior for the same step before concluding - it's new. +- The change range this scenario covers touches many files across validation, + fee estimation, error banners, and identity removal — if a defect surfaces, + compare directly against the baseline build's behavior for the same step + before concluding it's new. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/identity-key-recovery-migration.md b/docs/gui-testing/scenarios/identity-key-recovery-migration.md index c1e5fb52e..39c2246ac 100644 --- a/docs/gui-testing/scenarios/identity-key-recovery-migration.md +++ b/docs/gui-testing/scenarios/identity-key-recovery-migration.md @@ -19,6 +19,11 @@ SHAs in that campaign's own artifacts, not here) — do not skip or alter steps between runs. Record what actually happens on each build; do not presuppose which behaviors are "the fix." +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet @@ -111,13 +116,15 @@ step above, then apply this rule: now corrupts or loses a key on HEAD), or if any step reveals a **data-loss** outcome (a key's private half becoming permanently unrecoverable) on either build. -- **NOT blocking**: a timing-dependent race that only sometimes reproduces; - a UI glitch with no functional consequence; or an issue that reproduces - identically on both builds (pre-existing — note it, don't block on it). -- If a step's affordance doesn't exist at all on baseline (e.g. the Keys - screen or the restore offer are new features introduced within this - diff), that's expected — record it as "not present on baseline, present - on HEAD" rather than a failure of the baseline run. +- **NOT blocking**: a timing-dependent race *with no data-loss consequence* + that only sometimes reproduces; a UI glitch with no functional consequence; + or an issue that reproduces identically on both builds (pre-existing — note + it, don't block on it). A race that strands or destroys a key's private half + stays blocking under the rule above, however rarely it reproduces. +- If a step's affordance exists on only one of the two selected builds (e.g. + the Keys screen or the restore offer is absent on one side), that is a + feature-presence difference, not a step failure on the build that lacks it + — record which build has it and compare the behavior only where both do. ## Known gotchas @@ -127,10 +134,14 @@ step above, then apply this rule: identity correctly being listed as "cannot be restored automatically" is expected, not a defect. - A voting key stored directly on the identity's own record (rather than a - separate voting identity) has a documented residual on HEAD: saving/ - removing it by hand can affect a same-numbered voting key on a linked - voting identity. After doing so, re-open the keys list and check every key - still reads as expected before treating anything odd here as a new - regression. + separate voting identity) can, when saved/removed by hand, affect a + same-numbered voting key on a linked voting identity. Never treat that as + expected upfront on either build: if it leaves a key's private half + unrecoverable it is the data-loss outcome step 8 exists to catch, and stays + **BLOCKING** regardless of which build shows it. Compare the observed + behavior across the two selected build SHAs, and record any confirmed + exception — with the evidence for it — in that campaign's own artifacts. In + every case, re-open the keys list after such a save/removal and check every + key still reads as expected. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent diff --git a/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md index 0f6a118d3..795d474e8 100644 --- a/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md +++ b/docs/gui-testing/scenarios/platform-shielded-availability-after-sync.md @@ -15,6 +15,11 @@ protocol-version-detection round trip. development binaries selected for the current campaign (record their exact SHAs in that campaign's own artifacts, not here). +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet @@ -76,10 +81,10 @@ rule: available when the network supports them, generic error banner absent) is the same or better on HEAD; an issue reproducing identically on both builds. -- If baseline's shielded features never activate at all after sync (the bug - this PR fixes), and HEAD's do — that's the expected direction of - improvement, not something to flag as a baseline "failure" beyond noting - it factually. +- If shielded features never activate after sync on one of the two selected + builds but do on the other, record that factually as an observed difference + between the builds — do not assume either direction was intended, and apply + the blocker rule to the user-visible outcome rather than to a presumed fix. ## Known gotchas diff --git a/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md index fa3aa14b8..91a10e1d6 100644 --- a/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md +++ b/docs/gui-testing/scenarios/wallet-max-send-asset-lock.md @@ -20,6 +20,11 @@ development binaries selected for the current campaign (record their exact SHAs in that campaign's own artifacts, not here). Describe what you observe on each build without assuming which one is "correct." +Before any step that spends funds, consumes a deposit, or registers a +name, each build needs its own independently-funded equivalent fixture (or +a restored snapshot of the same starting state) — see [A/B build comparison +contract](../README.md#ab-build-comparison-contract). + ## Prerequisites - Network: testnet