Verify recorded controller traces offline #42

User and workflow

A game-agent or automation developer has a recorded controller trace and wants to check that each controller command still produces the recorded state transition after changing an agent or SDK version, before connecting a real host or game.

Direct evidence

• Brobot author on Hacker News, dated 2025-11-09, describes firsthand game-automation work and says two recurring problems were scripts breaking after small GUI changes and “inability to test” automation before deployment.
• OpenAI Developer Community game-development thread, started 2026-01-29, includes a developer wishing they could ask an AI to play a game repeatedly and then discuss the results. This is a stated goal, not evidence of a tested workflow or measured defect.

The first source concerns GUI automation generally and the second is an exploratory game-testing idea. Neither establishes an OpenController incident or prevalence.

Repository gap and interpretation

OpenController records normalized commands with stateBefore and stateAfter in replay files. packages/cli/src/commands/replay.ts currently summarizes event counts and timestamp bounds; packages/cli/src/commands/replay-export.ts exports records. Neither verifies whether each logged command's immediate controller-state transition matches the recorded event.

Timed commands are logged at their state-transition boundaries. For example, press logs the pressed state after the adapter send and before waiting; its later release is a separate logged command. Therefore a verifier must not call the public timed controller.press() and compare state after that method returns: it would include the duration and already-completed release. Instead, it should apply each event's immediate state transition without waiting or sending to an adapter, then compare the resulting control state with that event's stateAfter.

A dry-run event-boundary verifier is a bounded test aid. It validates OpenController's recorded state transitions, not GUI automation, adapter delivery, host behavior, or game outcomes.

Proposed outcome

Add opencontroller replay verify <events.jsonl> that validates each command event independently at its logged transition boundary, without adapter dispatch or timer waits. For supported records, apply the immediate command transition to stateBefore and compare deterministic input-control fields with stateAfter. Report the first mismatch with event/line location and field paths.

Comparison rules

• Compare deterministic input controls: buttons, analogButtons, triggers, sticks, dpad, touchpad, and motion.
• Ignore and report as unverifiable: updatedAt, feedback, and status, which are timestamped or host/asynchronously sourced. Treat session identity/connection metadata as context, not an input-transition assertion.
• Require a valid profile for normalization. If a legacy event lacks stateBefore, stateAfter, or required command fields, mark that event unverifiable with a reason instead of guessing.
• Treat the logged auto-release as its own event; do not wait for the recorded durationMs or synthesize an additional release while validating the preceding press.

Acceptance criteria

• Verify recorded command/state transitions offline without opening a native adapter or sending host input.
• Validate immediate state at each command event boundary, including a nonzero-duration press followed by a separate release event; verification must not wait for the press duration.
• Compare only the deterministic input-control fields above. Clearly list ignored/unverifiable metadata in command output and docs.
• Detect malformed logs, unsupported commands/profiles, missing required state fields, and the first mismatch with one-based line/event location.
• Add fixtures for a valid timed press/release trace, a mismatched transition, malformed input, and a legacy/unverifiable event.
• Document that verification checks OpenController state semantics only and cannot validate adapter handoff or game behavior.

Tests and validation

Add package-local CLI tests plus a small JSONL fixture. Run CLI tests, typecheck, and package build. No model, API key, hardware, delay wait, adapter, or game should be required.

Non-goals

• No playback to physical devices, native bridges, games, or GUI automation tools.
• No deterministic model/agent execution, game-state capture, visual comparison, performance benchmark, or replay-format migration.
• No waiting for command durations or claiming that a recorded press was observed by a host/game.
• No change to opencontroller replay summary or replay export behavior.

Confidence and limits

Low to medium. One automation author reports practical fragility and a testability gap; a separate game-development discussion expresses a repeated-playtesting goal. Neither asks for OpenController replay verification, so this is a repository-fit inference. The verifier is limited to recorded SDK input-state transitions.

Ownership and dependencies

Lane: CLI replay verification, likely packages/cli/src/commands/replay-verify.ts, CLI dispatch/tests, and docs/replay-logs.md; factor a pure transition helper into core only if needed to avoid duplicating runtime semantics. This is independent of packages/core timed-press cancellation and the new MCP example. No issue dependency is required.