Skip to content

[bug][runtime] Derive durable-state action UUID from the plan-unique action name - #1094

Open
purushah wants to merge 1 commit into
apache:mainfrom
purushah:fix-action-state-key-stability
Open

[bug][runtime] Derive durable-state action UUID from the plan-unique action name#1094
purushah wants to merge 1 commit into
apache:mainfrom
purushah:fix-action-state-key-stability

Conversation

@purushah

@purushah purushah commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Linked issue: N/A (bug found during recovery testing)

Purpose of change

ActionStateUtil.generateUUIDForAction derives the durable-state key from Action.hashCode(). Action's hash folds in JavaFunction's Class[] parameterTypes, and Class.hashCode() is the per-JVM identity hash — so every durable-state key silently changes when the JVM restarts. After any failure recovery, lookups can never hit: every action re-executes instead of replaying, which breaks the exactly-once contract of the action-state store (and re-bills every LLM call in agent workloads).

This PR derives the UUID from action.getName(), which is unique within a plan and stable across processes.

How it was found

Kill/restore recovery experiments with a deliberately non-deterministic agent and the Kafka action-state store: 0/134 re-processed actions replayed from the store before this fix; 90/90 replayed with 0% divergence after. The existing test suite doesn't catch it because all tests run inside a single JVM, where identity hashes are stable.

Compatibility note

The key format changes, so state written by prior versions won't be found on restore. Those keys were already unreadable across restarts (that's the bug), so no working deployment regresses; single-process re-reads within one JVM lifetime are unaffected in practice because recovery is the only reader.

Tests

  • Two new regression tests in ActionStateUtilTest: (1) the action-UUID key segment equals UUID.nameUUIDFromBytes(action.getName()) — pins the derivation, so any future key-format change is a conscious, reviewed break of cross-restart state compatibility (verified to FAIL against the pre-fix code); (2) two separately constructed Action instances with the same name — what "the same action, after a JVM restart" looks like — produce identical state keys.
  • mvn test -pl runtime -am -Dtest='*ActionState*' — 129 tests, 0 failures (includes ActionStateUtilTest, KafkaActionStateStoreTest, Fluss store integration tests), on current main (rebased over [Bug] Recovered Durable ActionState for keys owned by other subtasks is never pruned #1024).
  • Cross-JVM behavior additionally validated by the kill/restore experiment above.

API

No public API changes.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: Claude Code (Claude Fable 5)

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Sep 3, 2026
…action name

Action.hashCode() folds in JavaFunction's Class[] parameterTypes, and
Class.hashCode() is the per-JVM identity hash, so every durable-state key
changes across a process restart and recovery lookups can never hit.
Kill/restore trials: 0/134 replays before this fix; 90/90 with 0% divergence
after (non-deterministic strategy, Kafka action-state store).
@purushah
purushah force-pushed the fix-action-state-key-stability branch from ba6d9e9 to 42ef26c Compare September 3, 2026 23:22
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant