Skip to content
32 changes: 26 additions & 6 deletions comfy_cli/workflow_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@
* **Last-writer-wins on widgets** — two concurrent writes to the same widget
converge on the value with the higher causal ``stamp`` ``[base_version, actor]``
(``op_id`` breaks exact ties into a total order), independent of apply order.
Ops share one stamp within an ``apply_specs`` batch, and ``op_id`` is minted
in increasing order, so the later write in a batch wins unless a batch guard
refuses the batch; amendment v1.14 also names a rebuild shape that still
depends on apply order (#1101).
The winning stamp per target is tracked in ``_widget_stamps`` (apply-only
bookkeeping, stripped before serialization).
* **Deterministic structure** — forking a shared subgraph definition mints an id
Expand Down Expand Up @@ -61,7 +65,9 @@
import math
import random
import re
import uuid
import secrets
import threading
import time
from decimal import Decimal
from typing import Any

Expand Down Expand Up @@ -388,10 +394,25 @@ def mint_id() -> int:
return _ID_FLOOR | random.getrandbits(52)


_last_op_clock = 0
_op_clock_lock = threading.Lock()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — This module-level threading.Lock isn't reinitialized after fork(). If a multithreaded host forks while another thread is inside _new_op_id's critical section, the child inherits a held lock that no surviving thread owns, and its first mint hangs forever. The window is narrow, but the fix is cheap: reset the lock with os.register_at_fork(after_in_child=...). Raised by 2 of 6 reviewers (gpt-5.6-sol-xhigh adversarial, gpt-5.6-sol-xhigh edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this. comfy-cli mints ops on one thread and does not fork while minting. The lock covers two integer operations. Adding an at-fork reset would guard a host this code does not run in.



def _new_op_id() -> str:
"""32 lowercase hex chars (§8.2): a nanosecond clock that never repeats or
goes backwards in this process, then 64 random bits for uniqueness. Every op
in an ``apply_specs`` batch shares one stamp, so ``op_id`` is what breaks the
tie; minting it in increasing order makes the later spec win."""
global _last_op_clock
with _op_clock_lock:
clock = _last_op_clock = max(time.time_ns(), _last_op_clock + 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Nit — The first half of every emitted op_id is now the minting host's wall-clock time in nanoseconds, where uuid4 was opaque. That timestamp travels to collaborators, comes back in acks, and lands wherever op_ids are persisted or embedded in derived ids. If that's intended, say so in the amendment so that edit-time disclosure is part of the wire contract; otherwise consider coarsening or offsetting the clock bits. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 50408f2: the "Scope of the ordering" paragraph now says the prefix carries the minting host's clock to the nanosecond wherever the op_id travels. The ops already carry the actor and base_version, and edit time is not sensitive in this collaboration model, so I kept the full clock rather than coarsening it.

return f"{clock:016x}{secrets.token_hex(8)}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Nit — 016x sets a minimum width but doesn't cap clock at 64 bits. If the clock reaches 2**64 ns (year 2554, or a badly set system/VM clock), the id grows to 33+ chars, which breaks the §8.2 "exactly 32 lowercase hex" shape. Because ids compare as strings, a 1000… id would also sort below an earlier ffff… one and reverse the in-process ordering. Mask or clamp the value, or fail loudly when it exceeds 64 bits. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, gpt-5.6-sol-xhigh edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this. Overflow needs a clock past the year 2554. max(time.time_ns(), last + 1) only moves forward, so a clock set that far ahead would already have broken much more than op_id width.



def _new_op(kind: str, actor: str, base_version: int, **fields: Any) -> dict[str, Any]:
return {
"op": kind,
"op_id": uuid.uuid4().hex,
"op_id": _new_op_id(),
"actor": actor,
"base_version": base_version,
"stamp": [base_version, actor],
Expand Down Expand Up @@ -1313,8 +1334,8 @@ def set_dynamic_group_rows(

The writes run on a copy, so ``workflow`` is unchanged when any is refused.
The ops share one stamp, so an earlier write to the group at the same
stamp is refused rather than left to the ``op_id`` tiebreak (revisit once
#1007 makes the later write in a batch win). ``apply_op`` must replay them
stamp is refused rather than left to the ``op_id`` tiebreak (kept by
op-vocabulary amendment v1.14). ``apply_op`` must replay them
in order, because this workflow model materializes rows from the count.
"""
from comfy_cli.cql import engine as _engine
Expand Down Expand Up @@ -3698,8 +3719,7 @@ def unknown_size(class_type: Any) -> list[float] | None:
except KeyError as e:
raise ValueError(f"spec #{i} ({kind}) is missing required field {e}") from e
# A later write at the batch's stamp would tie with the replacement
# and be settled by the random op_id. Revisit once #1007 makes the
# later write in a batch win.
# on the op_id tiebreak. Amendment v1.14 keeps this refusal.
target = _write_target(op) if op.get("op") == "set_widget" else None
if target in replaced:
raise ValueError(
Expand Down
42 changes: 42 additions & 0 deletions docs/op-vocabulary-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -1618,3 +1618,45 @@ a non-insert op carrying `definitions`, was correctly rejected and left
`{"_applied_ops": []}` behind on a document that had no ledger at all.

Both now run before the seed. No message or code changes.

## 23. Amendment v1.14 — 2026-11-03 (later write in a batch wins)

`apply_specs` stamps every op in a batch with the same `[base_version,
actor]`, so two writes to one register in a batch tied on the stamp and the
random uuid4 `op_id` picked the winner (the §11.1 batch caveat). The later
write could be a silent no-op, and a write to a structural register (a
dynamic-combo selector, a DynamicGroup count) could lose to an earlier one and
make the specs that followed it fail.

**The rule.** `_new_op` now mints `op_id` as 16 hex chars of a nanosecond
clock that never repeats or goes backwards within the process, then 16 hex
chars of random bits. Ops a writer mints later compare higher, so within one
batch the later spec wins. This supersedes "uuid4 hex" in §§1, 2, 6 and 8.2
(including the `"<uuid4 hex>"` envelope examples) and the §11.1 batch caveat.

**Scope of the ordering.** The guarantee holds for ops one process mints.
Across separate invocations the prefix follows the wall clock, so a clock step
can reverse two runs' ops at the same stamp. A uuid4 `op_id` minted before this
amendment starts with a random hex digit, so at an equal stamp it usually
outranks a new one (whose prefix starts with `1`), where two uuid4 ids were an
even split. Ordering across processes needs a different stamp, not this
tiebreak. The prefix also carries the minting host's clock, to the nanosecond,
wherever the `op_id` travels.

**Still refused.** Three batch guards stay, and each refuses the whole batch
instead of letting the later spec win: a `set_dynamic_group_rows` whose group
was already written at the batch's stamp; a `set_widget` to a count or row
field an earlier `set_dynamic_group_rows` in the batch replaced; and a second
`connect` / `disconnect` to the same input. A follow-up may retire the first
two.

**Wire shape unchanged.** `op_id` is still exactly 32 lowercase hex chars
(§8.2), the stamp is still `[base_version, actor]`, and `_stamp_key` /
`_lww_gate` are unchanged. Receivers never parse or regenerate an `op_id`, so
ops minted before this amendment replay as before. One batch shape still lands
differently by apply order: a sub-widget write followed by a rebuild of its
selector or count, because the rebuild resets the value but not the stamp
(comfy-cli #1101).

**No change to §§3-5, 7, or §8 beyond §8.2.** No op kind was added, removed, or re-scoped;
`FROZEN_OPS` / `DEFERRED_OPS` / `BATCHABLE_OPS` are untouched.
17 changes: 5 additions & 12 deletions tests/comfy_cli/command/test_ack_flag.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
from __future__ import annotations

import json
from typing import Any

import pytest
from test_workflow_edit import ( # type: ignore[import-not-found]
Expand Down Expand Up @@ -122,7 +121,7 @@ def test_apply_ack_default_byte_identical(self, patched_graph, tmp_path, capsys,
"""`--ack full` must be byte-identical to today's default payload.

Soundness of the assertion: node ids (`mint_id`, `random.getrandbits`)
and op ids (`uuid.uuid4`) are the ONLY nondeterminism in the apply
and op ids (`_new_op_id`) are the ONLY nondeterminism in the apply
path (layout is documented "no randomness, no clock"; the CRDT stamp
is [base_version, actor]). We pin both to counters, run the same batch
on the same file path twice — resetting the file content and the
Expand All @@ -138,18 +137,12 @@ def fake_mint() -> int:
state["node"] += 1
return (1 << 40) + state["node"]

class _FakeUUID:
def __init__(self, n: int):
self.hex = f"{n:032x}"

class _FakeUuidModule:
@staticmethod
def uuid4() -> Any:
state["op"] += 1
return _FakeUUID(state["op"])
def fake_op_id() -> str:
state["op"] += 1
return f"{state['op']:032x}"

monkeypatch.setattr(workflow_ops, "mint_id", fake_mint)
monkeypatch.setattr(workflow_ops, "uuid", _FakeUuidModule)
monkeypatch.setattr(workflow_ops, "_new_op_id", fake_op_id)

initial = json.dumps({"nodes": [], "links": [], "last_node_id": 0, "last_link_id": 0}, indent=2)
path = tmp_path / "wf.json"
Expand Down
129 changes: 129 additions & 0 deletions tests/comfy_cli/test_batch_write_order.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
"""Two writes to one register inside an ``apply_specs`` batch: the later spec wins.

Every op in a batch carries the same stamp ``[base_version, actor]``, so the
``op_id`` tiebreak in ``_stamp_key`` decides between them. With uuid4 op_ids the
earlier write won about half the time and the later one became a silent no-op
(#1006). Amendment v1.14 mints increasing op_ids, so spec order decides.
"""

from __future__ import annotations

import copy

from comfy_cli import workflow_ops
from comfy_cli.cql.engine import Graph

_SUB = {"prompt": ["STRING", {"default": ""}]}
_MAX_SUB = {**_SUB, "mode": ["COMBO", {"default": "balanced", "options": ["balanced", "quality"]}]}

GRAPH = Graph.from_object_info(
{
"Gen": {
"input": {
"required": {
"model": [
"COMFY_DYNAMICCOMBO_V3",
{
"options": [
{"key": "base", "inputs": {"required": _SUB}},
{"key": "max", "inputs": {"required": _MAX_SUB}},
]
},
],
"label": ["STRING", {"default": ""}],
}
},
"input_order": {"required": ["model", "label"]},
"output": ["VIDEO"],
"output_name": ["VIDEO"],
"category": "test",
"display_name": "Gen",
"python_module": "nodes",
}
}
)

RUNS = 20


def _fresh() -> tuple[dict, int]:
wf = {"nodes": [], "links": [], "last_node_id": 0, "last_link_id": 0, "version": 0.4}
wf, op = workflow_ops.add_node(wf, GRAPH, "Gen")
return wf, op["node_id"]


def _widget(wf: dict, nid: int, name: str):
node = next(n for n in wf["nodes"] if n["id"] == nid)
values = node["widgets_values"]
return values[workflow_ops._widget_index(GRAPH, "Gen", name, values)]


def test_later_write_to_the_same_widget_wins():
for _ in range(RUNS):
wf, nid = _fresh()
wf, _, _ = workflow_ops.apply_specs(
wf,
GRAPH,
[
{"op": "set_widget", "node": nid, "widget": "label", "value": "x"},
{"op": "set_widget", "node": nid, "widget": "label", "value": "y"},
],
)
assert _widget(wf, nid, "label") == "y"


def test_later_selector_write_shapes_the_specs_after_it():
"""A structural register: the second selector write must win, or the
sub-widget write after it fails with "not found"."""
for _ in range(RUNS):
wf, nid = _fresh()
wf, _, _ = workflow_ops.apply_specs(
wf,
GRAPH,
[
{"op": "set_widget", "node": nid, "widget": "model", "value": "base"},
{"op": "set_widget", "node": nid, "widget": "model", "value": "max"},
{"op": "set_widget", "node": nid, "widget": "model.mode", "value": "quality"},
],
)
assert _widget(wf, nid, "model") == "max"
assert _widget(wf, nid, "model.mode") == "quality"


def test_batch_stamps_increase_in_spec_order():
wf, nid = _fresh()
specs = [{"op": "set_widget", "node": nid, "widget": "label", "value": str(i)} for i in range(50)]
_, ops, _ = workflow_ops.apply_specs(wf, GRAPH, specs)
keys = [workflow_ops._stamp_key(op) for op in ops]
assert keys == sorted(keys) and len(set(map(tuple, keys))) == len(keys)


def test_batch_replays_to_the_same_graph_in_any_order_and_twice():
base, nid = _fresh()
specs = [
{"op": "set_widget", "node": nid, "widget": "label", "value": "x"},
{"op": "set_widget", "node": nid, "widget": "label", "value": "y"},
]
local, ops, _ = workflow_ops.apply_specs(copy.deepcopy(base), GRAPH, specs)
expected = workflow_ops.canonical(local)
for order in (ops, ops[::-1], ops + ops[::-1]):
wf = copy.deepcopy(base)
for op in order:
workflow_ops.apply_op(wf, op, GRAPH)
assert workflow_ops.canonical(wf) == expected


def test_selector_batch_replays_to_the_same_graph_in_any_order_and_twice():
base, nid = _fresh()
specs = [
{"op": "set_widget", "node": nid, "widget": "model", "value": "base"},
{"op": "set_widget", "node": nid, "widget": "model", "value": "max"},
{"op": "set_widget", "node": nid, "widget": "model.mode", "value": "quality"},
]
local, ops, _ = workflow_ops.apply_specs(copy.deepcopy(base), GRAPH, specs)
expected = workflow_ops.canonical(local)
for order in (ops, ops[::-1], ops + ops[::-1]):
wf = copy.deepcopy(base)
for op in order:
workflow_ops.apply_op(wf, op, GRAPH)
assert workflow_ops.canonical(wf) == expected
6 changes: 3 additions & 3 deletions tests/comfy_cli/test_connect_lww.py
Original file line number Diff line number Diff line change
Expand Up @@ -690,9 +690,9 @@ def test_generated_two_writer_streams_converge(seed: int):
def test_same_batch_connects_resolve_by_op_id_not_batch_position():
"""``apply_specs`` stamps every op in a batch with the SAME base_version,
so a same-target pair inside one batch is decided by the op_id tiebreak,
not by spec order. Convergence holds — "last spec wins" does not. This has
been true of ``set_widget`` since the freeze; amendment v1.2 extends the
same property to ``connect`` and says so."""
not by arrival order. Since amendment v1.14 ``_new_op`` mints increasing
op_ids, so a minted batch resolves in spec order; these hand-authored ids
pin the tiebreak itself."""
first = _connect("k1", AGENT, 5, 9901, ENCODER)
second = _connect("k0", AGENT, 5, 9902, OTHER_ENCODER)
wf = _run(_base(), [first, second])
Expand Down
Loading