Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ history.

### Fixed

- Parallel build and deploy requests rejected on an old access token now adopt a peer's refreshed token under the shared refresh lock instead of rotating it again. Each rejected request retries at most once. If session recovery still fails while a refreshable sign-in remains stored, the watch reports that the remote work may continue instead of telling you to sign in again. `comfy build release create --watch`, `comfy deploy up`, `promote`, `rollback` and `comfy deploy status --watch` now end with the new code `watch_auth_lost` and exit code 75, as `deploy_watch_lost` does. Its hint names the command that reads the release or deployment again. A 401 before anything was accepted, on a workspace API key, or when no refreshable sign-in remains still reports `build_not_signed_in` or `deploy_not_signed_in`.

- `comfy update` no longer dies with a `FileNotFoundError` traceback when the
configured workspace directory does not exist (never installed, or deleted).
It now fails with a `not_in_workspace` error that points at `comfy install`,
Expand Down
7 changes: 7 additions & 0 deletions comfy_cli/builder_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ def __init__(
# handed a token (COMFY_BUILDER_TOKEN, COMFY_CLOUD_AUTH_TOKEN) or a key keeps
# it and lets a 401 surface.
self._refreshes_on_401 = False
self._sign_in_base_url: str | None = None
# What to replace once the builder refuses this client's credential.
self._refused_hint = (
credentials.refused_key_hint(api_key)
Expand Down Expand Up @@ -175,9 +176,15 @@ def from_session(cls, base_url: str) -> BuilderClient:
raise BuilderAuthError("not signed in: run `comfy cloud login` or set COMFY_CLOUD_API_KEY")
client = cls(base_url, session.access_token)
client._refreshes_on_401 = True
client._sign_in_base_url = session.base_url
client._refused_hint = "run `comfy cloud login` first"
return client

@property
def sign_in_base_url(self) -> str | None:
"""The environment of the stored sign-in this client's token came from; None for a key or a handed token."""
return self._sign_in_base_url if self._refreshes_on_401 else None

def _send(self, url: str, **kwargs) -> tuple[int, dict | list | None]:
"""Every builder request goes through here, so a long wait survives its token expiring.

Expand Down
23 changes: 22 additions & 1 deletion comfy_cli/cloud/oauth.py
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,7 @@ def ensure_fresh_session(
force: bool = False,
allow_clear: bool = True,
expect: Callable[[CloudSession], bool] | None = None,
rejected_access_token: str | None = None,
):
"""Return the stored cloud session, refreshing it first when the access
token is expired (or within ``leeway_s`` of expiring) and a refresh token
Expand All @@ -803,6 +804,10 @@ def ensure_fresh_session(
``expires_at`` recorded — still thinks it is valid. The refresh token is
still required; without one this is a no-op.

``rejected_access_token`` identifies the token that received the 401. If a
peer already replaced it, adopt that session instead of refreshing again.
This also covers a late 401 that arrives after the peer released the lock.

``allow_clear=False`` keeps the stored session on disk even when a refresh
hits a fatal token error (reuse-detection / ``invalid_grant``). Background,
read-mostly callers (the detached ``comfy run`` watcher) pass this so a
Expand Down Expand Up @@ -855,6 +860,7 @@ def ensure_fresh_session(
force=force,
allow_clear=allow_clear,
expect=expect,
rejected_access_token=rejected_access_token,
)
except TimeoutError:
# A peer has held the lock longer than a refresh should take. Don't
Expand All @@ -875,6 +881,7 @@ def _locked_refresh(
force: bool,
allow_clear: bool = True,
expect: Callable[[CloudSession], bool] | None = None,
rejected_access_token: str | None = None,
):
"""Decide → POST → persist, executed while holding the secrets lock.

Expand All @@ -890,11 +897,25 @@ def _locked_refresh(
if session is None or not session.refresh_token:
return session

# The rejected request may have been in flight while a peer finished its
# refresh, before even our first store read. Don't rotate that peer's token
# again just because this client's older request was refused.
if (
rejected_access_token is not None
and session.access_token != rejected_access_token
and session.expires_at is not None
and not session.is_expired(leeway_s=leeway_s)
):
return session

# Double-check after acquiring the lock. A peer may have refreshed while we
# blocked: if the stored refresh token rotated away from what we observed
# before locking, the family already advanced — use it, never re-send the
# consumed token (that is exactly what trips reuse-detection).
if session.refresh_token != observed_refresh_token:
# A usable reactive peer already returned above. Otherwise refresh its
# current family, never replaying the older token observed before locking.
# Ordinary proactive callers still coalesce on any peer rotation.
if session.refresh_token != observed_refresh_token and rejected_access_token is None:
return session
if not force and not session.is_expired(leeway_s=leeway_s):
# Peer advanced the access token's expiry without us noticing a token
Expand Down
36 changes: 35 additions & 1 deletion comfy_cli/command/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -3139,6 +3139,40 @@ def _selected_release_id(renderer, client, scope: _BuildScope, release: str | No
return _newest_release_id(renderer, client, _resolve_build_id(renderer, client, scope))


#: The exit a watch that lost its sign-in ends with, as `comfy deploy`'s watches do
#: (``deploy_runtime.EXIT_WATCH_LOST``): only the watch ended, so it must not exit 1,
#: which says the release failed. 75 is EX_TEMPFAIL in sysexits.h.
_EXIT_WATCH_LOST: Final = 75


def _watched_release(renderer, client, release_id: str) -> dict:
"""One read of a release this command has already cut.

The builder accepted the cut with this sign-in, so a 401 session recovery did
not clear ends the watch while a refreshable sign-in remains stored. The
release may continue remotely. A key, a handed token, or a sign-in without
a refresh token keeps ``build_not_signed_in``.
"""
from comfy_cli import credentials
from comfy_cli.builder_api import BuilderCredentialRefused

try:
return client.get_release(release_id)
except BuilderCredentialRefused as error:
if not credentials.sign_in_still_stored(getattr(client, "sign_in_base_url", None)):
raise
renderer.error(
code="watch_auth_lost",
message=f"the builder refused the sign-in (401) after attempting session recovery; "
f"release {release_id} may still be building on the server",
hint=f"read it with `comfy build release show {release_id}` until its status is complete; "
"the stored sign-in was preserved",
details={"releaseId": release_id},
exit_code=_EXIT_WATCH_LOST,
)
raise typer.Exit(code=_EXIT_WATCH_LOST) from error


def _poll_call(renderer, fn, usable=None):
"""``_builder_call`` for one poll of a watch, which a single timed-out or
dropped request must not end: the release keeps building either way. A
Expand Down Expand Up @@ -3196,7 +3230,7 @@ def _poll_status(renderer, client, release_id: str) -> dict:
word and ends the watch at the caller."""
return _poll_call(
renderer,
lambda: client.get_release(release_id),
lambda: _watched_release(renderer, client, release_id),
usable=lambda release: isinstance(release, dict) and isinstance(release.get("status"), str),
)

Expand Down
44 changes: 41 additions & 3 deletions comfy_cli/command/deploy_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import typer
from typing_extensions import assert_never

from comfy_cli import credentials
from comfy_cli.builder_api import BuilderClient, BuilderCredentialRefused
from comfy_cli.command.build import DEFAULT_BUILDER_URL
from comfy_cli.command.build_paths import BuildSpecNotFoundError, resolve_build_paths
Expand Down Expand Up @@ -121,13 +122,32 @@ def terminal_status_error(deployment_id: str, status: str) -> JsonObject:
class DeployWatchLostError(Exception):
"""The deploy API left a watch's reads unanswered for the whole retry window."""

code = "deploy_watch_lost"

def __init__(self, deployment_id: str, cause: Exception | None = None, *, message: str | None = None) -> None:
super().__init__(
message or f"the deploy API stopped answering while following deployment {deployment_id} ({cause})"
)
self.deployment_id = deployment_id


class DeployWatchAuthLostError(DeployWatchLostError):
"""The deploy API refused a watch's read while a refreshable sign-in remains stored.

The client has already attempted shared session recovery and at most one
retry. The deployment may continue remotely, so it is the watch that ended.
"""

code = "watch_auth_lost"

def __init__(self, deployment_id: str) -> None:
super().__init__(
deployment_id,
message=f"the deploy API refused the sign-in (401) while following deployment {deployment_id}, "
"after attempting session recovery",
)


def _unanswered(error: Exception) -> bool:
"""A read the service never answered: a 5xx, or no response at all.

Expand All @@ -139,12 +159,26 @@ def _unanswered(error: Exception) -> bool:
return error.code == "deploy_server_error" and (error.status is None or error.status >= 500)


def _sign_in_refused(client: DeployUpClient, error: Exception) -> bool:
"""A 401 on a watch's read, from a client whose refreshable sign-in remains.

A key, a handed token, or a sign-in without a refresh token keeps the
sign-in code, because changing that credential is the fix.
"""
if not isinstance(error, DeployAPIError) or error.status != 401:
return False
return credentials.sign_in_still_stored(getattr(client, "sign_in_base_url", None))


def exit_watch_lost(renderer: Renderer, error: DeployWatchLostError, hint: str | None = None) -> NoReturn:
"""Say it is the watch that gave out, not the deployment, and how to pick it up again."""
hint = hint or f"run `{reattach_hint(error.deployment_id)}` to watch it again"
if isinstance(error, DeployWatchAuthLostError):
hint = f"{hint}; the stored sign-in was preserved"
renderer.error(
code="deploy_watch_lost",
code=error.code,
message=f"{error}; the deployment may still be coming up",
hint=hint or f"run `{reattach_hint(error.deployment_id)}` to watch it again",
hint=hint,
details={"deployment_id": error.deployment_id},
exit_code=EXIT_WATCH_LOST,
)
Expand Down Expand Up @@ -172,7 +206,9 @@ def poll_deployment(
The progress a watcher shows rides the same read the loop already makes, so
watching costs the service nothing it was not already answering. A read the
service does not answer is retried, and the first of each run of them goes to
``on_unanswered``; past the window this raises ``DeployWatchLostError``.
``on_unanswered``; past the window this raises ``DeployWatchLostError``. A
401 session recovery did not clear, with a refreshable sign-in still stored,
raises ``DeployWatchAuthLostError`` at once.
"""
failures, since, started = 0, 0.0, clock()
while True:
Expand All @@ -186,6 +222,8 @@ def poll_deployment(
# The client maps a timeout, but a connection dropped before the status
# line or partway through a 200's body escapes it unmapped.
except (DeployAPIError, ConnectionError, http.client.HTTPException) as error:
if _sign_in_refused(client, error):
raise DeployWatchAuthLostError(deployment_id) from error
if not _unanswered(error):
raise
if failures == 0:
Expand Down
52 changes: 48 additions & 4 deletions comfy_cli/credentials.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ def get_session(
force: bool = False,
allow_clear: bool = True,
expect: Callable[[CloudSession], bool] | None = None,
rejected_access_token: str | None = None,
) -> CloudSession | None:
"""Read the stored Comfy Cloud OAuth session.

Expand All @@ -104,6 +105,8 @@ def get_session(
``force=True`` (implies the refresh path) refreshes unconditionally,
ignoring the local expiry check. Reserved for the *reactive* path: after a
server 401, the token is known-rejected even if our clock disagrees.
``rejected_access_token`` lets that path adopt a peer's newer access token
instead of rotating it again; the comparison happens under the refresh lock.

``allow_clear=False`` forwards to ``ensure_fresh_session`` so a fatal
refresh failure does NOT clear the stored session. Background watchers pass
Expand All @@ -117,9 +120,12 @@ def get_session(
if refresh or force:
from comfy_cli.cloud import oauth

options = {}
if expect is not None:
return oauth.ensure_fresh_session(force=force, allow_clear=allow_clear, expect=expect)
return oauth.ensure_fresh_session(force=force, allow_clear=allow_clear)
options["expect"] = expect
if rejected_access_token is not None:
options["rejected_access_token"] = rejected_access_token
return oauth.ensure_fresh_session(force=force, allow_clear=allow_clear, **options)
from comfy_cli.auth import store as auth_store

return auth_store.get_cloud_session()
Expand All @@ -140,14 +146,52 @@ def refreshed_access_token(
:func:`get_session`.
"""
if expect is not None:
session = get_session(refresh=True, force=True, allow_clear=allow_clear, expect=expect)
session = get_session(
refresh=True, force=True, allow_clear=allow_clear, expect=expect, rejected_access_token=rejected
)
else:
session = get_session(refresh=True, force=True, allow_clear=allow_clear)
session = get_session(refresh=True, force=True, allow_clear=allow_clear, rejected_access_token=rejected)
if session is None or not session.access_token or session.access_token == rejected:
return None
return session.access_token


def sign_in_still_stored(base_url: str | None) -> bool:
"""Whether a refreshable sign-in for the refused client's environment remains.

A 401 that outlives :func:`refreshed_access_token` and one retry does not show
that nobody is signed in: a peer process can hold the refresh lock past this
one's timeout, a refresh that fails on the network keeps the old token, and
the server can refuse a token it minted a moment ago. A sign-in without a
refresh token cannot recover this way and retains the sign-in guidance.
``base_url`` is the environment the client's sign-in came from; ``None``,
for a key or a handed token, never matches.
"""
if base_url is None:
return False
from comfy_cli import locking
from comfy_cli.auth import store as auth_store

try:
# Do not spend another lock budget after session recovery timed out.
# The store's nested read reuses this thread's lock reentrantly.
with locking.file_lock(auth_store.lock_path(), timeout=0):
session = get_session(refresh=False)
except TimeoutError:
# Contention cannot prove the held sign-in disappeared. End only the
# watch, rather than hanging or prescribing another login.
return True
except OSError as error:
_log.warning("could not re-read the stored sign-in after a 401: %s", error)
return False
return (
session is not None
and bool(session.access_token)
and bool(session.refresh_token)
and session.base_url == base_url
)


class DataPlaneToken:
"""The bearer one long command sends to a deployment endpoint, kept fresh.

Expand Down
13 changes: 13 additions & 0 deletions comfy_cli/error_codes.py
Original file line number Diff line number Diff line change
Expand Up @@ -1760,6 +1760,19 @@ class ErrorCode:
"after a move, read `comfy deploy show --deployment <id>` instead: the move has landed once it shows no "
"pendingUpdate and its releaseId is the release asked for",
),
ErrorCode(
"watch_auth_lost",
"A watch over work the server had already accepted (`comfy build release create --watch`, `comfy deploy "
"up`, `promote` and `rollback`, `comfy deploy status --watch`) was refused with HTTP 401 despite "
"serialized session recovery and at most one retry, while a refreshable sign-in remains stored. "
"Only the watch ended: the release may still be building and the deployment may still be coming up. "
"The exit code is 75, not 1. `details.releaseId` (build) or "
"`details.deployment_id` (deploy) names what is still running. A 401 before anything was accepted, on a "
"workspace API key, or with no refreshable sign-in remaining stays `build_not_signed_in` or "
"`deploy_not_signed_in`.",
"the stored sign-in was preserved: re-read the work with `comfy build "
"release show <releaseId>` or `comfy deploy status --deployment <deployment_id> --watch`",
),
ErrorCode(
"deploy_update_failed",
"`comfy deploy up`, `promote` or `rollback` moved a deployment onto another release, and the watch saw "
Expand Down
5 changes: 4 additions & 1 deletion comfy_cli/skills/comfy-build/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,10 @@ blob ls (hidden; workspace private blobs
that reaches the builder need it; a plain scan and a plain `validate` do not.
A CI job sets a workspace API key in `COMFY_CLOUD_API_KEY` instead, and that
key wins over a stored sign-in, so when the hint says that key was refused,
signing in cannot help: replace the key.
signing in cannot help: replace the key. `watch_auth_lost` (exit 75) means
`release create --watch` was refused after the cut despite session recovery,
while a refreshable sign-in remains stored. The release may still be building.
Read it with `comfy build release show <id>` before considering a new sign-in.
On `build_not_enabled` the platform is in limited beta and this account is not
enabled — stop and say so.

Expand Down
1 change: 1 addition & 0 deletions comfy_cli/skills/comfy-deploy-failures/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ argument you pass.
| `deploy_update_replaced` | A watched `up`, `promote` or `rollback` was replaced by a newer update; `details.replacing_release_id` names its release | Follow it with `comfy deploy status --deployment <id>`, or end it with `comfy deploy cancel --deployment <id>` |
| `deploy_updates_unavailable` | `promote`, `rollback`, `history`, `cancel` or `events --release` on a deployment with no `revision`, or `cancel` against a deploy service too old for it | `promote` or `rollback`: `up --create` on the release instead. `history`: read `events`. `cancel`: wait for the update to land or fail. `events`: drop `--release`; those events do not say which release made them |
| `deploy_watch_lost` | The watch's reads went unanswered for a minute, or a move by `up`, `promote` or `rollback` did not land within an hour (exit 75); the deployment may still be coming up | `comfy deploy status --deployment <id> --watch`, or after a move `comfy deploy show --deployment <id>` until it shows no `pendingUpdate`; do not redeploy |
| `watch_auth_lost` | A watch's read was refused with a 401 despite session recovery, and a refreshable sign-in remains stored (exit 75); the deployment may still be coming up | Reattach as for `deploy_watch_lost` before considering a new sign-in; do not redeploy |
| `deploy_delete_needs_confirm` | `delete` without `--yes` non-interactively | Confirm with the user, then pass `--yes` |

### Submitting a workflow
Expand Down
Loading
Loading