Skip to content

Report Studio's terminal websocket errors instead of reconnecting silently - #1959

Merged
dreadatour merged 4 commits into
mainfrom
fix-job-logs-terminal-errors
Sep 1, 2026
Merged

Report Studio's terminal websocket errors instead of reconnecting silently#1959
dreadatour merged 4 commits into
mainfrom
fix-job-logs-terminal-errors

Conversation

@dreadatour

Copy link
Copy Markdown
Contributor

datachain job logs <id> against a team name the server does not accept printed nothing but a reconnect countdown and never said why — 15 attempts with backoff capped at 60s, so roughly ten minutes of silence before "Failed to reconnect". In practice you interrupt it first:

DATACHAIN_STUDIO_TEAM=wrong-team datachain job logs a23466fb-f11a-4fbd-82ef-35feca494faf
>>>> WebSocket closed, reconnecting in 8s...^CError: Operation cancelled by the user

Studio does report the problem over the log-streaming websocket. The client dropped it, in two independent places.

Refused handshake. For an unknown team, a team you cannot access, or an invalid token, Studio refuses the websocket handshake with HTTP 403. websockets raises a handshake exception, which tail_job_logs caught in a blanket except (WebSocketException, OSError) and turned into a logger.debug plus a bare return — indistinguishable from a normal end-of-stream.

Error frame. For a malformed job id, or a job you cannot access, Studio accepts the connection, sends {"message": "Job ID is incorrect or not found"}, and closes with code 4404. That frame was yielded into a branch chain in show_logs_from_client that only handles log_blobs / logs / job, so it fell through every branch and was discarded.

What changed

  • tail_job_logs raises DataChainError when a refused handshake carries 401 or 403, naming the team. Every other failure — a dropped connection, a 5xx from a gateway mid-deploy — still returns and lets the caller reconnect, so job run keeps its REST-polling fallback.
  • show_logs_from_client raises the server's own message when it sees the terminal {"message": ...} frame.
  • The reconnect banner is now cleared before each connection attempt instead of when the first message arrives, and also when the backoff sleep is interrupted. A terminal error or a Ctrl-C no longer prints on top of a stale reconnecting in Ns... countdown.
  • Backoff moved into _wait_before_reconnect, next to the existing _print_reconnect_msg and _clear_line helpers. Behavior is identical — the sleep still uses the pre-increment retry count and the debug line still reports the same attempt number.

The error message names the team and the token as possible causes, because Studio deliberately answers the same way whether a team does not exist or you simply cannot reach it. The client cannot tell those apart and should not pretend to.

🤖 Co-authored by Claude Code and ChatGPT

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves Studio job-log error reporting and terminal cleanup.

Changes:

  • Surfaces terminal WebSocket handshake and server-frame errors.
  • Extracts reconnect backoff and clears stale banners.
  • Adds regression tests for error and interruption paths.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/datachain/studio.py Handles terminal frames and reconnect cleanup.
src/datachain/remote/studio.py Converts 401/403 handshakes into actionable errors.
tests/test_cli_studio.py Tests handshake, frame, retry, and interruption behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/datachain/remote/studio.py Outdated
Comment thread tests/test_cli_studio.py Outdated
Comment thread src/datachain/remote/studio.py
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment thread src/datachain/studio.py
return received, last_log_id


async def _wait_before_reconnect(retry_count: int) -> str:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: move out from show_logs_from_client to avoid # noqa: PLR0915.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 28, 2026

Copy link
Copy Markdown

Deploying datachain with  Cloudflare Pages  Cloudflare Pages

Latest commit: 014b3a2
Status: ✅  Deploy successful!
Preview URL: https://c3a97072.datachain-2g6.pages.dev
Branch Preview URL: https://fix-job-logs-terminal-errors.datachain-2g6.pages.dev

View logs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@dreadatour
dreadatour merged commit e89edc2 into main Sep 1, 2026
35 checks passed
@dreadatour
dreadatour deleted the fix-job-logs-terminal-errors branch September 1, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants