Skip to content

Commit d099390

Browse files
authored
fix(load): submit a managed load as a job and poll, don't hold a request open (#72)
* fix(load): submit a managed load as a job and poll, don't hold a request open A load's duration scales with the data, so waiting for one inside a single HTTP request makes it depend on every layer between the client and the engine staying up for the whole thing -- CDN, gateway, and this client's own socket read deadline. Any one of them ending the request loses the load: the server marks it `abandoned` and DISCARDS work it had already done, while the table's write lock is still held against the retry that follows. The retry then collides with that lock (409 RESOURCE_LOCKED) and the two can alternate indefinitely without the load ever completing. Seen in production on a table whose load runs past five minutes: the same load acquired, was abandoned at ~4m45s, and re-acquired, three times in a row, while the caller retried 409s for twenty-five minutes. Small tables on the same connection succeeded in under two seconds throughout, which is what ruled out the lock itself being at fault. The server already offers the fix and recommends it for this exact case -- "run the load as a background job and return a job ID to poll instead of blocking until it finishes. Recommended for large uploads, which can take longer than an HTTP request should stay open." So loads now go out with `async` set. `async_after_ms` keeps the common case unchanged. The server answers 200 with the result when the load finishes inside the window and only defers to a job when it does not, so nothing pays for polling that did not need it -- dlt's bookkeeping tables settle in under a second and would otherwise gain a submit-then-poll round trip each. The polling budget is the load's own, deliberately not the 300s that queries and results share: reusing that is what put a five-minute ceiling on loads to begin with. Bounded rather than unbounded, so a wedged job still surfaces as a timeout. `_poll_job` already existed for the index path, which has had this shape all along; this reuses it rather than adding a second poller. Worth noting for `append`: it is the one non-idempotent mode, and the reason a lost response is dangerous there is that "did it land?" becomes unanswerable. A job id outlives the request that created it, so that ambiguity is gone -- the retry rule is unchanged here, but it no longer rests on the response surviving. Verified against a live deployment: the server accepts the new fields, a fast load still returns inline, and no polling happens when it does. The deferred path is covered by unit tests -- reproducing it live needs a load slow enough to exceed the window, which the reporting workload produces and a synthetic 300k-row load (0.8s) does not. * fix(load): a failed status check must not discard a running job Review found a regression this PR introduced, reachable from the opposite direction to the one it fixes. `_poll_job` aborted on the first bad `GET /v1/jobs/{id}`. That exception reaches `ManagedDatabaseClient._request_with_retry`, which classifies a 502/503 as transient and re-invokes THE WHOLE OPERATION for every non-append mode -- so a single blipped status check discards a healthy running job and submits a second load against a table whose write lock the first one still holds. 409 RESOURCE_LOCKED, which is also transient, and round it goes. Worse than the behaviour it replaces, not better: before, a lost request took with it a load that was genuinely being torn down. Now the first job is neither going away nor releasing anything, and a load polls for up to `_LOAD_JOB_TIMEOUT_S`, so there are hundreds of status checks where there used to be one request -- every one a fresh chance to trigger the collision. Exposure up, collision from incidental to certain. So the poll tolerates a failed status check and keeps waiting, bounded by consecutive failures rather than by the first one. An isolated 502 is noise; a run of them means the API is gone and there is nothing to wait for. The poll's own deadline bounds the total wait either way. THE JOB ID IS NOW RETURNED. The docstring claimed the id removed `append`'s "did it land?" ambiguity, which was not true as implemented: it was a local, discarded on every failure path, and never reached the caller. `LoadManagedTableResult` carries it now, for the same reason `CreateIndexResult` does. `append` stays non-retryable -- knowing the id makes the question answerable, it does not make a blind re-submission safe, and that call belongs to the caller. The docstring says that instead of the stronger thing. `result` is read with `getattr(..., "actual_instance", ...)`, matching the index path rather than silently disagreeing with it: the strict read raised a bare AttributeError if the model ever arrived unwrapped. Did NOT add a strict isinstance check on the inline response. The index path can assert its inline type because it names it positively first; here the inline shape is read duck-typed and callers and tests rely on that, so asserting it narrows an interface this change has no business narrowing. Only the job branch is type-checked. Import order fixed for ruff's I rule; `ruff check` clean. 149 passed. * fix(load): measure poll-error tolerance in time, and name the job when giving up Two review points, both on paths where the caller is left not knowing whether the load landed. TOLERANCE WAS A COUNT, WHICH MEANS NOTHING ON ITS OWN. Five consecutive failures at the load's 2s interval is about eight seconds, and the thing being survived is a gateway blip: a rolling restart or a load balancer reconverging serves 502s for longer than that. Past it the poll aborted, the caller's retry re-submitted the load, and the first job was still holding the table -- the door this tolerance exists to close, just narrower than before. A count is also the wrong unit, because what it means in wall-clock depends on whichever `interval_s` the caller passed, and callers differ. Now a time bound: give up after 120s of CONTINUOUS failure, reset on any success, still bounded by the poll's own deadline. Failed checks also back off to 15s rather than retrying every 2s -- whatever is serving 502s does not need the extra traffic. THE GIVE-UP PATHS DROPPED THE JOB ID. `api_error_message(e)` is the gateway's message ("502: Bad Gateway"), and the failed-job raise preferred the server's message, which need not mention the id. Those are exactly the two paths where the answer is unknown and the id is the only handle for asking. Both name the job now, matching what `_poll_job`'s own TimeoutError already did. The test for the tolerance drives a FAKE CLOCK advanced by the sleeps. With `sleep` patched to a no-op and a real clock, it waited out the whole grace period in wall time -- 120 seconds of a spinning loop to assert one message, and it slowed the suite from 0.5s to 121s before I noticed. A test measuring a time bound has to own the clock. 152 passed in 0.53s. `ruff check` clean on the touched files.
1 parent f98c313 commit d099390

4 files changed

Lines changed: 459 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Fixed
11+
12+
- fix(load): submit managed loads as a job and poll, instead of holding one request open
13+
1014

1115
## [0.12.0] - 2026-08-11
1216

‎hotdata_framework/client.py‎

Lines changed: 119 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
from hotdata.models.index_info_response import IndexInfoResponse
2626
from hotdata.models.job_status_response import JobStatusResponse
2727
from hotdata.models.load_managed_table_request import LoadManagedTableRequest
28+
from hotdata.models.load_managed_table_response import LoadManagedTableResponse
2829
from hotdata.models.query_request import QueryRequest
2930
from hotdata.models.query_response import QueryResponse
3031
from hotdata.models.submit_job_response import SubmitJobResponse
@@ -81,6 +82,37 @@
8182
# Jobs have no "cancelled" state; "partially_succeeded" carries an error_message.
8283
_JOB_TERMINAL = frozenset({"succeeded", "partially_succeeded", "failed"})
8384

85+
# How long a load may finish INLINE before the server hands back a job instead.
86+
# Small enough that a slow load stops holding a request open, large enough that
87+
# the overwhelming majority never become jobs at all: dlt's bookkeeping tables
88+
# (`_dlt_version`, `_dlt_loads`, `_dlt_pipeline_state`) settle in under a second,
89+
# and paying a submit-then-poll round trip for those would be a regression.
90+
_LOAD_INLINE_WAIT_MS = 10_000
91+
92+
# A load's own polling budget. Deliberately NOT the 300s used for queries and
93+
# results: a load is the one operation here whose duration scales with the data,
94+
# and reusing the query budget is what put a five-minute ceiling on it in the
95+
# first place. Bounded rather than unbounded so a wedged job still surfaces.
96+
_LOAD_JOB_TIMEOUT_S = 3600.0
97+
98+
# How long a poll tolerates CONTINUOUS status-check failure before giving up.
99+
#
100+
# Time, not a count: a count means whatever the caller's `interval_s` makes it,
101+
# and callers differ -- five checks is eight seconds at the load interval and
102+
# something else at the index one. The thing being survived is a gateway blip,
103+
# and a rolling restart or a load balancer reconverging routinely serves 502s for
104+
# longer than a few seconds. Too short and the poll aborts, the caller's retry
105+
# re-submits the load, and the original job is still holding the table -- the
106+
# door this tolerance exists to close.
107+
#
108+
# Still bounded by the poll's own deadline, so this only decides how a stretch of
109+
# failures ends, never how long the wait can be.
110+
_JOB_POLL_ERROR_GRACE_S = 120.0
111+
112+
# Failed checks back off rather than hammering at `interval_s`: whatever is
113+
# serving 502s does not need the extra traffic.
114+
_JOB_POLL_ERROR_MAX_BACKOFF_S = 15.0
115+
84116

85117
@dataclass(frozen=True)
86118
class ResultSummary:
@@ -418,10 +450,25 @@ def load_managed_table(
418450
else:
419451
assert file is not None
420452
resolved_upload_id = self.upload_parquet(file)
453+
# ASKED FOR AS A JOB, not as a held-open request. A load's duration scales
454+
# with the data, and a single request that must survive minutes has to
455+
# survive every layer between here and the engine -- CDN, gateway, socket
456+
# read timeout -- any one of which ends it. When it ends, the server logs
457+
# the load `abandoned` and DISCARDS work it had already done, while the
458+
# table's write lock is still held against the retry that follows; the
459+
# retry then collides with it (409 RESOURCE_LOCKED) and the pair can spin
460+
# indefinitely without the load ever completing. Observed in production on
461+
# a table whose load runs past five minutes.
462+
#
463+
# `async_after_ms` keeps the common case unchanged: the server answers 200
464+
# with the result if it finishes inside the window, and only falls back to
465+
# a job when it does not. So nothing pays for polling that did not need it.
421466
request = LoadManagedTableRequest(
422467
mode=mode,
423468
upload_id=resolved_upload_id,
424469
key=key,
470+
var_async=True,
471+
async_after_ms=_LOAD_INLINE_WAIT_MS,
425472
)
426473
try:
427474
loaded = self.connections().load_managed_table(
@@ -432,12 +479,21 @@ def load_managed_table(
432479
)
433480
except ApiException as e:
434481
raise RuntimeError(api_error_message(e)) from e
482+
# Only the job branch is type-checked. The index path can also assert its
483+
# inline type because it names it positively first; here the inline shape is
484+
# read duck-typed, which callers and tests already rely on, so asserting it
485+
# would narrow an interface this change has no business narrowing.
486+
job_id: str | None = None
487+
if isinstance(loaded, SubmitJobResponse):
488+
job_id = loaded.id
489+
loaded = self._load_response_from_job(job_id)
435490
return LoadManagedTableResult(
436491
connection_id=loaded.connection_id,
437492
schema_name=loaded.schema_name,
438493
table_name=loaded.table_name,
439494
row_count=loaded.row_count,
440495
full_name=f"{db.id}.{loaded.schema_name}.{loaded.table_name}",
496+
job_id=job_id,
441497
)
442498

443499
def add_managed_table(
@@ -880,11 +936,40 @@ def _poll_job(
880936
jobs = self._jobs_api()
881937
deadline = time.monotonic() + timeout_s
882938
last: JobStatusResponse | None = None
939+
# When the current run of failures began, or None while checks succeed.
940+
failing_since: float | None = None
941+
error_backoff = interval_s
883942
while time.monotonic() < deadline:
884943
try:
885944
last = jobs.get_job(job_id)
945+
failing_since = None
946+
error_backoff = interval_s
886947
except ApiException as e:
887-
raise RuntimeError(api_error_message(e)) from e
948+
# A failed STATUS CHECK is not a failed job. Aborting here throws
949+
# away work that is still running, and the caller's retry then
950+
# re-submits it while the original still holds its resources -- so
951+
# one blip becomes a collision with the job it just abandoned. A
952+
# load polls for up to `_LOAD_JOB_TIMEOUT_S`, so the longer the
953+
# wait the more chances to hit it, which is exactly backwards.
954+
#
955+
# CONSECUTIVE failures are the signal: an isolated 502 is noise, a
956+
# run of them means the API is gone and there is nothing to wait
957+
# for. The poll's own deadline bounds the total wait regardless.
958+
now = time.monotonic()
959+
if failing_since is None:
960+
failing_since = now
961+
elif now - failing_since >= _JOB_POLL_ERROR_GRACE_S:
962+
# Name the job. This is one of the two paths where the caller
963+
# cannot tell whether the load landed, so the id is the only
964+
# thing that makes the question answerable -- and it is exactly
965+
# what a message like "502: Bad Gateway" leaves out.
966+
raise RuntimeError(
967+
f"Job {job_id} status checks failed for "
968+
f"{_JOB_POLL_ERROR_GRACE_S:.0f}s: {api_error_message(e)}"
969+
) from e
970+
time.sleep(error_backoff)
971+
error_backoff = min(error_backoff * 2, _JOB_POLL_ERROR_MAX_BACKOFF_S)
972+
continue
888973
if last.status in _JOB_TERMINAL:
889974
return last
890975
time.sleep(interval_s)
@@ -893,6 +978,39 @@ def _poll_job(
893978
f"Job {job_id} did not finish within {timeout_s}s (last status: {last_status})"
894979
)
895980

981+
def _load_response_from_job(self, job_id: str) -> LoadManagedTableResponse:
982+
"""The result of a load the server chose to run as a job.
983+
984+
Polling replaces waiting on the request, so the outcome is read from
985+
durable state rather than from a connection that has to stay alive. That
986+
also gives a caller a handle: the job id is returned on
987+
`LoadManagedTableResult`, so "did it land?" is answerable after a lost
988+
response. `append` stays non-retryable -- knowing the id makes the question
989+
answerable, it does not make a blind re-submission safe, and that call is
990+
the caller's to make.
991+
992+
`partially_succeeded` is terminal and carries a message, so it is raised
993+
rather than returned -- a caller asked for a table's contents to be
994+
replaced or appended to, and "some of it" is not an answer it can use.
995+
"""
996+
final = self._poll_job(job_id, timeout_s=_LOAD_JOB_TIMEOUT_S)
997+
status = enum_value(final.status)
998+
if status != "succeeded":
999+
# The id goes in whether or not the server's message mentions it: the
1000+
# caller is being told the load did not succeed, and "which load" is
1001+
# the next thing it needs.
1002+
detail = final.error_message or f"finished {status}"
1003+
raise RuntimeError(f"load job {job_id} {status}: {detail}")
1004+
# `result` is a oneOf wrapper today; tolerate the model arriving directly,
1005+
# the same way the index path does rather than disagreeing with it.
1006+
payload = getattr(final.result, "actual_instance", final.result)
1007+
if not isinstance(payload, LoadManagedTableResponse):
1008+
raise RuntimeError(
1009+
f"load job {job_id} succeeded without a load result "
1010+
f"(got {type(payload).__name__})"
1011+
)
1012+
return payload
1013+
8961014
def _wait_result_ready(
8971015
self,
8981016
result_id: str,

‎hotdata_framework/databases.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,11 @@ class LoadManagedTableResult:
8383
table_name: str
8484
row_count: int
8585
full_name: str
86+
# Set when the server ran the load as a background job. Carried for the same
87+
# reason CreateIndexResult carries it: it is the only handle a caller has to
88+
# ask "did that land?" after a lost response, and without it the question is
89+
# unanswerable. `None` when the load finished inline.
90+
job_id: str | None = None
8691

8792
def to_dict(self) -> dict[str, Any]:
8893
return asdict(self)

0 commit comments

Comments
 (0)