Skip to content

Commit d687a37

Browse files
committed
fix(ingest): continuous datasources are not validate_only
new-datasource unconditionally set validate_only=true ("discover schema, don't load"), so --continuous produced a spec with BOTH continuous and validate_only. That is the exact combination the worker now rejects (422): the scheduler re-runs it every tick but the drain falls to full-replace, reloading the whole bucket forever. It was the root cause of the otel_telemetry incident. A continuous datasource is a persistent, self-loading source the scheduler keeps synced — the opposite of a one-off schema preview. Set validate_only only when NOT continuous. Without this, --continuous is unusable against a worker carrying the guard (dlthubworker #193): every attempt 422s.
1 parent bef5aec commit d687a37

1 file changed

Lines changed: 12 additions & 2 deletions

File tree

‎src/commands/ingest.rs‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -828,8 +828,14 @@ fn build_create_request(
828828
}
829829
}
830830
};
831-
// Adding a datasource discovers the schema only — never loads data.
832-
req.validate_only = true;
831+
// Adding a datasource discovers the schema only — never loads data — EXCEPT
832+
// a continuous one, which is a persistent, self-loading datasource the
833+
// scheduler keeps synced. Sending validate_only with continuous is
834+
// contradictory (a one-off preview that is also permanently synced): the
835+
// worker rejects the pair 422, and before it did, the datasource was
836+
// re-run every tick but fell to the full-replace path — reloading the whole
837+
// bucket forever. So continuous datasources are created ready to sync.
838+
req.validate_only = !req.continuous;
833839
req.name = args.name;
834840
req.database_id = args.database_id;
835841
Ok(req)
@@ -1896,13 +1902,17 @@ mod tests {
18961902
assert_eq!(req.family, "filesystem");
18971903
assert_eq!(req.bucket_url.as_deref(), Some("s3://b/prefix"));
18981904
assert!(req.continuous); // --continuous rides through to the request body
1905+
// A continuous datasource is self-loading, so it is NOT validate_only —
1906+
// the worker 422s the pair, and it was the original full-replace bug.
1907+
assert!(!req.validate_only);
18991908

19001909
// Default is off, and it serializes only when true (skip_serializing_if).
19011910
let mut off = create_args();
19021911
off.bucket_url = Some("s3://b".into());
19031912
off.format = Some("jsonl".into());
19041913
let req_off = build_create_request(&e, off, None).unwrap();
19051914
assert!(!req_off.continuous);
1915+
assert!(req_off.validate_only); // a non-continuous add still discovers schema only
19061916
assert!(
19071917
!serde_json::to_string(&req_off)
19081918
.unwrap()

0 commit comments

Comments
 (0)