Repository navigation
Commit d687a37
committed
fix(ingest): continuous datasources are not validate_only
new-datasource unconditionally set validate_only=true ("discover schema, don't
load"), so --continuous produced a spec with BOTH continuous and validate_only.
That is the exact combination the worker now rejects (422): the scheduler
re-runs it every tick but the drain falls to full-replace, reloading the whole
bucket forever. It was the root cause of the otel_telemetry incident.
A continuous datasource is a persistent, self-loading source the scheduler keeps
synced — the opposite of a one-off schema preview. Set validate_only only when
NOT continuous. Without this, --continuous is unusable against a worker carrying
the guard (dlthubworker #193): every attempt 422s.1 parent bef5aec commit d687a37
1 file changed
Lines changed: 12 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
828 | 828 | | |
829 | 829 | | |
830 | 830 | | |
831 | | - | |
832 | | - | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
833 | 839 | | |
834 | 840 | | |
835 | 841 | | |
| |||
1896 | 1902 | | |
1897 | 1903 | | |
1898 | 1904 | | |
| 1905 | + | |
| 1906 | + | |
| 1907 | + | |
1899 | 1908 | | |
1900 | 1909 | | |
1901 | 1910 | | |
1902 | 1911 | | |
1903 | 1912 | | |
1904 | 1913 | | |
1905 | 1914 | | |
| 1915 | + | |
1906 | 1916 | | |
1907 | 1917 | | |
1908 | 1918 | | |
| |||
0 commit comments