fix: dedupe copy_into(qualify=True) via temp table and merge

[pquadri]

copy_into(qualify=True) appended into the live table and then ran create or replace table ... qualify row_number() = 1, so readers saw duplicate keys for the minutes between the two steps. Route it through _merge instead: COPY into

_temp, dedupe there, MERGE into the live table. full_refresh keeps the old path.

• _merge takes sync_tags so the caller's flag is honoured
• re-sync tags after the first-load qualify, which drops them

Claude-Session: https://claude.ai/code/session_011PFLPuv77fp5M3JXdZvW1v

---

Note

Medium Risk
Changes core Snowflake load/merge behavior for qualified incremental copies; incorrect merge or replication-key logic could drop updates or leave stale rows, though behavior is heavily tested.

Overview
Fixes a visibility bug where copy_into(..., qualify=True) used to load into the live table and only then rebuild it with QUALIFY, so readers could see duplicate primary keys between those steps.

For the default case (not full_refresh), qualified loads now go through _merge: data is COPY’d into a uniquely named <table>_temp_<id> table, deduped there, then MERGE’d into the destination so the live table is never left in a duplicate-key state. full_refresh still copies into the live table and runs qualify on it afterward.

_merge gains sync_tags and target_columns from the caller, uses UUID-suffixed temp tables with cleanup on failure, limits MERGE UPDATE columns to loaded/target (+ PK/metadata) columns, and when deduping applies _is_newer_or_equal so matched rows update only when incoming replication keys are newer. README documents the new qualify flow; tests cover routing, full_refresh, merge SQL, temp cleanup, and column scoping.

^{Reviewed by Cursor Bugbot for commit 358d827. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0917d6f. Configure here.}

[giamo]

Another potentially breaking change is this doesn't allow multiple COPY INTOs at the same time since they'd all reuse the same temp table name. I don't think it's relevant for our pipelines, but we could use a unique temp table name per run to fix it

[pquadri]

Done in 358d827: unique temp table name per run, as suggested.

[giamo]

I just taught about a side effect of this - if the pod is killed while copying into the temp table, nobody cleans it up (and it's not tagged either, so we risk a leak). I think we should make the temp table TEMPORARY running all commands in the same connection?

[pquadri]

Follow-up in #36: the staging table is now TEMPORARY and the whole load runs in one session, so a killed pod no longer leaves a table in the schema (Snowflake discards it when that session expires). Thanks for the catch.