Repository navigation
Shadow via bulk load: skip indexes during copy, rebuild after - #1
Merged
Merged
Conversation
carldaws
requested changes
Apr 17, 2026
carldaws
left a comment
carldaws
left a comment
Owner
There was a problem hiding this comment.
This was my next move and you beat me to it. A few things to fix though.
CREATE TABLE LIKE ... INCLUDING ALL creates all indexes up front, so the subsequent INSERT SELECT pays for per-row index maintenance on every index. On large tables this dominates shadow time. Instead, create the shadow with INCLUDING ALL EXCLUDING INDEXES, bulk copy the rows, then recreate indexes in a single sort pass each. The primary key is attached via ALTER TABLE ... USING INDEX so it remains a PRIMARY KEY constraint rather than just a unique index. Benchmarks on a widgets table with 1 PK + 5 indexes (1 unique, 1 composite, 1 partial, 2 plain): 1M rows: 14.05s -> 5.98s (2.35x) 3M rows: 51.18s -> 19.25s (2.66x) Speedup grows with scale because sort-based index builds amortize better than per-row B-tree inserts. Semantics are preserved: same indexes, same PK designation, same uniqueness, same check constraints, same defaults and NOT NULL — covered by new tests pinning each of these. Index redirection is done in SQL via REPLACE with quote_ident(), which matches pg_get_indexdef's own output exactly. Tables whose names need quoting (uppercase, reserved words) work as a result, and the single catalog query returns index name, is-primary flag, and pre-rewritten CREATE INDEX statement together.
mhenrixon
force-pushed
the
fast-shadow-bulk-load
branch
from
April 17, 2026 16:23
7e7488e to
26d037f
Compare
Contributor
Author
|
Great feedback! Thanks for that |
carldaws
approved these changes
Apr 17, 2026
Owner
|
Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Shadowing a table that already has indexes pays for per-row B-tree maintenance on every insert.
CREATE TABLE ... LIKE ... INCLUDING ALLcreates all indexes up front, and thenINSERT INTO ... SELECT * FROM public.Xupdates each index for every row. On a table with millions of rows and several indexes this dominates the shadow runtime — and the shadow runs the first time any migration touches a given table, right when a developer is waiting for their feature branch to come up.Fix
Flip the order:
public.Xon the branch copy in a single sort pass. The primary key is re-attached viaALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY USING INDEXso it remains a truePRIMARY KEYrather than just a unique index.This is the standard Postgres bulk-load pattern: drop indexes, load data, rebuild. Per-row random-I/O becomes per-index sequential sorting.
Numbers
Widgets table with 1 PK + 5 indexes (one unique, one composite, one partial, two plain). Single-node Postgres 16, local SSD.
INCLUDING ALL+ INSERT SELECT)Speedup grows with scale because sort-based index builds amortize better than per-row B-tree inserts. On tables with more indexes or slower disks the gap widens further. A reproducible benchmark script is included under
bench/shadow_bench.rb.Semantics
Nothing observable changes. Seven new tests in
test/shadow_test.rbpin the parts the old code got for free viaINCLUDING ALL:PRIMARY KEYconstraint (pg_index.indisprimary), not just a unique index — matters for ActiveRecord's PK lookuppg_index.indisunique+ insert that shouldRecordNotUnique)NOT NULLand column defaults are preservedAll existing tests still pass (76 runs, 174 assertions).
Not addressed (can be follow-ups)
CREATE TABLE LIKEstill doesn't copy foreign keys — same as before. Worth tackling separately.UNLOGGEDtable option for even more dev-only speedup by skipping WAL. Opening as a separate PR once this one lands.