Conversation
Adds the schema half of feature history. A table definition may now set history = true, which makes osm2pgsql: - add a "valid_from timestamptz DEFAULT now()" column to the table. It is create_only, so it never appears in the COPY column list and the database default stamps it on every insert. The config cannot declare valid_from or valid_to itself; doing so is an error, so a table with history can never end up without a validity interval. - create <table>_history, mirroring the table column for column plus "valid_to", which closes the interval of a superseded version. Mirroring exactly lets the rows deleted from the live table be inserted without naming columns. The history table is created IF NOT EXISTS and is never dropped, including in create mode, so history outlives a re-import. Moving superseded rows into it, and the create-mode schema check, follow in later commits. Refs: MAPCO-11663
Row replacement in append mode is a plain DELETE, so the old version of every modified or deleted feature was discarded. For a table with history enabled that DELETE becomes a data-modifying CTE which inserts the deleted rows into <table>_history with now() as valid_to, closing their validity interval. Both deleters are covered: the by-id form and the by-type-and-id form, whose USING (VALUES ...) join needs RETURNING p.* rather than a bare * so the VALUES columns are not returned alongside the row. The history table is an exact column-for-column mirror of the live table, so the insert needs no column list and gfid, the geometry and every attribute column ride along unchanged. Tables without history are unaffected: the generated SQL is identical to before. Stage-2 and pending-object reprocessing also delete and re-insert, which produces empty [now, now) intervals. Those never match a point-in-time query, so they need no filtering. Refs: MAPCO-11663
The history table was created in append mode too, which looks convenient but cannot work: valid_from is only added to the live table when that table is created, so on a database predating history the live table has n columns while the history table has n+2. DELETE ... RETURNING * then yields n values for an n+1 column insert and every modified feature fails, at run time, after a clean startup. History is established by a create, as ADR-0003 has it. Enabling it on an existing database needs a re-import, or an ALTER TABLE adding valid_from before the next run. Verifying this in append mode belongs with the schema check, which is still to come. Refs: MAPCO-11663
Append mode replaces a row by deleting and re-inserting it, and the delete runs before the copy (db-copy.cpp:213 then :216), so the history CTE cannot tell a real edit from a rewrite of identical content. Two things produce the latter: re-ingesting a diff that was already applied, and node changes, which mark every way containing the node as pending and rewrite it unchanged. Both leave a history version that records no change. After the copy has synced, and only then, each superseded row this run stamped is compared against the row now live and dropped when every written column matches. Scoped by valid_to to the current run's stamp rather than the whole table: on the buildings table that is 20 rows to consider instead of 49231, so the cost follows the change, not the corpus. create_only columns are excluded from the comparison — osm2pgsql never writes them, so valid_from and a derived geom3d cannot count as a difference. Refs: MAPCO-11663
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.