Skip to content

Feature/db repair dates 1051 - #1338

Open
nagisml wants to merge 2 commits into
OpenSAK-Org:betafrom
nagisml:feature/db-repair-dates-1051
Open

nagisml wants to merge 2 commits into
OpenSAK-Org:betafrom
nagisml:feature/db-repair-dates-1051

Conversation

@nagisml

@nagisml nagisml commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Check and repair dates in the database Repair dialog

Part of #1051 (phase 2 of 3: dates). Depends on #1337.

What changed and why

SQLite has no date type. SQLAlchemy (2.0 and 2.1) stores a DateTime as text, YYYY-MM-DD HH:MM:SS.ffffff, and reads it back with datetime.fromisoformat().

  • Unreadable values break loading: if a value can't be read, loading the cache, its logs or its waypoints fails.
  • Other formats sort and filter wrongly: a value in another format may still load, but it compares wrongly as text, and filters and sorting compare dates as text.
  • NUMERIC affinity: a DATETIME column has NUMERIC affinity, so "20240305" is stored as an integer and fails to load.

OpenSAK itself always writes the stored format (ORM, the GSAK fast path through Core, triggers, macro writes). Other formats only come from raw SQL or other tools.

Every DATETIME column of the database is checked, so new columns are covered too. Values in the stored format are recognized in SQL, and only the other values are read into Python. The new code is in db/repair_dates.py.

Checks

Check Fix
Dates in a format OpenSAK doesn't use (ISO with or without time / T / offset, one-digit month or day, / or ., DD.MM.YYYY, YYYYMMDD number) Convert to the stored format. An offset is converted to UTC, as the GPX importer does (#617), so a repair and a re-import store the same value.
Dates that can't be read (2024-02-30, 0000-00-00, empty text, other numbers, blobs) Clear. The NOT NULL columns caches.imported_at and user_notes.updated_at get "now", like their default.
Placeholder date 1970-01-01 (GSAK's "no date" in Waypoints.cDate, copied by the GSAK import) Clear
Dates before May 2000 or more than a day in the future Report only. The placed date of an event or a GPS Adventures Maze exhibit may be in the future (e.g. GCBGCCC).

Seeing and copying the affected rows

  • Field and value in every row: every check (not only dates) now lists each affected row with its GC code, the field and the value as stored, e.g. GCBGCCC hidden_date 2027-07-24 00:00:00.000000. A waypoint also shows its prefix (waypoints.wp_date S1), so it can be found in GSAK too.
  • Tooltip: the examples cell shows five rows, and its tooltip shows twenty.
  • Copy list: a new button, plus the context menu and Ctrl+C. It copies up to 1000 rows per check (the count is always complete) as tab-separated text (check, GC code, field, value), so report-only problems can be fixed by hand.

Tests

tests/unit-tests/test_db_repair_dates_1051.py covers:

  • reading each format, including the unreadable ones;
  • the checks and fixes, with a regression test: a cache with a bad date fails to load before the repair and loads after it;
  • NOT NULL columns, columns of other tables, report-only dates, and Maze exhibits;
  • the 1970 placeholder;
  • copying the list.

On the real database: 792 placeholder waypoints on 318 caches, exactly as many as GSAK's own Waypoints.cDate holds.

The new texts are in all 11 language files. The translations are machine-made.

To confirm

  1. Converting dates that load: @AgreeDK wrote that values should only change automatically when they break loading. This PR also converts dates that load but are in another format, because the format breaks sorting and filtering and no information is lost. It's a separate check, so it can be unticked. OK?
  2. The 1970 placeholder is fixable (clear). A GSAK re-import brings it back, so a follow-up issue could make the GSAK importer read 1970-01-01 as "no date".

The Maintenance dialog (OpenSAK-Org#1048) only checks the database file. Repair checks
the data in it and fixes what can be fixed safely, like GSAK's
Repair/Defrag. This is phase 1 of OpenSAK-Org#1051: the check framework, rows whose
cache is missing, broken links, and checks that only report.

Repair… opens a dialog that checks the database in the background without
changing it. It lists every check that found something, with the number of
problems, the fix and a few example GC codes. The user ticks what to fix.
Checks marked "Report only" can't be ticked. "Make a backup first" is ticked
and uses the backup core (OpenSAK-Org#952). The ticked fixes run in one transaction.
Then the database is optimized, and compacted when more than a quarter of
the file is free, as GSAK does. A summary follows and the check runs again.
If quick_check fails, the file itself is damaged and SQLite can't repair
it, so the dialog points to File → Restore from backup… (OpenSAK-Org#954) instead.

Fixable checks:
- rows that point to a missing cache, found with foreign_key_check (logs,
  waypoints, attributes, trackables, user notes): deleted
- custom waypoints whose parent_gc_code points to a cache that isn't in the
  database: link cleared
- waypoints whose parent_gc_code isn't their own cache's code: set from the
  cache, as the importer and macros do
- the corrected flag set without corrected coordinates, so filters count the
  cache as corrected while the map and distances don't: flag cleared. The
  cache keeps its last_user_update (the OpenSAK-Org#1140 trigger would set it to now),
  because a repair is not a user edit.
- companion database rows whose cache is gone: companion.cleanup()

Report only: coordinates missing or out of range, corrected coordinates out
of range, difficulty or terrain outside 1–5 or not in 0.5 steps, an empty
GC code or name.

There is no duplicate check. The unique constraints on attributes, log_id
and waypoints exist in every database, and the importers don't fill their
NULL gaps twice.

The checks and fixes are in db/repair.py, which has no GUI code. The dialog
works on the active database (its connections are released first, as for
Compact) and on any other database in the list.

The new texts are in all language files; the translations are machine-made.

Part of OpenSAK-Org#1051
Phase 2 of OpenSAK-Org#1051. SQLite has no date type. SQLAlchemy stores a DateTime as
text ("YYYY-MM-DD HH:MM:SS.ffffff") and reads it back with
datetime.fromisoformat(), in both 2.0 and 2.1. If a value can't be read,
loading the cache, its logs or its waypoints fails. A value in another
format may still load, but it compares wrongly as text, and OpenSAK's
filters and sorting compare dates as text. A DATETIME column has NUMERIC
affinity, so "20240305" is stored as the integer 20240305.

OpenSAK itself always writes the stored format: the ORM, the GSAK fast path
through Core, the triggers and the macros' cache writes. Other formats only
come from raw SQL in macros or from other tools.

Every DATETIME column of the database is checked, so new columns are
covered too. Values in the stored format are recognized in SQL, and only
the other values are read into Python. New checks:

- Dates in a format OpenSAK doesn't use (fixable): converted to the stored
  format. This covers ISO dates with or without a time, a "T" or an offset;
  one-digit month or day; "/" or "." as separator; DD.MM.YYYY; and an
  8-digit YYYYMMDD number. An offset is converted to UTC, as the GPX
  importer does (OpenSAK-Org#617), so a repair and a re-import store the same value.
- Dates that can't be read (fixable): cleared. This covers impossible dates
  such as 2024-02-30 or 0000-00-00, empty text, other numbers and blobs.
  The two NOT NULL bookkeeping columns, caches.imported_at and
  user_notes.updated_at, are set to now, as their database default does.
- Placeholder date 1970-01-01 (fixable): cleared. GSAK stores the Unix
  epoch as "no date" (Waypoints.cDate), and the GSAK import copies it. A
  real 150k-cache database had 792 such waypoints on 318 caches, exactly as
  many as in GSAK.
- Dates before May 2000 or more than a day in the future (report only).
  The placed date of an event or a GPS Adventures Maze exhibit may be in
  the future, because it is the day the event or exhibit takes place
  (e.g. GCBGCCC).

Each affected row is now listed with its GC code, the field (for a
waypoint also its prefix, e.g. "waypoints.wp_date S1") and the value as
stored. This applies to every check, so report-only problems can be found
and fixed by hand. Each check keeps up to 1000 rows; the count is always
complete. The examples cell shows the first five, and its tooltip shows
twenty. "Copy list", the context menu and Ctrl+C copy them for the
selected rows, or all rows, as tab-separated text (check, GC code, field,
value), ready for a spreadsheet.

The new texts are in all language files; the translations are machine-made.

Part of OpenSAK-Org#1051

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant