Repository navigation
Conversation
The Maintenance dialog (OpenSAK-Org#1048) only checks the database file. Repair checks the data in it and fixes what can be fixed safely, like GSAK's Repair/Defrag. This is phase 1 of OpenSAK-Org#1051: the check framework, rows whose cache is missing, broken links, and checks that only report. Repair… opens a dialog that checks the database in the background without changing it. It lists every check that found something, with the number of problems, the fix and a few example GC codes. The user ticks what to fix. Checks marked "Report only" can't be ticked. "Make a backup first" is ticked and uses the backup core (OpenSAK-Org#952). The ticked fixes run in one transaction. Then the database is optimized, and compacted when more than a quarter of the file is free, as GSAK does. A summary follows and the check runs again. If quick_check fails, the file itself is damaged and SQLite can't repair it, so the dialog points to File → Restore from backup… (OpenSAK-Org#954) instead. Fixable checks: - rows that point to a missing cache, found with foreign_key_check (logs, waypoints, attributes, trackables, user notes): deleted - custom waypoints whose parent_gc_code points to a cache that isn't in the database: link cleared - waypoints whose parent_gc_code isn't their own cache's code: set from the cache, as the importer and macros do - the corrected flag set without corrected coordinates, so filters count the cache as corrected while the map and distances don't: flag cleared. The cache keeps its last_user_update (the OpenSAK-Org#1140 trigger would set it to now), because a repair is not a user edit. - companion database rows whose cache is gone: companion.cleanup() Report only: coordinates missing or out of range, corrected coordinates out of range, difficulty or terrain outside 1–5 or not in 0.5 steps, an empty GC code or name. There is no duplicate check. The unique constraints on attributes, log_id and waypoints exist in every database, and the importers don't fill their NULL gaps twice. The checks and fixes are in db/repair.py, which has no GUI code. The dialog works on the active database (its connections are released first, as for Compact) and on any other database in the list. The new texts are in all language files; the translations are machine-made. Part of OpenSAK-Org#1051
Phase 2 of OpenSAK-Org#1051. SQLite has no date type. SQLAlchemy stores a DateTime as text ("YYYY-MM-DD HH:MM:SS.ffffff") and reads it back with datetime.fromisoformat(), in both 2.0 and 2.1. If a value can't be read, loading the cache, its logs or its waypoints fails. A value in another format may still load, but it compares wrongly as text, and OpenSAK's filters and sorting compare dates as text. A DATETIME column has NUMERIC affinity, so "20240305" is stored as the integer 20240305. OpenSAK itself always writes the stored format: the ORM, the GSAK fast path through Core, the triggers and the macros' cache writes. Other formats only come from raw SQL in macros or from other tools. Every DATETIME column of the database is checked, so new columns are covered too. Values in the stored format are recognized in SQL, and only the other values are read into Python. New checks: - Dates in a format OpenSAK doesn't use (fixable): converted to the stored format. This covers ISO dates with or without a time, a "T" or an offset; one-digit month or day; "/" or "." as separator; DD.MM.YYYY; and an 8-digit YYYYMMDD number. An offset is converted to UTC, as the GPX importer does (OpenSAK-Org#617), so a repair and a re-import store the same value. - Dates that can't be read (fixable): cleared. This covers impossible dates such as 2024-02-30 or 0000-00-00, empty text, other numbers and blobs. The two NOT NULL bookkeeping columns, caches.imported_at and user_notes.updated_at, are set to now, as their database default does. - Placeholder date 1970-01-01 (fixable): cleared. GSAK stores the Unix epoch as "no date" (Waypoints.cDate), and the GSAK import copies it. A real 150k-cache database had 792 such waypoints on 318 caches, exactly as many as in GSAK. - Dates before May 2000 or more than a day in the future (report only). The placed date of an event or a GPS Adventures Maze exhibit may be in the future, because it is the day the event or exhibit takes place (e.g. GCBGCCC). Each affected row is now listed with its GC code, the field (for a waypoint also its prefix, e.g. "waypoints.wp_date S1") and the value as stored. This applies to every check, so report-only problems can be found and fixed by hand. Each check keeps up to 1000 rows; the count is always complete. The examples cell shows the first five, and its tooltip shows twenty. "Copy list", the context menu and Ctrl+C copy them for the selected rows, or all rows, as tab-separated text (check, GC code, field, value), ready for a spreadsheet. The new texts are in all language files; the translations are machine-made. Part of OpenSAK-Org#1051
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check and repair dates in the database Repair dialog
Part of #1051 (phase 2 of 3: dates). Depends on #1337.
What changed and why
SQLite has no date type. SQLAlchemy (2.0 and 2.1) stores a DateTime as text,
YYYY-MM-DD HH:MM:SS.ffffff, and reads it back withdatetime.fromisoformat().DATETIMEcolumn has NUMERIC affinity, so"20240305"is stored as an integer and fails to load.OpenSAK itself always writes the stored format (ORM, the GSAK fast path through Core, triggers, macro writes). Other formats only come from raw SQL or other tools.
Every DATETIME column of the database is checked, so new columns are covered too. Values in the stored format are recognized in SQL, and only the other values are read into Python. The new code is in
db/repair_dates.py.Checks
T/ offset, one-digit month or day,/or.,DD.MM.YYYY,YYYYMMDDnumber)2024-02-30,0000-00-00, empty text, other numbers, blobs)caches.imported_atanduser_notes.updated_atget "now", like their default.1970-01-01(GSAK's "no date" inWaypoints.cDate, copied by the GSAK import)Seeing and copying the affected rows
GCBGCCC hidden_date 2027-07-24 00:00:00.000000. A waypoint also shows its prefix (waypoints.wp_date S1), so it can be found in GSAK too.Tests
tests/unit-tests/test_db_repair_dates_1051.pycovers:On the real database: 792 placeholder waypoints on 318 caches, exactly as many as GSAK's own
Waypoints.cDateholds.The new texts are in all 11 language files. The translations are machine-made.
To confirm
1970-01-01as "no date".