Repository navigation
Conversation
The Maintenance dialog (OpenSAK-Org#1048) only checks the database file. Repair checks the data in it and fixes what can be fixed safely, like GSAK's Repair/Defrag. This is phase 1 of OpenSAK-Org#1051: the check framework, rows whose cache is missing, broken links, and checks that only report. Repair… opens a dialog that checks the database in the background without changing it. It lists every check that found something, with the number of problems, the fix and a few example GC codes. The user ticks what to fix. Checks marked "Report only" can't be ticked. "Make a backup first" is ticked and uses the backup core (OpenSAK-Org#952). The ticked fixes run in one transaction. Then the database is optimized, and compacted when more than a quarter of the file is free, as GSAK does. A summary follows and the check runs again. If quick_check fails, the file itself is damaged and SQLite can't repair it, so the dialog points to File → Restore from backup… (OpenSAK-Org#954) instead. Fixable checks: - rows that point to a missing cache, found with foreign_key_check (logs, waypoints, attributes, trackables, user notes): deleted - custom waypoints whose parent_gc_code points to a cache that isn't in the database: link cleared - waypoints whose parent_gc_code isn't their own cache's code: set from the cache, as the importer and macros do - the corrected flag set without corrected coordinates, so filters count the cache as corrected while the map and distances don't: flag cleared. The cache keeps its last_user_update (the OpenSAK-Org#1140 trigger would set it to now), because a repair is not a user edit. - companion database rows whose cache is gone: companion.cleanup() Report only: coordinates missing or out of range, corrected coordinates out of range, difficulty or terrain outside 1–5 or not in 0.5 steps, an empty GC code or name. There is no duplicate check. The unique constraints on attributes, log_id and waypoints exist in every database, and the importers don't fill their NULL gaps twice. The checks and fixes are in db/repair.py, which has no GUI code. The dialog works on the active database (its connections are released first, as for Compact) and on any other database in the list. The new texts are in all language files; the translations are machine-made. Part of OpenSAK-Org#1051
Phase 2 of OpenSAK-Org#1051. SQLite has no date type. SQLAlchemy stores a DateTime as text ("YYYY-MM-DD HH:MM:SS.ffffff") and reads it back with datetime.fromisoformat(), in both 2.0 and 2.1. If a value can't be read, loading the cache, its logs or its waypoints fails. A value in another format may still load, but it compares wrongly as text, and OpenSAK's filters and sorting compare dates as text. A DATETIME column has NUMERIC affinity, so "20240305" is stored as the integer 20240305. OpenSAK itself always writes the stored format: the ORM, the GSAK fast path through Core, the triggers and the macros' cache writes. Other formats only come from raw SQL in macros or from other tools. Every DATETIME column of the database is checked, so new columns are covered too. Values in the stored format are recognized in SQL, and only the other values are read into Python. New checks: - Dates in a format OpenSAK doesn't use (fixable): converted to the stored format. This covers ISO dates with or without a time, a "T" or an offset; one-digit month or day; "/" or "." as separator; DD.MM.YYYY; and an 8-digit YYYYMMDD number. An offset is converted to UTC, as the GPX importer does (OpenSAK-Org#617), so a repair and a re-import store the same value. - Dates that can't be read (fixable): cleared. This covers impossible dates such as 2024-02-30 or 0000-00-00, empty text, other numbers and blobs. The two NOT NULL bookkeeping columns, caches.imported_at and user_notes.updated_at, are set to now, as their database default does. - Placeholder date 1970-01-01 (fixable): cleared. GSAK stores the Unix epoch as "no date" (Waypoints.cDate), and the GSAK import copies it. A real 150k-cache database had 792 such waypoints on 318 caches, exactly as many as in GSAK. - Dates before May 2000 or more than a day in the future (report only). The placed date of an event or a GPS Adventures Maze exhibit may be in the future, because it is the day the event or exhibit takes place (e.g. GCBGCCC). Each affected row is now listed with its GC code, the field (for a waypoint also its prefix, e.g. "waypoints.wp_date S1") and the value as stored. This applies to every check, so report-only problems can be found and fixed by hand. Each check keeps up to 1000 rows; the count is always complete. The examples cell shows the first five, and its tooltip shows twenty. "Copy list", the context menu and Ctrl+C copy them for the selected rows, or all rows, as tab-separated text (check, GC code, field, value), ready for a spreadsheet. The new texts are in all language files; the translations are machine-made. Part of OpenSAK-Org#1051
Phase 3 of OpenSAK-Org#1051. A cache keeps values derived from its logs, waypoints and trackables, so the grid, the filters and sorting don't have to count them each time. The importers and macros keep these values up to date. They can still drift: in older databases, after raw SQL, or after a repair has removed rows or cleared log dates. The totals mean "what is in the database", which is how every import and macro already computes them: - log_count, last_log_date, last_found_date (latest found-type log by anyone) and last_four_logs (logs with the same date may be in any order, so the same lines in another order are not a difference) - waypoint_count and trackable_count - found_log_count, the user's own found-type logs. It is only raised, never lowered: a missing log doesn't prove fewer finds, because a Pocket Query brings only some logs and OpenSAK-Org#1310 will delete logs. find_count (the site's total, or GSAK's) is not derived from the logs and is never touched. New checks: - Counts or dates that don't match the cache's rows (fixable): recalculated. The dialog lists each change as "old → new". - Caches with one of the user's own found logs that aren't marked found (fixable): marked found. If the found date is empty, it is set from the latest own found log with log_calendar_midnight() (OpenSAK-Org#875). found is never cleared, because Adventure Labs and manual marks have no log behind them. dnf is not checked, because the importers only take it from GSAK's flag. The checks of the user's own logs need the account from Settings and are skipped without one, as the importer does. The totals are calculated from the log dates as the date checks leave them (converted, or no date when unreadable or 1970-01-01). The repair recalculates them after deleting rows without a cache and fixing dates, in the same transaction. Caches and logs are read in one pass ordered by cache id, so memory stays flat. On a real database of 152k caches and 732k logs, every stored value matched these rules, and the scan took 4 s longer with an 11 MB peak. The new texts are in all language files; the translations are machine-made. Fixes OpenSAK-Org#1051
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recalculate log totals and the found mark in the database Repair dialog
Fixes #1051 (phase 3 of 3: totals and found). Depends on #1338. Log cleanup was moved to #1310.
What changed and why
A cache keeps values derived from its logs, waypoints and trackables, so the grid, filters and sorting don't have to count them each time. The importers and macros keep them up to date. They can still drift: in older databases, after raw SQL, or after a repair has removed rows or cleared log dates.
The totals are recalculated as "what is in the database", the same rules every import and macro already uses. The new code is in
db/repair_totals.py.log_count,last_log_date,last_found_date,last_four_logswaypoint_count,trackable_countfound_log_countfind_countChecks
old → new.log_calendar_midnight(), #875).foundis never cleared: Adventure Labs and manual marks have no log behind them.dnfis not checked: the importers only take it from GSAK's flag.Tests
tests/unit-tests/test_db_repair_totals_1051.pycovers every derived column; logs with the same date; raising but never loweringfound_log_count; matching own logs by id or name; and marking found while keeping an existing found date. It also covers: no account; found without a log being kept; logs without a cache; only the ticked fix running; recalculating after the date fixes; and the dialog.On the real database (152k caches, 732k logs), every stored value matched these rules, and 83 caches marked found without an own log (Labs, manual marks) were correctly left alone. The check made the scan 4 s longer, with an 11 MB peak.
The new texts are in all 11 language files. The translations are machine-made.
To confirm
@AgreeDK asked in #1051 whether totals mean "what's in the database" or "what we know about the cache". This PR uses "what's in the database", as every import and macro does today.
find_count(the site's or GSAK's total, #1028) is never recalculated, andfound_log_countis never lowered. The real conflict only appears once #1310 deletes logs. Thenlog_countwould drop, and #1310 can decide how to handle it. OK?