Skip to content

Feature/db repair totals 1051 - #1339

Open
nagisml wants to merge 3 commits into
OpenSAK-Org:betafrom
nagisml:feature/db-repair-totals-1051
Open

nagisml wants to merge 3 commits into
OpenSAK-Org:betafrom
nagisml:feature/db-repair-totals-1051

Conversation

@nagisml

@nagisml nagisml commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Recalculate log totals and the found mark in the database Repair dialog

Fixes #1051 (phase 3 of 3: totals and found). Depends on #1338. Log cleanup was moved to #1310.

What changed and why

A cache keeps values derived from its logs, waypoints and trackables, so the grid, filters and sorting don't have to count them each time. The importers and macros keep them up to date. They can still drift: in older databases, after raw SQL, or after a repair has removed rows or cleared log dates.

The totals are recalculated as "what is in the database", the same rules every import and macro already uses. The new code is in db/repair_totals.py.

Value Rule
log_count, last_log_date, last_found_date, last_four_logs From the cache's logs. Logs with the same date may be in any order.
waypoint_count, trackable_count From the cache's rows
found_log_count The user's own found-type logs. Only raised, never lowered: a missing log doesn't prove fewer finds (a PQ brings only some logs, and #1310 will delete logs).
find_count Not derived from logs: never touched

Checks

Check Fix
Counts or dates that don't match the cache's rows Recalculate. The dialog shows old → new.
Caches with one of the user's own found logs that aren't marked found Mark found. An empty found date is set from the latest own found log (log_calendar_midnight(), #875).
  • found is never cleared: Adventure Labs and manual marks have no log behind them.
  • dnf is not checked: the importers only take it from GSAK's flag.
  • Account needed: the checks of the user's own logs need the account from Settings and are skipped without one, as the importer does.
  • Order of fixes: the repair recalculates the totals after deleting rows without a cache and fixing dates, in the same transaction, so the totals match the repaired logs.
  • Memory: caches and logs are read in one pass ordered by cache id, so memory stays flat.

Tests

tests/unit-tests/test_db_repair_totals_1051.py covers every derived column; logs with the same date; raising but never lowering found_log_count; matching own logs by id or name; and marking found while keeping an existing found date. It also covers: no account; found without a log being kept; logs without a cache; only the ticked fix running; recalculating after the date fixes; and the dialog.

On the real database (152k caches, 732k logs), every stored value matched these rules, and 83 caches marked found without an own log (Labs, manual marks) were correctly left alone. The check made the scan 4 s longer, with an 11 MB peak.

The new texts are in all 11 language files. The translations are machine-made.

To confirm

@AgreeDK asked in #1051 whether totals mean "what's in the database" or "what we know about the cache". This PR uses "what's in the database", as every import and macro does today. find_count (the site's or GSAK's total, #1028) is never recalculated, and found_log_count is never lowered. The real conflict only appears once #1310 deletes logs. Then log_count would drop, and #1310 can decide how to handle it. OK?

The Maintenance dialog (OpenSAK-Org#1048) only checks the database file. Repair checks
the data in it and fixes what can be fixed safely, like GSAK's
Repair/Defrag. This is phase 1 of OpenSAK-Org#1051: the check framework, rows whose
cache is missing, broken links, and checks that only report.

Repair… opens a dialog that checks the database in the background without
changing it. It lists every check that found something, with the number of
problems, the fix and a few example GC codes. The user ticks what to fix.
Checks marked "Report only" can't be ticked. "Make a backup first" is ticked
and uses the backup core (OpenSAK-Org#952). The ticked fixes run in one transaction.
Then the database is optimized, and compacted when more than a quarter of
the file is free, as GSAK does. A summary follows and the check runs again.
If quick_check fails, the file itself is damaged and SQLite can't repair
it, so the dialog points to File → Restore from backup… (OpenSAK-Org#954) instead.

Fixable checks:
- rows that point to a missing cache, found with foreign_key_check (logs,
  waypoints, attributes, trackables, user notes): deleted
- custom waypoints whose parent_gc_code points to a cache that isn't in the
  database: link cleared
- waypoints whose parent_gc_code isn't their own cache's code: set from the
  cache, as the importer and macros do
- the corrected flag set without corrected coordinates, so filters count the
  cache as corrected while the map and distances don't: flag cleared. The
  cache keeps its last_user_update (the OpenSAK-Org#1140 trigger would set it to now),
  because a repair is not a user edit.
- companion database rows whose cache is gone: companion.cleanup()

Report only: coordinates missing or out of range, corrected coordinates out
of range, difficulty or terrain outside 1–5 or not in 0.5 steps, an empty
GC code or name.

There is no duplicate check. The unique constraints on attributes, log_id
and waypoints exist in every database, and the importers don't fill their
NULL gaps twice.

The checks and fixes are in db/repair.py, which has no GUI code. The dialog
works on the active database (its connections are released first, as for
Compact) and on any other database in the list.

The new texts are in all language files; the translations are machine-made.

Part of OpenSAK-Org#1051
Phase 2 of OpenSAK-Org#1051. SQLite has no date type. SQLAlchemy stores a DateTime as
text ("YYYY-MM-DD HH:MM:SS.ffffff") and reads it back with
datetime.fromisoformat(), in both 2.0 and 2.1. If a value can't be read,
loading the cache, its logs or its waypoints fails. A value in another
format may still load, but it compares wrongly as text, and OpenSAK's
filters and sorting compare dates as text. A DATETIME column has NUMERIC
affinity, so "20240305" is stored as the integer 20240305.

OpenSAK itself always writes the stored format: the ORM, the GSAK fast path
through Core, the triggers and the macros' cache writes. Other formats only
come from raw SQL in macros or from other tools.

Every DATETIME column of the database is checked, so new columns are
covered too. Values in the stored format are recognized in SQL, and only
the other values are read into Python. New checks:

- Dates in a format OpenSAK doesn't use (fixable): converted to the stored
  format. This covers ISO dates with or without a time, a "T" or an offset;
  one-digit month or day; "/" or "." as separator; DD.MM.YYYY; and an
  8-digit YYYYMMDD number. An offset is converted to UTC, as the GPX
  importer does (OpenSAK-Org#617), so a repair and a re-import store the same value.
- Dates that can't be read (fixable): cleared. This covers impossible dates
  such as 2024-02-30 or 0000-00-00, empty text, other numbers and blobs.
  The two NOT NULL bookkeeping columns, caches.imported_at and
  user_notes.updated_at, are set to now, as their database default does.
- Placeholder date 1970-01-01 (fixable): cleared. GSAK stores the Unix
  epoch as "no date" (Waypoints.cDate), and the GSAK import copies it. A
  real 150k-cache database had 792 such waypoints on 318 caches, exactly as
  many as in GSAK.
- Dates before May 2000 or more than a day in the future (report only).
  The placed date of an event or a GPS Adventures Maze exhibit may be in
  the future, because it is the day the event or exhibit takes place
  (e.g. GCBGCCC).

Each affected row is now listed with its GC code, the field (for a
waypoint also its prefix, e.g. "waypoints.wp_date S1") and the value as
stored. This applies to every check, so report-only problems can be found
and fixed by hand. Each check keeps up to 1000 rows; the count is always
complete. The examples cell shows the first five, and its tooltip shows
twenty. "Copy list", the context menu and Ctrl+C copy them for the
selected rows, or all rows, as tab-separated text (check, GC code, field,
value), ready for a spreadsheet.

The new texts are in all language files; the translations are machine-made.

Part of OpenSAK-Org#1051
Phase 3 of OpenSAK-Org#1051. A cache keeps values derived from its logs, waypoints
and trackables, so the grid, the filters and sorting don't have to count
them each time. The importers and macros keep these values up to date.
They can still drift: in older databases, after raw SQL, or after a repair
has removed rows or cleared log dates.

The totals mean "what is in the database", which is how every import and
macro already computes them:

- log_count, last_log_date, last_found_date (latest found-type log by
  anyone) and last_four_logs (logs with the same date may be in any order,
  so the same lines in another order are not a difference)
- waypoint_count and trackable_count
- found_log_count, the user's own found-type logs. It is only raised, never
  lowered: a missing log doesn't prove fewer finds, because a Pocket Query
  brings only some logs and OpenSAK-Org#1310 will delete logs.

find_count (the site's total, or GSAK's) is not derived from the logs and
is never touched.

New checks:
- Counts or dates that don't match the cache's rows (fixable): recalculated.
  The dialog lists each change as "old → new".
- Caches with one of the user's own found logs that aren't marked found
  (fixable): marked found. If the found date is empty, it is set from the
  latest own found log with log_calendar_midnight() (OpenSAK-Org#875).

found is never cleared, because Adventure Labs and manual marks have no
log behind them. dnf is not checked, because the importers only take it
from GSAK's flag. The checks of the user's own logs need the account from
Settings and are skipped without one, as the importer does.

The totals are calculated from the log dates as the date checks leave them
(converted, or no date when unreadable or 1970-01-01). The repair
recalculates them after deleting rows without a cache and fixing dates, in
the same transaction.

Caches and logs are read in one pass ordered by cache id, so memory stays
flat. On a real database of 152k caches and 732k logs, every stored value
matched these rules, and the scan took 4 s longer with an 11 MB peak.

The new texts are in all language files; the translations are machine-made.

Fixes OpenSAK-Org#1051

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant