Add OCR and sync logging overviews to settings by Copilot · Pull Request #67 · maxi07/ScanSync

Copilot · 2026-07-02T07:36:37Z

Only file naming jobs were surfaced in the web UI; OCR and sync (upload) jobs had no visibility. This adds OCR and Sync log tables mirroring the existing File Naming logs.

Persistence

sync_jobs table added to schema.sql; the pre-existing but unused ocr_jobs table is now populated.
ocr_service writes a PROCESSING row on start and finalizes with status + a human-readable error on every completion/failure branch.
upload_service records the sync lifecycle (including missing-OCR-file and upload-failure paths) with a success flag and error description; ProcessItem gains sync_db_id.

API

New GET /api/ocr-logs and GET /api/sync-logs, paginated and filterable (all/success/failed), via a shared _fetch_job_logs helper. Table/filter fragments are hardcoded constants; only pagination values are bound parameters.

Frontend

OCR Logs accordion in the OCR tab; Sync Logs accordion in the OneDrive tab.
The per-table logs JS is consolidated into a reusable createLogsTable factory (own pagination/filter state, lazy-load on expand) driving all three tables:

createLogsTable({
    endpoint: '/api/ocr-logs',
    collapseId: 'ocr-logsCollapse', tableId: 'ocr-logs-table', /* ... */
    renderRow: (log) => `<tr><td>${log.id}</td><td>${getStatusBadge(log.ocr_status)}</td>...</tr>`
});

Full file-name/error text is shown via a data-fulltext attribute + addEventListener rather than inline onclick string interpolation, avoiding injection from unescaped backslashes.

Tests

tests/test_logs_api.py covers pagination, filters, null-count handling, and error responses for both endpoints.

Note: the click-to-expand uses alert() for parity with the existing File Naming table; a more accessible modal is left as a potential follow-up.

Copilot

Pull request overview

This PR adds end-to-end visibility for OCR and OneDrive sync (upload) job activity in the Settings UI, bringing those services to parity with the existing File Naming logs by persisting job lifecycle data, exposing new paginated API endpoints, and rendering new log tables in the frontend.

Changes:

Added persistence for OCR and Sync job lifecycles (ocr_jobs population + new sync_jobs table) from ocr_service and upload_service.
Added new paginated/filterable API endpoints (/api/ocr-logs, /api/sync-logs) using a shared _fetch_job_logs helper.
Extended the Settings UI with OCR and Sync log accordions and refactored frontend log rendering into a reusable createLogsTable factory.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
web_service/src/templates/settings-tab/settings-tab-onedrive.html	Adds a Sync Logs accordion/table to the OneDrive settings tab.
web_service/src/templates/settings-tab/settings-tab-ocr.html	Replaces placeholder OCR text with an OCR Logs accordion/table.
web_service/src/static/js/settings.js	Introduces `createLogsTable`/`initLogTables` and shared rendering helpers for all log tables.
web_service/src/routes/api.py	Adds `_fetch_job_logs` plus new `/api/ocr-logs` and `/api/sync-logs` endpoints.
upload_service/main.py	Inserts/updates `sync_jobs` records across the upload lifecycle.
tests/test_logs_api.py	Adds API tests for OCR/sync logs pagination and filtering.
scansynclib/scansynclib/ProcessItem.py	Adds `sync_db_id` field to `ProcessItem`.
scansynclib/scansynclib/db/schema.sql	Adds the new `sync_jobs` table schema.
ocr_service/main.py	Inserts `ocr_jobs` rows at start and finalizes status/error on completion/failure.

Copilot · 2026-07-02T15:12:09Z

+    page = int(request.args.get('page', 1))
+    per_page = int(request.args.get('per_page', 20))
+    filter = request.args.get('filter', 'all').lower()
+    offset = (page - 1) * per_page
+


Fixed in the latest commit. Both params are now parsed inside try/except (ValueError, TypeError) blocks with safe defaults. page is clamped to max(1, ...) with no upper limit, and per_page is clamped to max(1, min(100, ...)) to prevent zero/negative values and unbounded queries.

Copilot · 2026-07-02T15:12:09Z

        logger.exception(f"Failed processing {body}.")
        item.status = ProcessStatus.SYNC_FAILED
        update_scanneddata_database(item, {"file_status": item.status.value})
+        finalize_sync_job(item, "Unexpected error during upload")


Fixed in the latest commit. item is now initialized to None before the try block. The except clause checks item is not None and isinstance(item, ProcessItem) before updating status/DB, and additionally checks item.sync_db_id is not None before calling finalize_sync_job.

Copilot · 2026-07-02T15:12:09Z

+# Mock Redis before any scansynclib imports, since settings.py connects at module level
+import redis as _real_redis
+_orig_from_url = _real_redis.Redis.from_url
+
+
+def _mock_from_url(*args, **kwargs):
+    mock_client = MagicMock()
+    mock_client.get.return_value = None  # No existing settings in Redis
+    mock_client.set.return_value = True
+    mock_client.publish.return_value = 0
+    mock_pubsub = MagicMock()
+    mock_pubsub.subscribe.return_value = None
+    mock_pubsub.listen.return_value = iter([])  # Empty iterator
+    mock_client.pubsub.return_value = mock_pubsub
+    return mock_client
+
+
+_real_redis.Redis.from_url = _mock_from_url
+


Fixed in the latest commit. The module-level assignment and the unused _orig_from_url have been removed. The patch is now applied and torn down in a @pytest.fixture(scope="session", autouse=True) fixture that saves the original, yields, and restores it—preventing any leak into other test modules.

…lback, use autouse fixture for Redis mock

maxi07 · 2026-07-02T15:46:52Z

@copilot I dont like your addition of the our status text within a JS file and the except blocks within ocr_service/main.py. Please, similar to the ProcessItem definition, define all the error descriptions for the OCR failures once and make them reusable, so we can display the message also to the user in the web instead of creating new error messages in the js file with getOcrStatusText.

…s in ocr_service and expose via API

Copilot · 2026-07-02T15:55:19Z

@copilot I dont like your addition of the our status text within a JS file and the except blocks within ocr_service/main.py. Please, similar to the ProcessItem definition, define all the error descrip...

Done. Here's what changed:

scansynclib/ProcessItem.py – OCRStatus now follows the same pattern as FileNamingStatus: each member's .value is its human-readable description (e.g. DPI_ERROR = "Image DPI too low for accurate OCR").
ocr_service/main.py – All ad-hoc string literals in except blocks are replaced with OCRStatus.<STATUS>.value, with exception detail appended where available (e.g. f"{OCRStatus.DPI_ERROR.value}: {dpiex}").
web_service/src/routes/api.py – The /api/ocr-logs response enriches each log entry with ocr_status_text = OCRStatus[status_name].value, so the frontend receives the server-defined description directly.
settings.js – getStatusBadge accepts an optional displayText parameter; the OCR logs row passes log.ocr_status_text so the badge uses the Python-defined text instead of a JS toTitleCase conversion.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

maxi07 · 2026-07-02T16:23:06Z

@copilot resolve the merge conflicts in this pull request

…sItem and ocr_service

Copilot · 2026-07-02T16:28:38Z

@copilot resolve the merge conflicts in this pull request

Done. Merged origin/main into this branch in commit e126ae6. The only conflicts were in scansynclib/ProcessItem.py and ocr_service/main.py:

ProcessItem.py: Kept our string enum values for OCRStatus and added the new NO_TEXT = "No text found in OCR output" member from main.
ocr_service/main.py: Used main's refined structure (result=None init, if item.ocr_db_id guard, OCRStatus.NO_TEXT for the no-text-found path) and removed the duplicate DB INSERT/ocr_error declarations left by the conflict markers. Error strings match the existing test expectations in test_ocr_job_tracking.py.

Initial plan

be72c78

Copilot AI assigned Copilot and maxi07 Jul 2, 2026

Copilot AI linked an issue Jul 2, 2026 that may be closed by this pull request

Add a logging overview #13

Closed

Copilot started work on behalf of maxi07 July 2, 2026 07:36 View session

Copilot AI added 4 commits July 2, 2026 07:42

Plan: add OCR and sync logging overview

8d8d4d7

Add OCR and sync logging overview to settings

9d40273

Harden log cell rendering and revert generated egg-info churn

43743b2

Guard toTitleCase against empty input

550baae

Copilot AI changed the title ~~[WIP] Add logging overview for sync and OCR processes~~ Add OCR and sync logging overviews to settings Jul 2, 2026

Copilot AI requested a review from maxi07 July 2, 2026 07:54

Copilot finished work on behalf of maxi07 July 2, 2026 07:54

maxi07 requested a review from Copilot July 2, 2026 14:59

Copilot started reviewing on behalf of maxi07 July 2, 2026 14:59 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Copilot started work on behalf of maxi07 July 2, 2026 15:07 View session

Fix review feedback: validate page/per_page, guard item in upload cal…

4c53bfd

…lback, use autouse fixture for Redis mock

Copilot finished work on behalf of maxi07 July 2, 2026 15:12

Copilot started work on behalf of maxi07 July 2, 2026 15:47 View session

Centralize OCR failure descriptions in OCRStatus enum; use enum value…

5de22e6

…s in ocr_service and expose via API

Copilot finished work on behalf of maxi07 July 2, 2026 15:56

maxi07 requested a review from Copilot July 2, 2026 16:08

Copilot started reviewing on behalf of maxi07 July 2, 2026 16:09 View session

maxi07 marked this pull request as ready for review July 2, 2026 16:12

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread ocr_service/main.py

Comment thread scansynclib/scansynclib.egg-info/SOURCES.txt Outdated

maxi07 and others added 2 commits July 2, 2026 18:13

Potential fix for pull request finding

46d80d0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

226d1d5

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot started work on behalf of maxi07 July 2, 2026 16:23 View session

Merge origin/main: add NO_TEXT OCRStatus, resolve conflicts in Proces…

e126ae6

…sItem and ocr_service

Copilot finished work on behalf of maxi07 July 2, 2026 16:29

set per page to 10

6f6bb16

maxi07 merged commit 512ca42 into main Jul 2, 2026
4 checks passed

maxi07 deleted the copilot/add-logging-overview branch July 2, 2026 17:13

Copilot AI mentioned this pull request Jul 4, 2026

Unify RabbitMQ handling with persistent, auto-reconnecting connections #70

Merged

Navigation Menu

Repository navigation

Add OCR and sync logging overviews to settings - #67

Add OCR and sync logging overviews to settings#67
maxi07 merged 11 commits into
mainfrom
copilot/add-logging-overview

Copilot AI commented Jul 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 2, 2026

Uh oh!

Copilot AI Jul 2, 2026

Uh oh!

Copilot AI Jul 2, 2026

Uh oh!

maxi07 commented Jul 2, 2026

Uh oh!

Copilot AI commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

maxi07 commented Jul 2, 2026

Uh oh!

Copilot AI commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Persistence

API

Frontend

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

maxi07 commented Jul 2, 2026

Uh oh!

Copilot AI commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

maxi07 commented Jul 2, 2026

Uh oh!

Copilot AI commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 2, 2026 •

edited

Loading