Skip to content

Add opt-in URL deduplication and persistent success caching - #25

Merged
dblock merged 1 commit into
mainfrom
feature/deduplication-and-success-cache
Nov 7, 2026
Merged

dblock merged 1 commit into
mainfrom
feature/deduplication-and-success-cache

Conversation

@dblock

@dblock dblock commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Changes

Add optional deduplicate: true: coalesce identical in-flight requests, replay earlier events for late subscribers, reuse successes for the checker lifetime, and recheck completed failures/errors. Every submission retains its original URL and metadata, including with results: false.

Request identity includes the full URL (query values and fragments remain distinct), backend, methods, headers, user-agent, and request settings. Responses are not memoized if settings changed while queued. URLs containing user-info credentials bypass reuse.

Add optional cache: { file: ..., ttl: ... } with a required positive TTL. Store only successful statuses/methods/final URLs and timestamps under hashed request identities; no metadata or request/response headers are persisted. Reads tolerate missing, corrupt, malformed, or unreadable data. Writes merge under a file lock and atomically replace the JSON file. Cache failures do not fail successful link checks; credential-bearing redirect targets are not written.

Cache hits expose cached? and emit fresh result/success events. Persistent hits use LinkChecker::CachedResult rather than native response objects. Caching alone does not coalesce in-flight work; deduplication reuse lasts for the checker lifetime independently of the disk TTL.

Validation

bundle exec rubocop -a passed. Full RSpec suite: 166 examples, zero failures. Coverage includes all-backend metadata fanout, failure rechecks, distinct query values, header identity, persistence across instances, expiry, corruption recovery, merged writes, credential exclusion, IO failures, and reentrant/in-flight subscriptions. git diff --check passed.

Independent of #23 and #24; shared replay-fixture fixes are identical. Neither HTML parsing nor fragment validation is added.

Implemented by AI on behalf of @dblock.

@codecov-commenter

codecov-commenter commented Nov 7, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.45763% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.30%. Comparing base (70ff87e) to head (e63116b).

Files with missing lines Patch % Lines
lib/ruby-link-checker/cache.rb 96.61% 2 Missing ⚠️
lib/ruby-link-checker/checker.rb 98.11% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   99.66%   99.30%   -0.37%     
==========================================
  Files          25       26       +1     
  Lines         603      719     +116     
==========================================
+ Hits          601      714     +113     
- Misses          2        5       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dblock
dblock force-pushed the feature/deduplication-and-success-cache branch from ec5a843 to 0835bd0 Compare November 7, 2026 12:57
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Danger Report

No issues found.

View run

@dblock
dblock force-pushed the feature/deduplication-and-success-cache branch from 0835bd0 to 4034199 Compare November 7, 2026 20:40
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dblock
dblock force-pushed the feature/deduplication-and-success-cache branch from 4034199 to e63116b Compare November 7, 2026 20:43
@dblock
dblock merged commit 79c8b50 into main Nov 7, 2026
10 checks passed
@dblock
dblock deleted the feature/deduplication-and-success-cache branch November 7, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants