Refresh Exodus tracker data on a schedule instead of by hand - #20
Merged
Conversation
exodusTrackers.json had not been refreshed since January 2025. Rather than fetch it during the Railway build or at boot, which would make every deploy depend on reports.exodus-privacy.eu.org being up for data that changes a handful of times a year, refresh it on a schedule and land it as a PR. - scripts/update-exodus-trackers.js fetches, validates and normalises the Exodus API payload, writing only when the content actually changed. It keeps just the fields views/form.pug reads (id, name, categories) and sorts keys and categories: Exodus returns categories in an unstable order, so writing the raw response produced ~90 spurious entry changes per refresh and buried the real ones. Also emits a Markdown change summary for the PR body. - .github/workflows/update-exodus-trackers.yml runs it monthly and opens or updates a PR when something changed. - Refreshed the data itself: adds ACRA, Backtrace, Sentry and Yueying Crash SDK, renames Gigya to SAP CDC (Gigya), and fixes categories for AccountKit, AdTiming and MixPanel. Of these only Sentry matches a name the analyser can currently detect. The rest of the diff is the one-time reformat, which also takes the file from 570 KB to 50 KB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TT3FoeNrXRT7A6BETh3yc1
kasnder
marked this pull request as ready for review
August 12, 2026 16:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
exodusTrackers.jsonhad not been refreshed since January 2025. This adds a refresh script and a monthly workflow that opens a PR when the data actually changed, and applies the pending refresh.Why not fetch at build or boot time
The obvious alternative is to fetch the catalogue during the Railway build. That would attach a third-party network call to the highest-frequency operation in the project —
maintook ~70 pushes in the last four months — in service of data that changed meaningfully 8 times in 19 months.routes/index.jsreads the file withreadFileSync+JSON.parseat module load, so a truncated or failed fetch means the container does not boot, blocking unrelated hotfixes whenever Exodus has a bad afternoon. It would also make deploys non-reproducible and ship unreviewed third-party content straight into rendered pages.A monthly PR gets merged and deployed within days at this cadence, so it buys the same freshness without the coupling.
Changes
scripts/update-exodus-trackers.js— fetches, validates and normalises the Exodus API payload, writing only when the content actually changed. Supports--check(validate, never write),--input(local file),--summary(Markdown change summary for the PR body). Exposed asnpm run update-trackers..github/workflows/update-exodus-trackers.yml— runs it monthly and on demand, opening or updating a single PR when something changed. UsesghwithGITHUB_TOKENrather than a third-party action.test/updateExodusTrackers.test.js— covers field projection, sort stability, the truncation and malformed-payload guards, and change detection.README.md— new "Tracker Metadata" section documenting what the file is for and how to refresh it.exodusTrackers.json— the refresh itself.Normalisation
The payload is projected down to the three fields
views/form.pugreads (id,name,categories), with keys and categories sorted. Exodus returns categories in an unstable order: 93 of the 96 entries that differed against the committed copy differed only by category array ordering. Writing the raw response would open a churny PR every month with the real changes invisible inside it. As a side effect the file drops from 570 KB to 50 KB, and pretty-printing makes future diffs readable.Data changes in this refresh
432 trackers, up from 428.
[]→[Identification], AdTiming[]→[Advertisement], MixPanel dropsAdvertisementThis is display data only — detection comes from
analyser/data/ios_signatures_v3.json, so a stale file costs a link and some badges, never a missed tracker. Of the changes above only Sentry matches a name the analyser currently detects. The other 34 signature names without an Exodus entry (Facebook, Google Firebase, WeChat,__ALL_CLASSES__, …) are naming mismatches or non-Exodus concepts that no refresh will fix; an alias map would be the separate change for that.Testing
node --test test/updateExodusTrackers.test.js— 5/5 pass.npm test— 25 pass, 9 fail; the same 9 fail on a clean checkout in this environment (they need a live PostgreSQL).routes/index.jsexpects, including the renamed and newly added entries.--inputon its own output reports "already up to date") and that--checknever writes.🤖 Generated with Claude Code
https://claude.ai/code/session_01TT3FoeNrXRT7A6BETh3yc1
Generated by Claude Code