diff --git a/README.md b/README.md index 8f2a191..4d9610e 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,9 @@ The website also includes jurisdiction analysis, showing which companies and cou - Detect embedded tracker signatures and declared tracking domains. - Store current and historical analysis results. - Show tracker, permission, and jurisdiction summaries. +- Reverse lookup: which apps contain a given tracker or a given company's trackers. +- An about page explaining how apps are analysed, and what a report does and does not tell you. +- Sitemap, `robots.txt`, canonical links, and Open Graph/Twitter card metadata. - Run the analyser from macOS or a Raspberry Pi host. Only free App Store apps are queued for analysis. The queue prioritises apps with more stored App Store reviews, then rechecks stale analyses over time. @@ -34,6 +37,30 @@ public/ Browser assets static/ Static image assets ``` +## Public Pages + +| Path | Purpose | +| --- | --- | +| `/` | Search and headline statistics | +| `/analysis/:appId` | Per-app tracker, permission, and jurisdiction report | +| `/statistics` | Aggregate jurisdiction statistics | +| `/trackers`, `/companies` | Directories of every tracker and company seen in an analysed app | +| `/tracker/:slug`, `/company/:slug` | Reverse lookup: the apps a tracker or company was found in | +| `/about` | How apps are analysed, what a report means, jurisdiction labels, project background and contact | +| `/sitemap.xml`, `/robots.txt` | Crawler metadata | + +The reverse lookup pages are served from an inverted index built by +`lib/reverseIndex.js` and cached under `CACHE_DIR` alongside the aggregate site +data. It is rebuilt whenever the set of stored analyses changes, so no extra +work happens per request. + +`CACHE_DIR` is a persistent volume in production, so cache entries outlive the +code that wrote them. An entry is only rebuilt when the set of stored analyses +changes, which cannot detect a change in how the cached data is *derived*. +After editing `buildSiteData` or `buildReverseIndex`, bump `SCHEMA_VERSION` in +`lib/cache.js` so the deploy discards entries built by the previous logic — +otherwise the old figures are served until the next analysis lands. + ## Requirements Website: @@ -73,6 +100,25 @@ PORT=3000 `PUBLIC_FORM_BODY_LIMIT` is the smaller limit for the public analysis request form. `APP_STORE_CACHE_RETENTION_DAYS` controls how long cached App Store metadata is kept. +Set `SITE_URL` in production to the public origin, for example +`SITE_URL=https://ios.trackercontrol.org`. Canonical links, Open Graph URLs, +`robots.txt`, and `sitemap.xml` use it. Without it, those URLs are derived from +the request in development, which yields `http://` links when TLS is terminated +by a proxy. It is required in production, so an untrusted Host header cannot +become a public canonical URL — the server refuses to start without it rather +than answering 500 on every route. + +Rate limits are applied per IP over a five-minute window, and published pages +are budgeted separately from the App Store entry points because `sitemap.xml` +points crawlers at every app, tracker and company URL. `RATE_LIMIT_BROWSE_MAX` +(default 300) covers `GET`/`HEAD` of the published pages, which are served from +the cached site data. `RATE_LIMIT_FORM_MAX` (default 20) covers everything else, +including `/search` and `/request/:appId` — these are `GET`s so that Cloudflare +can challenge them, but each one reaches the App Store, so they are budgeted as +the form submissions they are rather than as page views. Authenticated analyser +traffic is exempt from both. `robots.txt` disallows both paths as well, so a +crawler neither spends App Store calls nor collects challenge interstitials. + ### Bot protection Everything that costs an App Store call is protected by Cloudflare WAF rules diff --git a/index.js b/index.js index 2f65c2d..26cfaa4 100644 --- a/index.js +++ b/index.js @@ -2,6 +2,7 @@ require('dotenv').config(); const { getOriginSecret } = require('./lib/originGate'); +const siteUrl = require('./lib/siteUrl'); // Load the actual app const app = require('./server'); @@ -12,6 +13,15 @@ if (env == 'production') { app.set('trust proxy', 1); if (!getOriginSecret()) console.warn('CLOUDFLARE_ORIGIN_SECRET is not set: the origin is trusted to be reachable only through Cloudflare.'); + // Every route builds canonical/social URLs from this, so a missing value + // fails the request rather than degrading it. Refuse to boot instead of + // serving 500s from a process the platform considers healthy. + try { + siteUrl.assertSiteUrlConfiguration(); + } catch (err) { + console.error(`Site URL configuration error: ${err.message}`); + throw err; + } } // Server express HTTP server diff --git a/lib/appMetadata.js b/lib/appMetadata.js index 221fdd8..575af09 100644 --- a/lib/appMetadata.js +++ b/lib/appMetadata.js @@ -48,4 +48,34 @@ function buildReportMetadata({ }; } -module.exports = { buildReportMetadata }; +// Fields the listing pages render. The rest of the queue-time snapshot is +// carried through untouched so this stays a refresh rather than a projection. +const LISTING_FIELDS = ['title', 'icon', 'url', 'version', 'primaryGenre', 'reviews']; + +/** + * Display details for the pages that list many apps at once — the homepage, + * the directories, the lookup pages and the sitemap. + * + * apps.details is the snapshot taken when the app was queued and is never + * updated; the App Store cache row is what the metadata jobs refresh. Listing + * pages read the snapshot directly, which would leave them showing a title the + * report page contradicts. Refreshed values therefore win field by field, so a + * new title is not paired with an icon dropped from a partial response. + */ +function buildListingDetails({ queueSnapshot = null, storefront = null } = {}) { + const current = detailsFromStorefront(storefront) || {}; + const merged = { ...(queueSnapshot || {}) }; + + for (const field of LISTING_FIELDS) { + // Tested directly rather than through firstNonEmpty, whose `||` maps a + // numeric 0 to null. An app really can hold zero reviews, and that is a + // refreshed value like any other. + const value = current[field]; + if (value !== null && value !== undefined && value !== '') + merged[field] = value; + } + + return merged; +} + +module.exports = { buildReportMetadata, buildListingDetails }; diff --git a/lib/cache.js b/lib/cache.js index a4175ef..1bbd9e4 100644 --- a/lib/cache.js +++ b/lib/cache.js @@ -14,14 +14,24 @@ function cachePath(key) { return path.join(CACHE_DIR, key + '.json'); } +// CACHE_DIR is a persistent volume in production, so entries outlive the code +// that produced them. Callers detect new *data* through their own meta +// signature, which cannot see a change in how that data is derived: after a +// deploy that alters buildSiteData or buildReverseIndex, the old entry still +// matches the signature and would be served indefinitely. Bump this whenever +// the shape or the derivation of any cached payload changes. +const SCHEMA_VERSION = 2; + /** - * Read from cache. Returns { data, meta } or null if no cache exists. + * Read from cache. Returns { data, meta } or null if no cache exists or the + * entry was written by an incompatible version of the builders. */ function read(key) { const file = cachePath(key); try { const raw = fs.readFileSync(file, 'utf-8'); const cached = JSON.parse(raw); + if (cached._schema !== SCHEMA_VERSION) return null; return { data: cached.data, meta: cached._meta || {} }; } catch (err) { return null; @@ -34,7 +44,7 @@ function read(key) { function write(key, data, meta) { const file = cachePath(key); try { - const payload = JSON.stringify({ data, _meta: meta || {} }); + const payload = JSON.stringify({ data, _meta: meta || {}, _schema: SCHEMA_VERSION }); fs.writeFileSync(file, payload, 'utf-8'); } catch (err) { console.error('Cache write error:', err.message); @@ -50,4 +60,4 @@ function invalidate(key) { } } -module.exports = { read, write, invalidate }; +module.exports = { read, write, invalidate, SCHEMA_VERSION }; diff --git a/lib/jurisdiction.js b/lib/jurisdiction.js index d82a68a..c0fdba4 100644 --- a/lib/jurisdiction.js +++ b/lib/jurisdiction.js @@ -134,6 +134,22 @@ function getCountryName(code) { return countryNames[code.toUpperCase()] || code.toUpperCase(); } +/** + * Whether a signature name is an iOS system API rather than a third-party + * tracker. These are reported by the analyser but excluded from jurisdiction + * analysis. + */ +function isSystemSignature(trackerName) { + if (!trackerName) return false; + return excludedSignatures.has(String(trackerName).toLowerCase().trim()); +} + +// The partial match below scans every known company, and the same handful of +// tracker names recur across thousands of apps, so results are memoised. The +// company database is static after module load, which makes this safe. +const resolutionCache = new Map(); +const MAX_RESOLUTION_CACHE_SIZE = 10000; + /** * Resolve a tracker name to a company. * Tries exact match, then partial/substring match against Xray owner names. @@ -142,6 +158,16 @@ function resolveTrackerName(trackerName) { if (!trackerName) return null; const key = trackerName.toLowerCase().trim(); + if (resolutionCache.has(key)) return resolutionCache.get(key); + + const resolved = resolveTrackerNameUncached(key); + if (resolutionCache.size >= MAX_RESOLUTION_CACHE_SIZE) + resolutionCache.delete(resolutionCache.keys().next().value); + resolutionCache.set(key, resolved); + return resolved; +} + +function resolveTrackerNameUncached(key) { // Skip system APIs that aren't third-party trackers if (excludedSignatures.has(key)) return null; @@ -454,6 +480,7 @@ module.exports = { europeanAlternatives, computeAggregateStats, resolveTrackerName, + isSystemSignature, resolveHost, classifyRegion, countryFlag, diff --git a/lib/reverseIndex.js b/lib/reverseIndex.js new file mode 100644 index 0000000..860b0f5 --- /dev/null +++ b/lib/reverseIndex.js @@ -0,0 +1,385 @@ +// Reverse lookup index: tracker -> apps and company -> apps. +// +// The website's per-app reports answer "what is in this app?". Journalists and +// researchers usually arrive with the opposite question: "which apps contain +// this tracker?". This module builds that inverted view once per analysis +// generation so the lookup pages can be served from a cached structure instead +// of scanning every stored analysis per request. +// +// The index is normalised: app metadata is stored once in `apps`, and tracker +// and company entries reference apps by bundle ID. That keeps the cached JSON +// small enough to read cheaply even when every app appears in several lists. +const jurisdiction = require('./jurisdiction'); +const crypto = require('node:crypto'); + +const MAX_SLUG_LENGTH = 80; +// Allows persisted legacy slugs as well as deterministic hash suffixes. +const MAX_SLUG_LENGTH_WITH_SUFFIX = MAX_SLUG_LENGTH + 8; +const SLUG_HASH_LENGTH = 8; +const SLUG_PATTERN = /^[a-z0-9][a-z0-9-]*$/; + +/** + * Build a URL-safe slug for a tracker or company name. + * Names contain spaces, dots and other punctuation ("Mob.com", "Unity3d Ads"), + * so the slug is lossy; collisions are resolved when slugs are assigned. + */ +function slugify(name) { + const slug = String(name == null ? '' : name) + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, MAX_SLUG_LENGTH) + .replace(/-+$/, ''); + return slug || 'unnamed'; +} + +/** + * Whether a value can be a slug at all. Route handlers check this before + * touching the index so arbitrary path segments never reach a property lookup. + */ +function isValidSlug(value) { + return typeof value === 'string' + && value.length > 0 + && value.length <= MAX_SLUG_LENGTH_WITH_SUFFIX + && SLUG_PATTERN.test(value); +} + +function reviewCount(details) { + const raw = details && details.reviews; + const parsed = typeof raw === 'number' ? raw : parseInt(raw, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : 0; +} + +function percentage(count, total) { + return total > 0 ? (count / total * 100).toFixed(1) : '0.0'; +} + +function createTrackerEntry(displayName) { + const resolved = jurisdiction.resolveTrackerName(displayName); + const company = resolved ? jurisdiction.getUltimateParent(resolved) : null; + const country = resolved ? jurisdiction.getUltimateCountry(resolved) : null; + + return { + name: displayName, + // System APIs are reported by the analyser but are not third-party + // trackers, and jurisdiction analysis excludes them. Flagging them keeps + // the directory honest rather than listing them as unattributed trackers. + system: jurisdiction.isSystemSignature(displayName), + company: company || null, + country: company ? country : null, + countryName: company ? jurisdiction.getCountryName(country) : null, + flag: company ? jurisdiction.countryFlag(country) : '', + region: company ? jurisdiction.classifyRegion(country) : 'Unresolved', + appIds: [] + }; +} + +function compareNames(a, b) { + const left = a.toLowerCase(); + const right = b.toLowerCase(); + if (left < right) return -1; + if (left > right) return 1; + if (a < b) return -1; + if (a > b) return 1; + return 0; +} + +/** + * Add a deterministic discriminator for a new name that collides with an + * existing slug. The name is canonicalised because tracker and company maps + * are case-insensitive, so changing only the display-name casing cannot move + * a URL. + */ +function collisionSlug(name, base, hashLength = SLUG_HASH_LENGTH) { + const canonical = String(name == null ? '' : name).toLowerCase().trim(); + const hash = crypto.createHash('sha256').update(canonical).digest('hex'); + const suffix = hash.slice(0, hashLength); + const prefixLength = MAX_SLUG_LENGTH - suffix.length - 1; + const prefix = base.slice(0, prefixLength).replace(/-+$/, '') || 'unnamed'; + return `${prefix}-${suffix}`; +} + +/** + * Assign a unique slug to every entry. Previously assigned slugs win, so a + * newly discovered name cannot take over a URL that was already published. + * New collisions use a name-derived hash instead of a positional counter. + */ +function assignSlugs(entries, previousSlugs = {}) { + const used = new Set(); + const pending = []; + + for (const entry of [...entries].sort((a, b) => compareNames(a.name, b.name))) { + const key = String(entry.name).toLowerCase().trim(); + const previous = previousSlugs && previousSlugs[key]; + if (isValidSlug(previous) && !used.has(previous)) { + used.add(previous); + entry.slug = previous; + } else { + pending.push(entry); + } + } + + for (const entry of pending) { + const base = slugify(entry.name); + let slug = base; + if (used.has(slug)) { + let hashLength = SLUG_HASH_LENGTH; + do { + slug = collisionSlug(entry.name, base, hashLength); + hashLength += 8; + } while (used.has(slug) && hashLength <= 64); + if (used.has(slug)) { + throw new Error(`Unable to assign a unique slug for ${entry.name}`); + } + } + used.add(slug); + entry.slug = slug; + } +} + +/** + * Build the reverse index from the rows returned by Apps.getAllApps(). + * + * `totalApps` counts every successfully analysed app, including apps where no + * tracker was detected, so it matches the denominator used by the aggregate + * jurisdiction statistics and can be quoted as "N of M apps". + */ +function buildReverseIndex(allApps, previousIndex = null) { + const apps = {}; + const trackerEntries = new Map(); // lowercased tracker name -> entry + let totalApps = 0; + let trackedApps = 0; + let latestAnalysis = null; + + for (const app of allApps || []) { + const analysis = app && app.analysis; + if (!app.appid || !analysis || analysis.success === false) continue; + + totalApps++; + + const analysedAt = app.analysed ? new Date(app.analysed) : null; + const analysedValid = analysedAt && !Number.isNaN(analysedAt.getTime()); + if (analysedValid && (!latestAnalysis || analysedAt > latestAnalysis)) + latestAnalysis = analysedAt; + + const trackerNames = analysis.trackers ? Object.keys(analysis.trackers) : []; + const thirdPartyTrackerNames = trackerNames.filter( + (trackerName) => !jurisdiction.isSystemSignature(trackerName) + ); + if (thirdPartyTrackerNames.length > 0) trackedApps++; + + const details = app.details || {}; + const analysed = jurisdiction.analyseApp(analysis); + + // Every analysed app enters the directory, including apps with no detected + // tracker: the lookup pages only reference the tracked ones, but the + // sitemap covers all of them. + apps[app.appid] = { + appid: app.appid, + title: details.title || app.appid, + icon: details.icon || null, + category: details.primaryGenre || null, + reviews: reviewCount(details), + trackerCount: thirdPartyTrackerNames.length, + classification: analysed.classification, + analysed: analysedValid ? analysedAt.toISOString() : null + }; + + for (const trackerName of trackerNames) { + const key = String(trackerName).toLowerCase().trim(); + if (!key) continue; + + let entry = trackerEntries.get(key); + if (!entry) { + entry = createTrackerEntry(trackerName); + trackerEntries.set(key, entry); + } + + // Two tracker names in the same app can normalise to one key; count the + // app once. + if (entry.appIds[entry.appIds.length - 1] !== app.appid) + entry.appIds.push(app.appid); + } + } + + // Trackers resolve to a company deterministically, so companies are derived + // from the tracker entries rather than recomputed from each app. + const companyEntries = new Map(); + for (const entry of trackerEntries.values()) { + if (!entry.company) continue; + + const key = entry.company.toLowerCase(); + let company = companyEntries.get(key); + if (!company) { + company = { + name: entry.company, + country: entry.country, + countryName: entry.countryName, + flag: entry.flag, + region: entry.region, + trackers: [], + appIdSet: new Set() + }; + companyEntries.set(key, company); + } + company.trackers.push(entry); + for (const appid of entry.appIds) company.appIdSet.add(appid); + } + + const trackerArray = [...trackerEntries.values()]; + const companyArray = [...companyEntries.values()]; + assignSlugs(trackerArray, previousIndex && previousIndex.trackerSlugs); + assignSlugs(companyArray, previousIndex && previousIndex.companySlugs); + + const companySlugByName = new Map(); + for (const company of companyArray) + companySlugByName.set(company.name.toLowerCase(), company.slug); + + // Most popular app first: the apps a reader recognises are the ones worth + // showing on page one. + const byPopularity = (a, b) => + apps[b].reviews - apps[a].reviews + || compareNames(apps[a].title, apps[b].title) + || compareNames(a, b); + + const trackers = {}; + for (const entry of trackerArray) { + entry.appIds.sort(byPopularity); + trackers[entry.slug] = { + slug: entry.slug, + name: entry.name, + system: entry.system, + company: entry.company, + companySlug: entry.company + ? companySlugByName.get(entry.company.toLowerCase()) || null + : null, + country: entry.country, + countryName: entry.countryName, + flag: entry.flag, + region: entry.region, + appCount: entry.appIds.length, + pct: percentage(entry.appIds.length, totalApps), + appIds: entry.appIds + }; + } + + const companies = {}; + for (const company of companyArray) { + const appIds = [...company.appIdSet].sort(byPopularity); + companies[company.slug] = { + slug: company.slug, + name: company.name, + country: company.country, + countryName: company.countryName, + flag: company.flag, + region: company.region, + appCount: appIds.length, + pct: percentage(appIds.length, totalApps), + appIds, + trackers: company.trackers + .map((entry) => ({ + name: entry.name, + slug: entry.slug, + appCount: entry.appIds.length + })) + .sort((a, b) => b.appCount - a.appCount || compareNames(a.name, b.name)) + }; + } + + const byPrevalence = (source) => (a, b) => + source[b].appCount - source[a].appCount || compareNames(source[a].name, source[b].name); + + // Name -> slug maps let other pages (app reports, statistics tables) link + // into the lookup pages without re-deriving a slug that may have been + // deduplicated here. + const trackerSlugs = {}; + for (const entry of trackerArray) trackerSlugs[entry.name.toLowerCase().trim()] = entry.slug; + const companySlugs = {}; + for (const entry of companyArray) companySlugs[entry.name.toLowerCase().trim()] = entry.slug; + + return { + trackerSlugs, + companySlugs, + totalApps, + trackedApps, + latestAnalysis: latestAnalysis ? latestAnalysis.toISOString() : null, + apps, + trackers, + trackerList: Object.keys(trackers).sort(byPrevalence(trackers)), + companies, + companyList: Object.keys(companies).sort(byPrevalence(companies)) + }; +} + +/** + * Look up an entry by slug. Uses an own-property check so that slugs like + * "constructor" cannot reach inherited properties. + */ +function lookup(collection, slug) { + if (!collection || !isValidSlug(slug)) return null; + return Object.prototype.hasOwnProperty.call(collection, slug) + ? collection[slug] + : null; +} + +function lookupTracker(index, slug) { + return index ? lookup(index.trackers, slug) : null; +} + +function lookupCompany(index, slug) { + return index ? lookup(index.companies, slug) : null; +} + +/** + * Slug for a tracker or company name, or null when the name does not appear in + * any analysed app. + */ +function slugForName(slugMap, name) { + if (!slugMap || !name) return null; + const key = String(name).toLowerCase().trim(); + return Object.prototype.hasOwnProperty.call(slugMap, key) ? slugMap[key] : null; +} + +/** + * Resolve a page of app IDs into app records, dropping any ID that is missing + * from the directory. + */ +function paginate(appIds, appDirectory, page, perPage) { + const items = (appIds || []).filter((appid) => + appDirectory + && Object.prototype.hasOwnProperty.call(appDirectory, appid) + && appDirectory[appid] + ); + const totalPages = Math.max(1, Math.ceil(items.length / perPage)); + const currentPage = Math.min(Math.max(1, page), totalPages); + const start = (currentPage - 1) * perPage; + + return { + page: currentPage, + totalPages, + perPage, + total: items.length, + from: items.length === 0 ? 0 : start + 1, + to: Math.min(start + perPage, items.length), + apps: items + .slice(start, start + perPage) + .map((appid) => appDirectory[appid]) + .filter(Boolean) + }; +} + +function parsePage(value) { + const parsed = parseInt(value, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : 1; +} + +module.exports = { + slugify, + isValidSlug, + buildReverseIndex, + lookupTracker, + lookupCompany, + slugForName, + paginate, + parsePage +}; diff --git a/lib/siteUrl.js b/lib/siteUrl.js new file mode 100644 index 0000000..faa040e --- /dev/null +++ b/lib/siteUrl.js @@ -0,0 +1,42 @@ +'use strict'; + +// Absolute base URL of this deployment, used for canonical links, social card +// metadata, robots.txt and the sitemap. +// +// SITE_URL pins the origin when the site runs behind a proxy that terminates +// TLS, where req.protocol would otherwise report http, and so that an +// untrusted Host header cannot become a public canonical URL. It is therefore +// required in production. index.js asserts it at startup: without that, the +// first request would throw from routing middleware and every route — the +// health check included — would answer 500 on a deployment that looked +// healthy at boot. + +function configuredSiteUrl() { + return (process.env.SITE_URL || '').trim().replace(/\/+$/, ''); +} + +function getSiteUrlConfigurationError() { + if (process.env.NODE_ENV === 'production' && !configuredSiteUrl()) + return 'SITE_URL is not set'; + + return null; +} + +function assertSiteUrlConfiguration() { + const error = getSiteUrlConfigurationError(); + if (error) throw new Error(error); +} + +function siteBaseUrl(req) { + const configured = configuredSiteUrl(); + if (configured) return configured; + + assertSiteUrlConfiguration(); + return `${req.protocol}://${req.get('host')}`; +} + +module.exports = { + siteBaseUrl, + getSiteUrlConfigurationError, + assertSiteUrlConfiguration +}; diff --git a/models/Apps.js b/models/Apps.js index c087b39..7501ec0 100644 --- a/models/Apps.js +++ b/models/Apps.js @@ -447,11 +447,27 @@ const updateAnalysis = async (appId, analysis, analysisVersion, claimToken) => { } } +// apps.details is the queue-time snapshot and is never updated, so the latest +// known storefront row travels with each app for callers that display a title +// or icon. See buildListingDetails in lib/appMetadata.js. const getAllApps = async () => { - const result = await pool.query("SELECT * FROM apps WHERE status = 'analysed'"); + const result = await pool.query(` + SELECT + apps.*, + cache.details AS current_storefront_details + FROM apps + LEFT JOIN app_store_cache cache + ON cache.appid_key = lower(apps.appid) + WHERE apps.status = 'analysed' + `); return result.rows; } +// Identifies the generation of data the cached site views were built from. +// latestStorefront covers the metadata refresh jobs: they change titles, +// icons and versions without adding an app or writing an analysis, so the +// first two components alone would leave a persisted cache serving metadata +// that the report pages have already moved past. const getSiteDataSignature = async () => { const result = await pool.query(` SELECT @@ -462,14 +478,16 @@ const getSiteDataSignature = async () => { MAX(analysed) FILTER ( WHERE status = 'analysed' AND analysis->'trackers' IS NOT NULL - ) AS latest_analysis + ) AS latest_analysis, + (SELECT MAX(fetched_at) FROM app_store_cache) AS latest_storefront FROM apps `); const row = result.rows[0]; return { appCount: parseInt(row.app_count, 10), - latestAnalysis: row.latest_analysis ? new Date(row.latest_analysis).toISOString() : null + latestAnalysis: row.latest_analysis ? new Date(row.latest_analysis).toISOString() : null, + latestStorefront: row.latest_storefront ? new Date(row.latest_storefront).toISOString() : null }; } diff --git a/public/css/styles.css b/public/css/styles.css index f44b507..13ec7c1 100644 --- a/public/css/styles.css +++ b/public/css/styles.css @@ -164,3 +164,80 @@ body { .listing-table th:first-child { border-right: 1px solid #666; } + +/* About page: how-an-app-is-analysed pipeline */ +.pipeline-step { + background-color: #fff; + border-radius: 8px; + box-shadow: 0 1px 3px rgba(0,0,0,0.08); + padding: 16px 12px; + height: 100%; + text-align: center; + position: relative; +} +.pipeline-icon { + color: #007bff; + margin-bottom: 8px; +} +.pipeline-step h4 { + font-size: 1rem; + font-weight: 600; + margin-bottom: 6px; +} +.pipeline-step p { + font-size: 0.85rem; + color: #6c757d; + margin-bottom: 0; +} +/* Chevrons between the steps, on the widths where they sit in one row. */ +@media (min-width: 768px) { + .pipeline [class^='col-']:not(:last-child) .pipeline-step::after { + content: ''; + position: absolute; + top: 50%; + right: -13px; + margin-top: -7px; + border-top: 7px solid transparent; + border-bottom: 7px solid transparent; + border-left: 8px solid #adb5bd; + } +} + +/* About page: what a report does and does not tell you */ +.result-card { + background-color: #fff; + border-radius: 8px; + border-left: 4px solid #6c757d; + box-shadow: 0 1px 3px rgba(0,0,0,0.08); + padding: 16px; + height: 100%; +} +.result-card h4 { + font-size: 1rem; + font-weight: 600; + margin-bottom: 10px; +} +.result-card ul { + padding-left: 20px; + font-size: 0.9rem; +} +.result-card li + li { + margin-top: 6px; +} +.result-card-yes { + border-left-color: #28a745; +} +.result-card-yes h4 { + color: #1e7e34; +} +.result-card-no { + border-left-color: #dc3545; +} +.result-card-no h4 { + color: #b02a37; +} + +/* Keep in-page anchors clear of the sticky navbar. */ +:target { + scroll-margin-top: 90px; +} diff --git a/public/js/filter.js b/public/js/filter.js new file mode 100644 index 0000000..bbd0ec5 --- /dev/null +++ b/public/js/filter.js @@ -0,0 +1,32 @@ +// Client-side filter for the tracker and company directories. The tables are +// rendered in full, so filtering is a local narrowing of what is already on the +// page and needs no requests. +(function () { + 'use strict'; + + var input = document.getElementById('directory-filter'); + if (!input) return; + + var table = document.querySelector(input.getAttribute('data-filter-target')); + if (!table) return; + + var rows = Array.prototype.slice.call(table.querySelectorAll('tbody tr')); + var empty = document.getElementById('directory-empty'); + + function apply() { + var term = input.value.trim().toLowerCase(); + var visible = 0; + + rows.forEach(function (row) { + var haystack = row.getAttribute('data-filter') || ''; + var matches = term === '' || haystack.indexOf(term) !== -1; + row.style.display = matches ? '' : 'none'; + if (matches) visible++; + }); + + if (empty) empty.classList.toggle('d-none', visible > 0); + } + + input.addEventListener('input', apply); + apply(); +})(); diff --git a/routes/index.js b/routes/index.js index 08cb4ae..334bcef 100644 --- a/routes/index.js +++ b/routes/index.js @@ -5,10 +5,12 @@ const store = require('../lib/appStore'); const Apps = require('../models/Apps'); const jurisdiction = require('../lib/jurisdiction'); const cache = require('../lib/cache'); +const reverseIndex = require('../lib/reverseIndex'); const { isValidAppId } = require('../lib/appId'); const { classifyAnalysisFailure } = require('../lib/analysisFailure'); const asyncHandler = require('../lib/asyncHandler'); -const { buildReportMetadata } = require('../lib/appMetadata'); +const { buildReportMetadata, buildListingDetails } = require('../lib/appMetadata'); +const { siteBaseUrl } = require('../lib/siteUrl'); // Taken from https://reports.exodus-privacy.eu.org/api/trackers const exodusTrackers = JSON.parse(fs.readFileSync('./exodusTrackers.json', 'utf-8')) @@ -19,6 +21,13 @@ for (const [key, value] of Object.entries(exodusTrackers.trackers)) const router = express.Router(); const COUNTRY = 'gb'; +const SITE_NAME = 'TrackerControl for iOS'; +const DEFAULT_DESCRIPTION = 'Find out which trackers are embedded in iOS apps, ' + + 'which companies control them, and under which jurisdiction that tracking falls.'; +const APPS_PER_PAGE = 50; +const MAX_SITEMAP_URLS = 50000; +const MAX_SITEMAP_BYTES = 50 * 1024 * 1024; + let lastPing = 0; // unix timestamp function requireValidAppId(req, res, next) { @@ -43,6 +52,28 @@ router.use(function (req, res, next) { next(); }); +function thirdPartyTrackerNames(analysis) { + return analysis && analysis.trackers + ? Object.keys(analysis.trackers).filter( + (trackerName) => !jurisdiction.isSystemSignature(trackerName) + ) + : []; +} + +// Social card and canonical link defaults. Individual routes override these +// with page-specific values by passing them to res.render. +router.use(function (req, res, next) { + const base = siteBaseUrl(req); + const path = req.path.length > 1 ? req.path.replace(/\/+$/, '') : req.path; + + res.locals.siteName = SITE_NAME; + res.locals.siteBaseUrl = base; + res.locals.canonicalUrl = base + path; + res.locals.pageDescription = DEFAULT_DESCRIPTION; + res.locals.ogImage = null; + next(); +}); + /** * Build all homepage + statistics data from DB. * Returns { homepage, statistics, appCount }. @@ -60,7 +91,7 @@ function buildSiteData(allApps) { // Top trackers enriched with company/country const trackerCounts = {}; for (const app of analysedApps) { - for (const tracker of Object.keys(app.analysis.trackers)) { + for (const tracker of thirdPartyTrackerNames(app.analysis)) { if (!trackerCounts[tracker]) trackerCounts[tracker] = 0; trackerCounts[tracker]++; } @@ -89,7 +120,7 @@ function buildSiteData(allApps) { const appsWithMostTrackers = analysedApps .filter(a => a.details && a.details.title) .map(a => { - const trackerCount = Object.keys(a.analysis.trackers).length; + const trackerCount = thirdPartyTrackerNames(a.analysis).length; const jd = jurisdiction.analyseApp(a.analysis); const topCountries = Object.entries(jd.countryBreakdown || {}) .sort((a, b) => b[1] - a[1]) @@ -138,6 +169,35 @@ function buildSiteData(allApps) { }; } +/** + * Whether cached data was built from the same generation of stored analyses + * and App Store metadata as the database currently holds. + */ +function signatureMatches(meta, signature) { + return Boolean(meta) + && meta.appCount === signature.appCount + && meta.latestAnalysis === signature.latestAnalysis + && meta.latestStorefront === signature.latestStorefront; +} + +// The signature is an aggregate over every stored app, and each cached view +// asks for it before serving: the report page needs the reverse index for its +// tracker links, and /statistics reads both cached views. Without this it +// would run twice per statistics request and once per report view. A few +// seconds of staleness only delays a rebuild that a background metadata job +// triggered; writes from this process clear the memo outright. +const SIGNATURE_TTL_MS = 5000; +let signatureMemo = null; // { at, signature } + +async function getSiteDataSignature() { + if (signatureMemo && Date.now() - signatureMemo.at < SIGNATURE_TTL_MS) + return signatureMemo.signature; + + const signature = await Apps.getSiteDataSignature(); + signatureMemo = { at: Date.now(), signature }; + return signature; +} + /** * Get site data: serve from cache if app count hasn't changed, otherwise rebuild. * Falls back to stale cache on any DB error. @@ -146,15 +206,12 @@ async function getSiteData() { const cached = cache.read('sitedata'); try { - const signature = await Apps.getSiteDataSignature(); - if (cached - && cached.meta - && cached.meta.appCount === signature.appCount - && cached.meta.latestAnalysis === signature.latestAnalysis) { + const signature = await getSiteDataSignature(); + if (cached && signatureMatches(cached.meta, signature)) { return cached.data; } - const allApps = await Apps.getAllApps(); + const allApps = await getAllAppsForSignature(signature); const data = buildSiteData(allApps); if (data.appCount > 0) { cache.write('sitedata', data, signature); @@ -168,12 +225,94 @@ async function getSiteData() { } } +// The reverse index is large compared with the aggregate site data, so it is +// kept in its own cache entry and only touched by the lookup pages and the +// sitemap. The in-process copy avoids re-parsing the cache file per request. +let reverseIndexMemo = null; // { meta, index } +let allAppsMemo = null; // { meta, apps } + +async function getAllAppsForSignature(signature) { + if (allAppsMemo && signatureMatches(allAppsMemo.meta, signature)) + return allAppsMemo.apps; + + // Resolve the display metadata once here so that buildSiteData and + // buildReverseIndex both see the refreshed title and icon under `details`, + // rather than each reaching into the storefront columns themselves. + const apps = (await Apps.getAllApps()).map((row) => ({ + ...row, + details: buildListingDetails({ + queueSnapshot: row.details, + storefront: { details: row.current_storefront_details } + }) + })); + allAppsMemo = { meta: signature, apps }; + return apps; +} + +/** + * Get the tracker/company reverse index, rebuilding it when new analyses have + * landed. Falls back to the last known index if the database is unavailable. + */ +async function getReverseIndex() { + try { + const signature = await getSiteDataSignature(); + + if (reverseIndexMemo && signatureMatches(reverseIndexMemo.meta, signature)) + return reverseIndexMemo.index; + + const cached = cache.read('reverseindex'); + if (cached && signatureMatches(cached.meta, signature)) { + reverseIndexMemo = { meta: cached.meta, index: cached.data }; + return cached.data; + } + + const allApps = await getAllAppsForSignature(signature); + const index = reverseIndex.buildReverseIndex(allApps, cached && cached.data); + if (index.totalApps > 0) { + cache.write('reverseindex', index, signature); + reverseIndexMemo = { meta: signature, index }; + console.log('Reverse index rebuilt for', index.totalApps, 'apps'); + } + return index; + } catch (err) { + console.error('DB error in getReverseIndex:', err.message); + if (reverseIndexMemo) return reverseIndexMemo.index; + const cached = cache.read('reverseindex'); + if (cached) return cached.data; + throw err; + } +} + +function invalidateSiteCaches() { + cache.invalidate('sitedata'); + cache.invalidate('reverseindex'); + reverseIndexMemo = null; + allAppsMemo = null; + signatureMemo = null; +} + +const EMPTY_REVERSE_INDEX = { + trackerSlugs: {}, + companySlugs: {}, + totalApps: 0, + trackedApps: 0, + latestAnalysis: null, + apps: {}, + trackers: {}, + trackerList: [], + companies: {}, + companyList: [] +}; + router.get('/', asyncHandler(async (req, res) => { try { const data = await getSiteData(); return res.render('form', { title: 'App Privacy Checker', data: req.body, + pageDescription: `Search ${data.headlines.totalApps} analysed iOS apps to see ` + + 'which trackers they embed, which companies control them, and under ' + + 'which jurisdiction that tracking falls.', headlines: data.headlines, appsWithMostTrackers: data.appsWithMostTrackers, jurisdictionStats: data.jurisdictionStats, @@ -192,20 +331,35 @@ router.get('/', asyncHandler(async (req, res) => { } })); +/** + * Attach reverse-lookup slugs to the statistics tables so every tracker and + * company in them links to the apps it was found in. + */ +function withLookupSlugs(data, index) { + const trackerSlugs = index.trackerSlugs || {}; + const companySlugs = index.companySlugs || {}; + const trackers = (data.topTrackersEnriched || []).map((tracker) => ({ + ...tracker, + slug: reverseIndex.slugForName(trackerSlugs, tracker.name), + companySlug: reverseIndex.slugForName(companySlugs, tracker.company) + })); + const companies = (data.jurisdictionStats && data.jurisdictionStats.topCompaniesSorted || []) + .map((company) => ({ + ...company, + slug: reverseIndex.slugForName(companySlugs, company.name) + })); + + return { + topTrackersEnriched: trackers, + jurisdictionStats: { ...data.jurisdictionStats, topCompaniesSorted: companies } + }; +} + // Statistics detail page router.get('/statistics', asyncHandler(async (req, res) => { + let data; try { - const data = await getSiteData(); - return res.render('statistics', { - title: 'Detailed Statistics', - data: req.body, - headlines: data.headlines, - jurisdictionStats: data.jurisdictionStats, - jurisdictionMeta: jurisdiction.classificationMeta, - topTrackersEnriched: data.topTrackersEnriched, - europeanAlternatives: jurisdiction.europeanAlternatives, - xrayCompanyCount: jurisdiction.xrayCompanyCount - }); + data = await getSiteData(); } catch (err) { console.error('Statistics error:', err.message); return res.render('statistics', { @@ -219,6 +373,28 @@ router.get('/statistics', asyncHandler(async (req, res) => { xrayCompanyCount: jurisdiction.xrayCompanyCount }); } + + let index = EMPTY_REVERSE_INDEX; + try { + index = await getReverseIndex(); + } catch (err) { + console.error('Statistics lookup data unavailable:', err.message); + } + const linked = withLookupSlugs(data, index); + + return res.render('statistics', { + title: 'Detailed Statistics', + data: req.body, + pageDescription: `Tracking jurisdiction across ${data.headlines.totalApps} ` + + 'analysed iOS apps: the most prevalent trackers, the companies behind ' + + 'them, and how they break down by country and App Store category.', + headlines: data.headlines, + jurisdictionStats: linked.jurisdictionStats, + jurisdictionMeta: jurisdiction.classificationMeta, + topTrackersEnriched: linked.topTrackersEnriched, + europeanAlternatives: jurisdiction.europeanAlternatives, + xrayCompanyCount: jurisdiction.xrayCompanyCount + }); })); router.get('/healthz', asyncHandler(async (req, res) => { @@ -325,8 +501,12 @@ router.get('/analysis/:appId', requireValidAppId, asyncHandler(async (req, res) if (analysis.success !== undefined && analysis.success === false) app.analysisFailure = analysis.reason === 'app_not_found' ? "App not found on App Store." : "Analysis failed." else { - if (analysis.trackers) - app.trackers = "Found trackers: " + Object.keys(analysis.trackers).join(", "); + if (analysis.trackers) { + const trackerNames = thirdPartyTrackerNames(analysis); + app.trackers = trackerNames.length > 0 + ? "Found trackers: " + trackerNames.join(", ") + : "No trackers found."; + } else app.trackers = "No trackers found." @@ -346,12 +526,47 @@ router.get('/analysis/:appId', requireValidAppId, asyncHandler(async (req, res) jurisdictionData.sovereigntyNote = jurisdiction.sovereigntyNotes[jurisdictionData.classification]; } + // Slugs let each detected tracker and company link through to the other apps + // that share it. + let trackerSlugs = {}; + let companySlugs = {}; + try { + const index = await getReverseIndex(); + trackerSlugs = index.trackerSlugs || {}; + companySlugs = index.companySlugs || {}; + } catch (err) { + console.error('Tracker links unavailable:', err.message); + } + + const trackerCount = app.analysis && app.analysis.trackers && app.analysis.success !== false + ? thirdPartyTrackerNames(app.analysis).length + : null; + const systemTrackerNames = app.analysis && app.analysis.trackers && app.analysis.success !== false + ? Object.keys(app.analysis.trackers).filter((trackerName) => + jurisdiction.isSystemSignature(trackerName) + ) + : []; + // Social metadata follows the same title/icon precedence as the report body, + // so a refreshed storefront title is not contradicted by the card. + const displayTitle = app.reportMetadata.title || app.details.title; + const pageDescription = trackerCount === null + ? `Tracker analysis of ${displayTitle} for iOS.` + : `${trackerCount === 0 ? 'No trackers were' : `${trackerCount} tracker${trackerCount === 1 ? ' was' : 's were'}`}` + + ` detected in ${displayTitle} for iOS` + + (jurisdictionData && jurisdictionData.meta ? `: ${jurisdictionData.meta.label.toLowerCase()}.` : '.'); + res.render('form', { - title: app.reportMetadata.title || app.details.title, + title: displayTitle, data: req.body, app: app, trackerNameToExodus: trackerNameToExodus, - jurisdictionData: jurisdictionData + trackerSlugs: trackerSlugs, + companySlugs: companySlugs, + jurisdictionData: jurisdictionData, + trackerCount, + systemTrackerNames, + pageDescription, + ogImage: app.reportMetadata.icon || app.details.icon || null }); })); @@ -420,13 +635,112 @@ router.post('/analysis/:appId', return res.redirect(303, `/analysis/${details.appId}`); })); -// About page +// About page: what this service does, what a report does and does not mean, +// where the country labels come from, and who is behind it. router.get('/about', (req, res) => { res.render('about', { - title: 'About' + title: 'About', + pageDescription: 'How this service analyses iOS apps for embedded trackers, ' + + 'what a report does and does not tell you, who runs it, and how to get ' + + 'in touch.', + jurisdictionMeta: jurisdiction.classificationMeta }); }); +/** + * Render a directory of every tracker or company seen in an analysed app. + */ +function renderDirectory(kind) { + return asyncHandler(async (req, res) => { + let index; + try { + index = await getReverseIndex(); + } catch (err) { + console.error('Directory error:', err.message); + index = EMPTY_REVERSE_INDEX; + } + + const isTracker = kind === 'tracker'; + const entries = isTracker + ? index.trackerList.map((slug) => index.trackers[slug]) + : index.companyList.map((slug) => index.companies[slug]); + + res.render('directory', { + title: isTracker ? 'Tracker directory' : 'Company directory', + kind, + entries, + totalApps: index.totalApps, + trackedApps: index.trackedApps, + latestAnalysis: index.latestAnalysis, + pageDescription: isTracker + ? `Every tracker detected across ${index.totalApps} analysed iOS apps, ` + + 'with the company and country behind it.' + : `Every company whose tracking code was detected across ${index.totalApps} ` + + 'analysed iOS apps, ranked by how many apps they reach.' + }); + }); +} + +router.get('/trackers', renderDirectory('tracker')); +router.get('/companies', renderDirectory('company')); + +/** + * Reverse lookup: the apps in which a given tracker, or any tracker belonging + * to a given company, was detected. + */ +function renderLookup(kind) { + return asyncHandler(async (req, res) => { + const isTracker = kind === 'tracker'; + let index; + try { + index = await getReverseIndex(); + } catch (err) { + console.error('Lookup error:', err.message); + return res.status(503).send('Lookup data is temporarily unavailable. Please try again later.'); + } + + const entry = isTracker + ? reverseIndex.lookupTracker(index, req.params.slug) + : reverseIndex.lookupCompany(index, req.params.slug); + + if (!entry) { + return res.status(404).send(isTracker + ? 'Unknown tracker. See /trackers for the full list.' + : 'Unknown company. See /companies for the full list.'); + } + + const pagination = reverseIndex.paginate( + entry.appIds, + index.apps, + reverseIndex.parsePage(req.query.page), + APPS_PER_PAGE + ); + + const attribution = entry.company || (isTracker ? null : entry.name); + const description = `${entry.name} was detected in ${entry.appCount} of ` + + `${index.totalApps} analysed iOS apps (${entry.pct}%)` + + (attribution && entry.countryName ? `. Operated by ${attribution} (${entry.countryName}).` : '.'); + + res.render('lookup', { + title: entry.name, + kind, + entry, + pagination, + totalApps: index.totalApps, + latestAnalysis: index.latestAnalysis, + jurisdictionMeta: jurisdiction.classificationMeta, + exodus: isTracker ? trackerNameToExodus[entry.name] : null, + companySlug: isTracker ? entry.companySlug : null, + pageDescription: description, + canonicalUrl: res.locals.canonicalUrl + + (pagination.page > 1 ? `?page=${pagination.page}` : '') + }); + }); +} + +router.get('/tracker/:slug', renderLookup('tracker')); +router.get('/company/:slug', renderLookup('company')); + // serve next task to analyser router.get('/queue', asyncHandler(async (req, res) => { const requestedAppId = req.query.appId || null; @@ -476,7 +790,7 @@ router.post('/uploadAnalysis', asyncHandler(async (req, res) => { if (result.rowCount === 0) return res.status(409).send('Analysis claim is no longer active.'); - cache.invalidate('sitedata'); + invalidateSiteCaches(); res.json({ ok: true }); })); @@ -505,36 +819,148 @@ router.post('/reportAnalysisFailure', asyncHandler(async (req, res) => { if (result.rowCount === 0) return res.status(409).send('Analysis claim is no longer active.'); - cache.invalidate('sitedata'); + invalidateSiteCaches(); res.json({ ok: true }); })); -/*router.get('/sitemap.xml', async (req, res) => { - try { - const apps = await Apps.getAllApps(); - - let sitemap = ` -`; - - for (const app of apps) { - sitemap += ` - - ${req.protocol}://${req.get('host')}/analysis/${app.appid} - ${new Date().toISOString()} - daily - 0.8 - `; - } +function escapeXml(value) { + return String(value) + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); +} - sitemap += ` +function sitemapEntry(base, path, { lastmod, changefreq, priority } = {}) { + const parts = [` ${escapeXml(base + path)}`]; + if (lastmod) parts.push(` ${escapeXml(lastmod)}`); + if (changefreq) parts.push(` ${changefreq}`); + if (priority) parts.push(` ${priority}`); + return ` \n${parts.join('\n')}\n `; +} + +function renderSitemap(entries) { + return ` + +${entries.join('\n')} `; +} - res.header('Content-Type', 'application/xml'); - res.send(sitemap); - } catch (err) { - console.error('Error generating sitemap:', err); - res.status(500).send('Error generating sitemap'); +function renderSitemapIndex(base, pageCount) { + const entries = Array.from({ length: pageCount }, (_, index) => + ` \n ${escapeXml(base + `/sitemap-${index + 1}.xml`)}\n ` + ); + return ` + +${entries.join('\n')} +`; +} + +function splitSitemapEntries(entries) { + const pages = []; + let page = []; + const emptyPageBytes = Buffer.byteLength(renderSitemap([]), 'utf8'); + let pageBytes = emptyPageBytes; + + for (const entry of entries) { + const entryBytes = Buffer.byteLength(entry, 'utf8'); + const candidateBytes = pageBytes + (page.length > 0 ? 1 : 0) + entryBytes; + const tooManyUrls = page.length + 1 > MAX_SITEMAP_URLS; + const tooLarge = candidateBytes > MAX_SITEMAP_BYTES; + if (page.length > 0 && (tooManyUrls || tooLarge)) { + pages.push(page); + page = [entry]; + pageBytes = emptyPageBytes + entryBytes; + } else { + page.push(entry); + pageBytes = candidateBytes; } -});*/ + } + + if (page.length > 0) pages.push(page); + return pages; +} + +function sitemapEntries(base, index) { + const updated = index.latestAnalysis; + const entries = [ + sitemapEntry(base, '/', { lastmod: updated, changefreq: 'daily', priority: '1.0' }), + sitemapEntry(base, '/statistics', { lastmod: updated, changefreq: 'daily', priority: '0.9' }), + sitemapEntry(base, '/trackers', { lastmod: updated, changefreq: 'daily', priority: '0.9' }), + sitemapEntry(base, '/companies', { lastmod: updated, changefreq: 'daily', priority: '0.8' }), + sitemapEntry(base, '/about', { changefreq: 'monthly', priority: '0.7' }) + ]; + + for (const slug of index.trackerList) + entries.push(sitemapEntry(base, `/tracker/${slug}`, { lastmod: updated, changefreq: 'weekly', priority: '0.7' })); + + for (const slug of index.companyList) + entries.push(sitemapEntry(base, `/company/${slug}`, { lastmod: updated, changefreq: 'weekly', priority: '0.6' })); + + for (const app of Object.values(index.apps)) + entries.push(sitemapEntry(base, `/analysis/${app.appid}`, { + lastmod: app.analysed, + changefreq: 'monthly', + priority: '0.6' + })); + + return entries; +} + +async function getSitemapPages(base) { + let index; + try { + index = await getReverseIndex(); + } catch (err) { + console.error('Sitemap error:', err.message); + index = EMPTY_REVERSE_INDEX; + } + return splitSitemapEntries(sitemapEntries(base, index)); +} + +// Sitemap over the report, lookup and reference pages. Built from the cached +// reverse index so a crawl does not read every stored analysis from the +// database. +router.get('/sitemap.xml', asyncHandler(async (req, res) => { + const base = siteBaseUrl(req); + const pages = await getSitemapPages(base); + + res.type('application/xml').send(pages.length === 1 + ? renderSitemap(pages[0]) + : renderSitemapIndex(base, pages.length)); +})); + +router.get('/sitemap-:page.xml', asyncHandler(async (req, res) => { + const pageNumber = Number(req.params.page); + if (!Number.isInteger(pageNumber) || pageNumber < 1) + return res.status(404).send('Sitemap not found.'); + + const base = siteBaseUrl(req); + const pages = await getSitemapPages(base); + if (pageNumber > pages.length) + return res.status(404).send('Sitemap not found.'); + + res.type('application/xml').send(renderSitemap(pages[pageNumber - 1])); +})); + +// /search and /request/ are GETs, so unlike a form post they are reachable by +// a crawler that finds the URL. Both spend an App Store call and both sit +// behind a Cloudflare Managed Challenge, so a crawl of them would burn quota +// and collect interstitials rather than content. +router.get('/robots.txt', (req, res) => { + res.type('text/plain').send([ + 'User-agent: *', + 'Allow: /', + 'Disallow: /search', + 'Disallow: /request/', + 'Disallow: /queue', + 'Disallow: /ping', + 'Disallow: /healthz', + '', + `Sitemap: ${siteBaseUrl(req)}/sitemap.xml`, + '' + ].join('\n')); +}); module.exports = router; // make accessible to /app.js diff --git a/server.js b/server.js index f856196..d2f08b5 100644 --- a/server.js +++ b/server.js @@ -38,6 +38,28 @@ const analyserPaths = new Set([ const isAnalyserPath = (req) => analyserPaths.has(req.path.toLowerCase().replace(/\/+$/, '')); +// /search and /request/:appId are GETs only so that a Cloudflare challenge can +// replay them; each one still reaches the App Store. The method therefore does +// not separate cheap from expensive here, and they are budgeted as the form +// submissions they are. +const appStorePaths = (path) => + path === '/search' || path === '/request' || path.startsWith('/request/'); + +const isAppStorePath = (req) => + appStorePaths(req.path.toLowerCase().replace(/\/+$/, '')); + +// Reads of the published pages are served from the cached site data and +// reverse index, so they cost far less than an App Store call. They also +// arrive in very different volumes: sitemap.xml points crawlers at every app, +// tracker and company URL, and a crawler works through those from a narrow +// range of addresses. Sharing one budget between the two means either +// throttling a normal crawl or loosening the limit that actually matters, so +// they are budgeted separately. +const isBrowseRequest = (req) => + (req.method === 'GET' || req.method === 'HEAD') + && !isAnalyserPath(req) + && !isAppStorePath(req); + // Optional hardening for the case where the origin becomes reachable without // Cloudflare: the WAF challenge rules protecting /search and the request page // only apply to traffic that goes through the edge. Inert unless @@ -45,14 +67,27 @@ const isAnalyserPath = (req) => app.use(originGate()) if(os.hostname().indexOf("local") <= -1) { // only on remote host - const limiter = rateLimit({ - windowMs: 5 * 60 * 1000, // 5 minutes - max: 100, // Limit each IP to 10 requests per `window` + const windowMs = 5 * 60 * 1000; // 5 minutes + const skipAnalyser = (req) => isAnalyserPath(req) && analyserAuthenticated(req); + + // Everything that is not a cacheable page view: the App Store entry points, + // the analysis request POST, and analyser endpoints called without + // credentials. + app.use(rateLimit({ + windowMs, + max: Number(process.env.RATE_LIMIT_FORM_MAX) || 20, + standardHeaders: false, + legacyHeaders: false, + skip: (req) => skipAnalyser(req) || isBrowseRequest(req), + })) + + app.use(rateLimit({ + windowMs, + max: Number(process.env.RATE_LIMIT_BROWSE_MAX) || 300, standardHeaders: false, legacyHeaders: false, - skip: (req) => isAnalyserPath(req) && analyserAuthenticated(req), - }) - app.use(limiter) + skip: (req) => skipAnalyser(req) || !isBrowseRequest(req), + })) } const analyserBodyLimit = process.env.BODY_LIMIT || '25mb'; diff --git a/test/appMetadata.test.js b/test/appMetadata.test.js index 71aa179..2b1e9f7 100644 --- a/test/appMetadata.test.js +++ b/test/appMetadata.test.js @@ -2,7 +2,7 @@ const assert = require('node:assert/strict'); const test = require('node:test'); -const { buildReportMetadata } = require('../lib/appMetadata'); +const { buildReportMetadata, buildListingDetails } = require('../lib/appMetadata'); test('report metadata prefers current storefront and exposes version divergence', () => { const metadata = buildReportMetadata({ @@ -71,3 +71,54 @@ test('queue snapshot supplies a version when no current storefront exists', () = assert.equal(metadata.currentVersion, '1.0'); assert.equal(metadata.currentVersionFromStorefront, false); }); + +test('listing details prefer the refreshed storefront over the queue snapshot', () => { + const details = buildListingDetails({ + queueSnapshot: { + title: 'Queue title', + icon: 'queue-icon', + primaryGenre: 'Games', + reviews: 100, + free: true + }, + storefront: { + details: { title: 'Renamed', icon: 'new-icon', primaryGenre: 'News', reviews: 5000 } + } + }); + + assert.equal(details.title, 'Renamed'); + assert.equal(details.icon, 'new-icon'); + assert.equal(details.primaryGenre, 'News'); + assert.equal(details.reviews, 5000); + // Fields the storefront does not carry survive the refresh. + assert.equal(details.free, true); +}); + +test('listing details fall back field by field, not wholesale', () => { + const details = buildListingDetails({ + queueSnapshot: { title: 'Queue title', icon: 'queue-icon', url: 'queue-url' }, + // A partial refresh must not blank the fields it omits. + storefront: { details: { title: 'Renamed', icon: '' } } + }); + + assert.equal(details.title, 'Renamed'); + assert.equal(details.icon, 'queue-icon'); + assert.equal(details.url, 'queue-url'); +}); + +test('listing details keep a refreshed review count of zero', () => { + const details = buildListingDetails({ + queueSnapshot: { title: 'Queued', reviews: 4000 }, + storefront: { details: { reviews: 0 } } + }); + + assert.equal(details.reviews, 0); +}); + +test('listing details tolerate a missing storefront row and a missing snapshot', () => { + assert.deepEqual( + buildListingDetails({ queueSnapshot: { title: 'Queued' }, storefront: { details: null } }), + { title: 'Queued' } + ); + assert.deepEqual(buildListingDetails(), {}); +}); diff --git a/test/cache.test.js b/test/cache.test.js new file mode 100644 index 0000000..44762eb --- /dev/null +++ b/test/cache.test.js @@ -0,0 +1,55 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const test = require('node:test'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +// Resolved at require time, so it must be set before lib/cache is loaded. +process.env.CACHE_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'tc-cache-schema-')); + +const cache = require('../lib/cache'); + +const cacheFile = (key) => path.join(process.env.CACHE_DIR, key + '.json'); + +test('the cache is keyed by builder schema as well as data signature', async (t) => { + t.after(() => fs.rmSync(process.env.CACHE_DIR, { recursive: true, force: true })); + + await t.test('a round-trip returns the data and its signature', () => { + const signature = { appCount: 2, latestAnalysis: '2025-01-02T03:04:05.000Z' }; + cache.write('roundtrip', { totalApps: 2 }, signature); + + const cached = cache.read('roundtrip'); + assert.deepEqual(cached.data, { totalApps: 2 }); + assert.deepEqual(cached.meta, signature); + }); + + await t.test('an entry from an older schema reads as a miss', () => { + // CACHE_DIR is a persistent volume, so entries written before a deploy + // survive it. The signature alone still matches when only the derivation + // changed, which is what the schema version exists to catch. + const signature = { appCount: 2, latestAnalysis: '2025-01-02T03:04:05.000Z' }; + fs.writeFileSync(cacheFile('stale'), JSON.stringify({ + data: { totalApps: 2 }, + _meta: signature, + _schema: cache.SCHEMA_VERSION - 1 + }), 'utf-8'); + + assert.equal(cache.read('stale'), null); + }); + + await t.test('an entry predating the schema field reads as a miss', () => { + fs.writeFileSync(cacheFile('legacy'), JSON.stringify({ + data: { totalApps: 2 }, + _meta: { appCount: 2, latestAnalysis: null } + }), 'utf-8'); + + assert.equal(cache.read('legacy'), null); + }); + + await t.test('a rewrite at the current schema is readable again', () => { + cache.write('stale', { totalApps: 3 }, { appCount: 3, latestAnalysis: null }); + assert.deepEqual(cache.read('stale').data, { totalApps: 3 }); + }); +}); diff --git a/test/lookupPages.test.js b/test/lookupPages.test.js new file mode 100644 index 0000000..f521586 --- /dev/null +++ b/test/lookupPages.test.js @@ -0,0 +1,258 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const test = require('node:test'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +// Must be set before the cache and server modules are loaded: the cache +// directory is resolved at require time. +process.env.CACHE_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'tc-cache-')); +process.env.UPLOAD_PASSWORD = 'test-secret'; +process.env.SITE_URL = 'https://example.test'; + +const Apps = require('../models/Apps'); +const app = require('../server'); + +const analysed = new Date('2025-01-02T03:04:05.000Z'); + +const corpus = [ + { + appid: 'com.example.two', + analysed, + details: { + appId: 'com.example.two', + title: 'Example Two', + icon: 'https://icons.test/two.png', + version: '2.0', + url: 'https://apps.apple.com/two', + reviews: 9000, + primaryGenre: 'News', + free: true + }, + analysis: { trackers: { 'Google Firebase Analytics': {} }, permissions: ['Camera'] } + }, + { + appid: 'com.example.one', + analysed, + details: { + appId: 'com.example.one', + title: 'Example One', + icon: 'https://icons.test/one.png', + version: '1.0', + url: 'https://apps.apple.com/one', + reviews: 500, + primaryGenre: 'Games', + free: true + }, + // The queue-time snapshot above is frozen; this is what the metadata + // refresh job keeps up to date. + current_storefront_details: { + title: 'Example One Renamed', + icon: 'https://icons.test/one-v2.png' + }, + analysis: { trackers: { 'Google Firebase Analytics': {}, 'Facebook Login': {} } } + } +]; + +async function withServer(run) { + const server = await new Promise((resolve) => { + const instance = app.listen(0, '127.0.0.1', () => resolve(instance)); + }); + + try { + await run(`http://127.0.0.1:${server.address().port}`); + } finally { + await new Promise((resolve, reject) => { + server.close((err) => err ? reject(err) : resolve()); + }); + } +} + +function stubDatabase() { + const original = { + getSiteDataSignature: Apps.getSiteDataSignature, + getAllApps: Apps.getAllApps, + findApp: Apps.findApp, + log: console.log + }; + + Apps.getSiteDataSignature = async () => ({ + appCount: corpus.length, + latestAnalysis: analysed.toISOString() + }); + Apps.getAllApps = async () => corpus; + Apps.findApp = async (appId) => + corpus.find((row) => row.appid.toLowerCase() === String(appId).toLowerCase()) || null; + console.log = () => {}; + + return () => { + Apps.getSiteDataSignature = original.getSiteDataSignature; + Apps.getAllApps = original.getAllApps; + Apps.findApp = original.findApp; + console.log = original.log; + }; +} + +test('reverse lookup, about page, sitemap and social metadata', async (t) => { + const restore = stubDatabase(); + + try { + await withServer(async (base) => { + await t.test('tracker directory lists trackers and links to lookups', async () => { + const response = await fetch(`${base}/trackers`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /Google Firebase Analytics/); + assert.match(body, /href="\/tracker\/google-firebase-analytics"/); + assert.match(body, /href="\/company\/alphabet"/); + }); + + await t.test('company directory renders', async () => { + const response = await fetch(`${base}/companies`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /href="\/company\/alphabet"/); + }); + + await t.test('tracker page lists the apps, most reviewed first', async () => { + const response = await fetch(`${base}/tracker/google-firebase-analytics`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /href="\/analysis\/com.example.two"/); + assert.match(body, /href="\/analysis\/com.example.one"/); + assert.ok( + body.indexOf('Example Two') < body.indexOf('Example One'), + 'the app with more reviews should come first' + ); + assert.match(body, //); + assert.match(body, /detected in 2 of 2 analysed iOS apps/); + }); + + await t.test('listing pages show refreshed storefront metadata, not the queue snapshot', async () => { + const response = await fetch(`${base}/tracker/google-firebase-analytics`); + const body = await response.text(); + + assert.match(body, /Example One Renamed/); + assert.match(body, /one-v2\.png/); + assert.doesNotMatch(body, /icons\.test\/one\.png/); + }); + + await t.test('company page aggregates its trackers', async () => { + const response = await fetch(`${base}/company/alphabet`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /href="\/tracker\/google-firebase-analytics"/); + assert.match(body, /href="\/analysis\/com.example.two"/); + }); + + await t.test('unknown slugs 404 instead of erroring', async () => { + for (const url of ['/tracker/no-such-tracker', '/company/no-such-company', '/tracker/__proto__']) { + const response = await fetch(`${base}${url}`); + assert.equal(response.status, 404, url); + } + }); + + await t.test('about page carries the sampling caveat and the jurisdiction labels', async () => { + const response = await fetch(`${base}/about`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /this is not a random sample of the App Store/i); + // The labels are rendered from the same metadata the reports use, so + // the key cannot drift out of the page without the reports changing too. + assert.match(body, /Every company we identified is based in the US/); + assert.match(body, //); + }); + + await t.test('the retired methodology page is gone', async () => { + const response = await fetch(`${base}/methodology`); + assert.equal(response.status, 404); + }); + + await t.test('app report links trackers to their lookup page and sets a social image', async () => { + const response = await fetch(`${base}/analysis/com.example.one`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /href="\/tracker\/google-firebase-analytics"/); + // The social card follows the same storefront precedence as the report + // body, so it cannot advertise a title or icon the page contradicts. + assert.match(body, //); + assert.match(body, /2 trackers were detected in Example One Renamed/); + }); + + await t.test('statistics page links its tables into the lookup pages', async () => { + const response = await fetch(`${base}/statistics`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /href="\/tracker\/google-firebase-analytics"/); + assert.match(body, /href="\/company\/alphabet"/); + }); + + await t.test('no page glues a word onto an inline tag', async () => { + // Pug joins sibling piped-text lines with a newline, but concatenates a + // sibling tag with no whitespace at all, so dropping the trailing space + // from `| ...read more` above `a(href=...) here` silently renders + // "read morehere". Catch it in the output rather than in review. + // One tag only: a run of several (``) is two + // elements sitting next to each other, which is a layout decision + // rather than a sentence that lost its space. + const inline = /<\/?(?:b|i|em|strong|a|span|code|abbr|small)\b[^>]*>/; + const glued = new RegExp(`[A-Za-z0-9.,:]${inline.source}[A-Za-z0-9]`, 'g'); + + for (const page of ['/', '/about', '/analysis/com.example.one', '/statistics', + '/trackers', '/companies', '/tracker/google-firebase-analytics']) { + const body = await (await fetch(`${base}${page}`)).text(); + const hits = [...body.matchAll(glued)] + .map((m) => body.slice(Math.max(0, m.index - 40), m.index + m[0].length + 20)); + + assert.deepEqual(hits, [], `${page} renders text with no space before or after an inline tag`); + } + }); + + await t.test('sitemap covers reports, lookups and reference pages', async () => { + const response = await fetch(`${base}/sitemap.xml`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(response.headers.get('content-type'), /xml/); + assert.match(body, /https:\/\/example.test\/<\/loc>/); + assert.match(body, /https:\/\/example.test\/about<\/loc>/); + assert.match(body, /https:\/\/example.test\/tracker\/google-firebase-analytics<\/loc>/); + assert.match(body, /https:\/\/example.test\/company\/alphabet<\/loc>/); + assert.match(body, /https:\/\/example.test\/analysis\/com.example.one<\/loc>/); + assert.match(body, new RegExp(`${analysed.toISOString()}`)); + }); + + await t.test('robots.txt points at the sitemap and withholds the App Store paths', async () => { + const response = await fetch(`${base}/robots.txt`); + const body = await response.text(); + + assert.equal(response.status, 200); + assert.match(body, /Sitemap: https:\/\/example.test\/sitemap.xml/); + // Both are GETs so that Cloudflare can challenge them, which also + // makes them crawlable; each one spends an App Store call. + assert.match(body, /^Disallow: \/search$/m); + assert.match(body, /^Disallow: \/request\/$/m); + }); + + await t.test('the sitemap never advertises a path that costs an App Store call', async () => { + const response = await fetch(`${base}/sitemap.xml`); + const body = await response.text(); + + assert.doesNotMatch(body, /[^<]*\/search/); + assert.doesNotMatch(body, /[^<]*\/request\//); + }); + }); + } finally { + restore(); + fs.rmSync(process.env.CACHE_DIR, { recursive: true, force: true }); + } +}); diff --git a/test/reverseIndex.test.js b/test/reverseIndex.test.js new file mode 100644 index 0000000..240853b --- /dev/null +++ b/test/reverseIndex.test.js @@ -0,0 +1,202 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const test = require('node:test'); +const reverseIndex = require('../lib/reverseIndex'); + +function app(appid, { trackers, title, reviews, analysed, success, genre } = {}) { + return { + appid, + analysed: analysed || '2025-01-01T00:00:00.000Z', + details: { + appId: appid, + title: title || appid, + icon: `https://example.test/${appid}.png`, + reviews: reviews === undefined ? 0 : reviews, + primaryGenre: genre || 'Utilities' + }, + analysis: success === false + ? { success: false, reason: 'app_not_found' } + : { trackers: (trackers || []).reduce((acc, name) => ({ ...acc, [name]: {} }), {}) } + }; +} + +const corpus = [ + app('com.example.one', { trackers: ['Google Firebase Analytics', 'Facebook Login'], reviews: 500, title: 'One' }), + app('com.example.two', { trackers: ['Google Firebase Analytics'], reviews: 9000, title: 'Two' }), + app('com.example.three', { trackers: [], reviews: 10, title: 'Three' }), + app('com.example.four', { trackers: ['Google Firebase Analytics'], success: false, title: 'Four' }) +]; + +test('index counts apps per tracker and uses all analysed apps as denominator', () => { + const index = reverseIndex.buildReverseIndex(corpus); + + // The failed analysis is excluded; the tracker-free app still counts. + assert.equal(index.totalApps, 3); + assert.equal(index.trackedApps, 2); + + const slug = index.trackerSlugs['google firebase analytics']; + const firebase = reverseIndex.lookupTracker(index, slug); + assert.equal(firebase.appCount, 2); + assert.equal(firebase.pct, '66.7'); + assert.ok(!firebase.appIds.includes('com.example.four')); +}); + +test('apps within a tracker are ordered by review count', () => { + const index = reverseIndex.buildReverseIndex(corpus); + const firebase = reverseIndex.lookupTracker(index, index.trackerSlugs['google firebase analytics']); + + assert.deepEqual(firebase.appIds, ['com.example.two', 'com.example.one']); +}); + +test('the app directory covers analysed apps without trackers', () => { + const index = reverseIndex.buildReverseIndex(corpus); + + assert.ok(index.apps['com.example.three']); + assert.equal(index.apps['com.example.three'].trackerCount, 0); + assert.equal(index.apps['com.example.three'].classification, 'no_tracking'); + assert.ok(!index.apps['com.example.four'], 'failed analyses stay out of the directory'); +}); + +test('trackers are grouped under the company that owns them', () => { + const index = reverseIndex.buildReverseIndex(corpus); + const firebase = reverseIndex.lookupTracker(index, index.trackerSlugs['google firebase analytics']); + + assert.equal(firebase.company, 'Alphabet'); + assert.equal(firebase.region, 'US'); + + const company = reverseIndex.lookupCompany(index, firebase.companySlug); + assert.equal(company.name, 'Alphabet'); + assert.equal(company.appCount, 2); + assert.ok(company.trackers.some((tracker) => tracker.name === 'Google Firebase Analytics')); +}); + +test('an app is counted once per company even with several of its trackers', () => { + const index = reverseIndex.buildReverseIndex([ + app('com.example.multi', { trackers: ['Google Firebase Analytics', 'Google AdMob'], reviews: 1 }) + ]); + + const firebase = reverseIndex.lookupTracker(index, index.trackerSlugs['google firebase analytics']); + const company = reverseIndex.lookupCompany(index, firebase.companySlug); + + assert.equal(company.appCount, 1); + assert.equal(company.trackers.length, 2); +}); + +test('system APIs are flagged rather than listed as unattributed trackers', () => { + const index = reverseIndex.buildReverseIndex([ + app('com.example.system', { trackers: ['AdID access'], reviews: 1 }), + app('com.example.empty', { trackers: [], reviews: 1 }) + ]); + + const entry = reverseIndex.lookupTracker(index, index.trackerSlugs['adid access']); + assert.equal(entry.system, true); + assert.equal(entry.company, null); + assert.equal(entry.region, 'Unresolved'); + assert.equal(index.totalApps, 2); + assert.equal(index.trackedApps, 0); + assert.equal(index.apps['com.example.system'].trackerCount, 0); +}); + +test('slugs are URL-safe and collisions get distinct slugs', () => { + assert.equal(reverseIndex.slugify('Mob.com'), 'mob-com'); + assert.equal(reverseIndex.slugify('Unity3d Ads'), 'unity3d-ads'); + assert.equal(reverseIndex.slugify('!!!'), 'unnamed'); + + const index = reverseIndex.buildReverseIndex([ + app('com.example.collide', { trackers: ['Mob.com', 'Mob com'], reviews: 1 }) + ]); + + const slugs = index.trackerList.slice().sort(); + assert.equal(slugs.length, 2); + assert.ok(slugs.includes('mob-com')); + assert.match(slugs.find((slug) => slug !== 'mob-com'), /^mob-com-[a-f0-9]{8}$/); + for (const slug of slugs) assert.ok(reverseIndex.lookupTracker(index, slug)); +}); + +test('previous slugs stay attached to their names as the corpus grows', () => { + const before = reverseIndex.buildReverseIndex([ + app('com.example.original', { trackers: ['Mob.com'] }) + ]); + const after = reverseIndex.buildReverseIndex([ + app('com.example.original', { trackers: ['Mob.com'] }), + app('com.example.new', { trackers: ['Mob com'] }) + ], before); + + assert.equal(after.trackerSlugs['mob.com'], before.trackerSlugs['mob.com']); + assert.match(after.trackerSlugs['mob com'], /^mob-com-[a-f0-9]{8}$/); + assert.equal(reverseIndex.lookupTracker(after, before.trackerSlugs['mob.com']).name, 'Mob.com'); +}); + +test('lookup rejects invalid slugs and inherited properties', () => { + const index = reverseIndex.buildReverseIndex(corpus); + + assert.equal(reverseIndex.lookupTracker(index, 'constructor'), null); + assert.equal(reverseIndex.lookupTracker(index, '__proto__'), null); + assert.equal(reverseIndex.lookupTracker(index, '../../etc/passwd'), null); + assert.equal(reverseIndex.lookupTracker(index, ''), null); + assert.equal(reverseIndex.isValidSlug('Mob-Com'), false); + assert.equal(reverseIndex.isValidSlug('mob-com'), true); +}); + +test('the index survives a JSON cache round-trip', () => { + const index = JSON.parse(JSON.stringify(reverseIndex.buildReverseIndex(corpus))); + const firebase = reverseIndex.lookupTracker(index, index.trackerSlugs['google firebase analytics']); + + assert.equal(firebase.appCount, 2); + assert.equal(index.trackerList[0], firebase.slug); +}); + +test('pagination clamps the page number and resolves app records', () => { + const index = reverseIndex.buildReverseIndex(corpus); + const firebase = reverseIndex.lookupTracker(index, index.trackerSlugs['google firebase analytics']); + + const first = reverseIndex.paginate(firebase.appIds, index.apps, 1, 1); + assert.equal(first.totalPages, 2); + assert.equal(first.from, 1); + assert.equal(first.to, 1); + assert.equal(first.apps[0].title, 'Two'); + + const clamped = reverseIndex.paginate(firebase.appIds, index.apps, 99, 1); + assert.equal(clamped.page, 2); + assert.equal(clamped.apps[0].title, 'One'); + + const empty = reverseIndex.paginate([], index.apps, 1, 50); + assert.equal(empty.total, 0); + assert.equal(empty.from, 0); + assert.equal(empty.apps.length, 0); +}); + +test('pagination counts only apps that still exist in the directory', () => { + const pagination = reverseIndex.paginate( + ['com.example.one', 'com.example.missing'], + { 'com.example.one': { title: 'One' } }, + 1, + 50 + ); + + assert.equal(pagination.total, 1); + assert.equal(pagination.from, 1); + assert.equal(pagination.to, 1); + assert.deepEqual(pagination.apps, [{ title: 'One' }]); +}); + +test('parsePage falls back to the first page for junk input', () => { + assert.equal(reverseIndex.parsePage('3'), 3); + assert.equal(reverseIndex.parsePage('0'), 1); + assert.equal(reverseIndex.parsePage('-2'), 1); + assert.equal(reverseIndex.parsePage('abc'), 1); + assert.equal(reverseIndex.parsePage(undefined), 1); +}); + +test('slugForName only resolves names that are present', () => { + const index = reverseIndex.buildReverseIndex(corpus); + + assert.equal( + reverseIndex.slugForName(index.trackerSlugs, 'GOOGLE FIREBASE ANALYTICS'), + index.trackerSlugs['google firebase analytics'] + ); + assert.equal(reverseIndex.slugForName(index.trackerSlugs, 'Not A Tracker'), null); + assert.equal(reverseIndex.slugForName(index.trackerSlugs, 'toString'), null); + assert.equal(reverseIndex.slugForName(index.trackerSlugs, null), null); +}); diff --git a/test/siteUrl.test.js b/test/siteUrl.test.js new file mode 100644 index 0000000..1214e9a --- /dev/null +++ b/test/siteUrl.test.js @@ -0,0 +1,58 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const test = require('node:test'); +const siteUrl = require('../lib/siteUrl'); + +function withEnv(values, run) { + const original = { SITE_URL: process.env.SITE_URL, NODE_ENV: process.env.NODE_ENV }; + Object.assign(process.env, values); + for (const [key, value] of Object.entries(values)) + if (value === undefined) delete process.env[key]; + + try { + run(); + } finally { + for (const [key, value] of Object.entries(original)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +} + +const request = (protocol, host) => ({ protocol, get: () => host }); + +test('production refuses to resolve a base URL from the request', () => { + withEnv({ NODE_ENV: 'production', SITE_URL: undefined }, () => { + assert.equal(siteUrl.getSiteUrlConfigurationError(), 'SITE_URL is not set'); + assert.throws(() => siteUrl.assertSiteUrlConfiguration(), /SITE_URL is not set/); + // An untrusted Host header must not become a public canonical URL. + assert.throws(() => siteUrl.siteBaseUrl(request('http', 'evil.test')), /SITE_URL is not set/); + }); +}); + +test('a blank SITE_URL counts as unset in production', () => { + withEnv({ NODE_ENV: 'production', SITE_URL: ' ' }, () => { + assert.equal(siteUrl.getSiteUrlConfigurationError(), 'SITE_URL is not set'); + }); +}); + +test('a configured SITE_URL wins over the request and loses its trailing slashes', () => { + withEnv({ NODE_ENV: 'production', SITE_URL: 'https://ios.example.org//' }, () => { + assert.equal(siteUrl.getSiteUrlConfigurationError(), null); + assert.equal( + siteUrl.siteBaseUrl(request('http', 'evil.test')), + 'https://ios.example.org' + ); + }); +}); + +test('development falls back to the request origin', () => { + withEnv({ NODE_ENV: 'development', SITE_URL: undefined }, () => { + assert.equal(siteUrl.getSiteUrlConfigurationError(), null); + assert.equal( + siteUrl.siteBaseUrl(request('http', 'localhost:3000')), + 'http://localhost:3000' + ); + }); +}); diff --git a/views/about.pug b/views/about.pug index 959b514..efcc895 100644 --- a/views/about.pug +++ b/views/about.pug @@ -3,58 +3,124 @@ extends layout block content h2 About TrackerControl for iOS p - | This service analyses iOS applications in order to - b list the embedded trackers and permissions - | . A tracker is a piece of software that collects data about - b you or your app usage behaviour - | . - p This project was motivated by - a(target='_blank' rel='noopener noreferrer' href='https://exodus-privacy.eu.org/') Exodus Privacy - | , which is a similar project for Android apps. Some of the underlying code as well as the design of this website is based on this project. - p The underlying analysis technique for iOS apps was developed in the PhD research of Konrad Kollnig at the Department of Computer Science of the University of Oxford. - | This research was published in a range of academic papers and is available at - a(target='_blank' rel='noopener noreferrer' href='https://www.platformcontrol.org/') PlatformControl.org - | . This research was, in turn, based - a(target='_blank' rel='noopener noreferrer' href='https://sociam.org/mobile-app-x-ray') on previous work - | by the Oxford research group led by Sir Nigel Shadbolt and led to - a(target='_blank' rel='noopener noreferrer' href='https://trackercontrol.org/') TrackerControl for Android - | . - p A key aim of this research and this tool is to enable anyone to analyse privacy in iOS apps without relying on jailbreaks or the circumvention of copyright protections. These were issues that held back iOS research in the past. - p Unfortunately, Apple currently encrypts every app downloaded from the App Store with its FairPlay DRM. The circumvention of this DRM might be illegal in some jurisdictions and is thus not done by this tool. This is one of the key innovations behind this work, and has been published in the highly renowed and selective - a(target='_blank' rel='noopener noreferrer' href='https://petsymposium.org/popets/2022/popets-2022-0033.pdf') Proceedings on Privacy Enhancing Technologies - | in 2022. - p This website operates a fair use policy and restricts disproportionate access. Scraping is not permitted. - p We never collect personal data. It's yours. This website is operated by Konrad Kollnig, Assistant Professor at Maastricht University. + | This service analyses iOS applications in order to #[b list the embedded trackers and permissions]. A tracker is a piece of software that collects data about #[b you or your app usage behaviour]. + + h3.mt-5#how-it-works How an app is analysed + .row.pipeline + .col-6.col-md-3.mb-3 + .pipeline-step + svg.pipeline-icon(xmlns='http://www.w3.org/2000/svg' width='34' height='34' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='1.6' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + path(d='M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4') + polyline(points='7 10 12 15 17 10') + line(x1='12' y1='15' x2='12' y2='3') + h4 Download + p We fetch the app from the UK App Store, like anyone else would. + .col-6.col-md-3.mb-3 + .pipeline-step + svg.pipeline-icon(xmlns='http://www.w3.org/2000/svg' width='34' height='34' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='1.6' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + rect(x='6' y='2' width='12' height='20' rx='2') + line(x1='11' y1='18' x2='13' y2='18') + h4 Run + p We install it on a real iPhone and start it up. + .col-6.col-md-3.mb-3 + .pipeline-step + svg.pipeline-icon(xmlns='http://www.w3.org/2000/svg' width='34' height='34' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='1.6' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + circle(cx='11' cy='11' r='7') + line(x1='16.5' y1='16.5' x2='21' y2='21') + h4 Look inside + p We list the tracking software the app comes with. + .col-6.col-md-3.mb-3 + .pipeline-step + svg.pipeline-icon(xmlns='http://www.w3.org/2000/svg' width='34' height='34' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='1.6' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + circle(cx='12' cy='12' r='9') + line(x1='3' y1='12' x2='21' y2='12') + path(d='M12 3a15 15 0 0 1 0 18a15 15 0 0 1 0-18z') + h4 Trace it back + p Each tracker is matched to the company behind it, and its home country. + .alert.alert-warning(role='alert') + p.mb-0#sample + | We analyse free apps from the UK App Store, and an app joins the queue when somebody searches for it here. #[b So this is not a random sample of the App Store.] It leans heavily towards popular apps: a percentage on this site describes the apps analysed so far, not iOS apps in general. - h3.mt-5 About Jurisdiction Analysis + h3.mt-5#results What a report tells you + .row + .col-md-6.mb-3 + .result-card.result-card-yes + h4 + svg(xmlns='http://www.w3.org/2000/svg' width='20' height='20' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='2.5' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + polyline(points='4 12 10 18 20 6') + | What we can see + ul.mb-0 + li Which tracking software is built into an app. + li Which companies it belongs to, and where they are based. + li Which permissions the app is able to ask you for. + .col-md-6.mb-3 + .result-card.result-card-no + h4 + svg(xmlns='http://www.w3.org/2000/svg' width='20' height='20' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='2.5' stroke-linecap='round' stroke-linejoin='round' role='img' aria-hidden='true') + line(x1='6' y1='6' x2='18' y2='18') + line(x1='18' y1='6' x2='6' y2='18') + | What we cannot + ul.mb-0 + li Whether a tracker ever ran, or whether anything left your phone. + li Tracking we have no way of seeing, such as an app sending data straight to its developer's own servers. An empty report is not a promise. + li What a company does with your data once it has it. + + h3.mt-5#jurisdiction Who controls the tracking p - | This tool also analyses the - b jurisdictional geography - | of third-party tracking: which countries control the tracking infrastructure in each app, and what that means for data sovereignty. + | Each tracker is traced to the company that runs it, up to the parent company at the top, and then to that company's home country. What counts is whose laws a company answers to, not where its servers are: a US company can be compelled to hand data over under the CLOUD Act wherever that data is stored. Every app gets one label: + .table-responsive + table.table.table-sm.table-striped + tbody + tr + td + span.badge.badge-pill(class=jurisdictionMeta.no_tracking.cssClass) !{jurisdictionMeta.no_tracking.icon} #{jurisdictionMeta.no_tracking.label} + td No third-party tracker was found. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.us_only.cssClass) !{jurisdictionMeta.us_only.icon} #{jurisdictionMeta.us_only.label} + td Every company we identified is based in the US. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.european_only.cssClass) !{jurisdictionMeta.european_only.icon} #{jurisdictionMeta.european_only.label} + td Every company we identified is based in the EU, EEA or UK, and subject to GDPR. This is rare. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.mixed_with_us_cn.cssClass) !{jurisdictionMeta.mixed_with_us_cn.icon} #{jurisdictionMeta.mixed_with_us_cn.label} + td The companies include both US and Chinese ones. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.mixed_with_us.cssClass) !{jurisdictionMeta.mixed_with_us.icon} #{jurisdictionMeta.mixed_with_us.label} + td The companies include US ones alongside others, but none from China. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.mixed_no_us.cssClass) !{jurisdictionMeta.mixed_no_us.icon} #{jurisdictionMeta.mixed_no_us.label} + td The companies span several parts of the world, but none is in the US. + tr + td + span.badge.badge-pill(class=jurisdictionMeta.unresolved_only.cssClass) !{jurisdictionMeta.unresolved_only.icon} #{jurisdictionMeta.unresolved_only.label} + td Trackers were found, but we could not tell which companies run them. + + h3.mt-5#limitations Where this stops + ul + li We can only find trackers we already know about, so new or renamed ones are missed. + li A tracker occasionally ends up attributed to the wrong company, so it is worth checking an individual result before leaning on it. + li A report describes the version of the app we analysed, on the date shown. That may no longer be the version in the App Store. p - b What this measures: - | Which companies control the tracking infrastructure in your apps, and what jurisdiction those companies fall under. + | If a result looks wrong, tell us — corrections make this better for everyone. + + h3.mt-5#background Where this comes from p - b Why it matters: - | Even if data is processed in compliance with GDPR, US-based companies are subject to the CLOUD Act, which allows US authorities to compel disclosure regardless of where data is stored. The Schrems II ruling invalidated the EU-US Privacy Shield for this reason. + | This project was motivated by #[a(target='_blank' rel='noopener noreferrer' href='https://exodus-privacy.eu.org/') Exodus Privacy], which is a similar project for Android apps. Some of the underlying code as well as the design of this website is based on this project. p - b What "US-only" means: - | Every identified tracker in the app is controlled by a US-headquartered company. This doesn't mean the data is stored in the US — it means the company - em could - | be compelled to hand it over under US law. + | The underlying analysis technique for iOS apps was developed in the PhD research of Konrad Kollnig at the Department of Computer Science of the University of Oxford. This research was published in a range of academic papers and is available at #[a(target='_blank' rel='noopener noreferrer' href='https://www.platformcontrol.org/') PlatformControl.org]. This research was, in turn, based #[a(target='_blank' rel='noopener noreferrer' href='https://sociam.org/mobile-app-x-ray') on previous work] by the Oxford research group led by Sir Nigel Shadbolt and led to #[a(target='_blank' rel='noopener noreferrer' href='https://trackercontrol.org/') TrackerControl for Android]. + p A key aim of this research and this tool is to enable anyone to analyse privacy in iOS apps without relying on jailbreaks or the circumvention of copyright protections. These were issues that held back iOS research in the past. p - b What "European-only" means: - | All tracking uses infrastructure controlled by companies in EU/UK/EEA countries, subject to GDPR. This is rare — across a study of ~24,000 apps, only 0.1% qualified. + | Unfortunately, Apple currently encrypts every app downloaded from the App Store with its FairPlay DRM. The circumvention of this DRM might be illegal in some jurisdictions and is thus not done by this tool. This is one of the key innovations behind this work, and has been published in the highly renowed and selective #[a(target='_blank' rel='noopener noreferrer' href='https://petsymposium.org/popets/2022/popets-2022-0033.pdf') Proceedings on Privacy Enhancing Technologies] in 2022. + p This website operates a fair use policy and restricts disproportionate access. Scraping is not permitted. + p We never collect personal data. It's yours. This website is operated by Konrad Kollnig, Assistant Professor at Maastricht University. - h4.mt-4 Limitations - ul - li Not all trackers can be identified (some hosts remain unresolved). - li Static analysis detects code presence even if inactive; dynamic analysis only captures traffic during testing. - li Company ownership changes over time; the database may not reflect the very latest acquisitions. - - h3.mt-5 Contact - p You can report issues and ask questions at - a(href="mailto:ios@trackercontrol.org") ios@trackercontrol.org - | . + h3.mt-5#contact Contact + p + | You can report issues and ask questions at #[a(href='mailto:ios@trackercontrol.org') ios@trackercontrol.org], or read the code at #[a(target='_blank' rel='noopener noreferrer' href='https://github.com/TrackerControl/tracker-control-ios') github.com/TrackerControl/tracker-control-ios]. block app diff --git a/views/directory.pug b/views/directory.pug new file mode 100644 index 0000000..9b072ec --- /dev/null +++ b/views/directory.pug @@ -0,0 +1,86 @@ +extends layout + +block content + - const isTracker = kind === 'tracker' + h2= isTracker ? 'Tracker directory' : 'Company directory' + + if entries && entries.length > 0 + p.text-muted + if isTracker + | Every tracker signature detected across #{totalApps} analysed iOS apps. Select one to see the apps it was found in. + else + | Every company whose tracking code was detected across #{totalApps} analysed iOS apps, ranked by reach. Select one to see the apps it reaches. + p.text-muted.small + | #{trackedApps} of #{totalApps} analysed apps contain at least one detected tracker. + if latestAnalysis + | Last analysis: #{new Date(latestAnalysis).toLocaleDateString('en-GB', {day:'numeric', month:'short', year:'numeric'})}. + | See #[a(href='/about#how-it-works') how these numbers are produced]. + + .form-group.mt-3 + input#directory-filter.form-control( + type='search' + placeholder=isTracker ? 'Filter trackers or companies' : 'Filter companies' + aria-label='Filter table' + data-filter-target='#directory-table') + + .table-responsive + table#directory-table.table.table-sm.table-striped + thead + tr + th # + th= isTracker ? 'Tracker' : 'Company' + if isTracker + th Company + else + th Trackers + th Country + th Region + th Apps + th % of apps + tbody + each entry, index in entries + - const haystack = `${entry.name} ${isTracker ? (entry.company || '') : ''} ${entry.countryName || ''}`.toLowerCase() + tr(data-filter=haystack) + td= index + 1 + td.font-weight-bold + a.report-link(href=`/${kind}/${entry.slug}`)= entry.name + if isTracker && entry.system + =" " + span.badge.badge-pill.region-unresolved System API + if isTracker + td + if entry.company && entry.companySlug + a.report-link(href=`/company/${entry.companySlug}`)= entry.company + else if entry.company + = entry.company + else if entry.system + span.text-muted Apple system API + else + span.text-muted Unattributed + else + td= entry.trackers.length + td + if entry.countryName + span #{entry.flag} #{entry.countryName} + else + span.text-muted Unknown + td + span.badge.badge-pill(class=`region-${entry.region.toLowerCase()}`) #{entry.region} + td= entry.appCount + td #{entry.pct}% + + p#directory-empty.text-muted.d-none No entries match that filter. + + p.text-muted.small.mt-4 + if isTracker + | A tracker is counted once per app, regardless of how many times its code appears. "System API" marks Apple platform APIs that the analyser reports but that are not third-party trackers; "Unattributed" marks signatures that could not be mapped to a company. + else + | Companies are counted at the level of the ultimate parent, so subsidiaries are grouped under their owner. An app is counted once per company even when several of that company's trackers are present. + else + .alert.alert-warning(role='alert') + p Directory data is not available right now. Please try again later. + +block app + +block scripts + script(src='/js/filter.js') diff --git a/views/form.pug b/views/form.pug index 4e33b0b..844f27a 100644 --- a/views/form.pug +++ b/views/form.pug @@ -57,7 +57,8 @@ block content div.text-muted.small Analysed #{new Date(app.analysed).toLocaleDateString('en-GB', {day:'numeric', month:'short', year:'numeric'})} p.text-center.mt-4 - a.btn.btn-outline-primary(href='/statistics') View detailed statistics → + a.btn.btn-outline-primary.mr-2.mb-2(href='/statistics') View detailed statistics → + a.btn.btn-outline-primary.mb-2(href='/trackers') Look up a tracker → if errors && errors.length > 0 .alert.alert-warning(role='alert') @@ -120,7 +121,7 @@ block app .col-md-8.col-12 a#trackers.anchor h3 - span.badge.badge-pill.badge-danger.reports #{Object.keys(app.analysis.trackers).length} + span.badge.badge-pill.badge-danger.reports #{trackerCount} =" " b trackers @@ -131,14 +132,21 @@ block app | of the following trackers in the application: if app.analysis.trackers + if systemTrackerNames && systemTrackerNames.length > 0 + p.text-muted.small Apple system API signatures are shown below for transparency and are not included in the tracker count. - const jdMap = {} - if (jurisdictionData && jurisdictionData.trackerDetails) { jurisdictionData.trackerDetails.forEach(d => { jdMap[d.name] = d }) } for tracker in Object.keys(app.analysis.trackers) + - const trackerSlug = trackerSlugs ? trackerSlugs[tracker.toLowerCase()] : null p.mb-0 - if tracker in trackerNameToExodus + if trackerSlug + a.link.black(href=`/tracker/${trackerSlug}`) #{tracker} + else if tracker in trackerNameToExodus a.link.black(target='_blank' rel='noopener noreferrer' href=`https://reports.exodus-privacy.eu.org/en/trackers/${trackerNameToExodus[tracker].id}/`) #{tracker} else | #{tracker} + if systemTrackerNames && systemTrackerNames.includes(tracker) + span.badge.badge-pill.region-unresolved.ml-1 System API - const jdEntry = jdMap[tracker] if jdEntry && jdEntry.flag span.ml-2 #{jdEntry.flag} @@ -151,8 +159,9 @@ block app =" " else span.text-muted No further information available for this tracker. - | A tracker is a piece of software meant to collect data about you or your usages. - a(target='_blank' rel='noopener noreferrer' href='https://reports.exodus-privacy.eu.org/en/info/trackers/') Learn more... + | A tracker is a piece of software meant to collect data about you or your usages. #[a(target='_blank' rel='noopener noreferrer' href='https://reports.exodus-privacy.eu.org/en/info/trackers/') Learn more...] + p.text-muted.small.mt-2 + | Select a tracker to see every other analysed app it was found in. Detection means the tracker's software is built into this app, not that data was observed leaving it — see #[a(href='/about#results') what a report tells you]. if jurisdictionData .row.justify-content-sm-center.mb-5 @@ -186,9 +195,19 @@ block app th Region tbody each detail in jurisdictionData.trackerDetails + - const detailTrackerSlug = trackerSlugs ? trackerSlugs[detail.name.toLowerCase()] : null + - const detailCompanySlug = detail.company && companySlugs ? companySlugs[detail.company.toLowerCase()] : null tr - td= detail.name - td= detail.company || 'Unknown' + td + if detailTrackerSlug + a.report-link(href=`/tracker/${detailTrackerSlug}`)= detail.name + else + = detail.name + td + if detailCompanySlug + a.report-link(href=`/company/${detailCompanySlug}`)= detail.company + else + = detail.company || 'Unknown' td if detail.flag span #{detail.flag} #{detail.countryName} @@ -212,8 +231,7 @@ block app .text-muted.mt-3 small - | Based on - a(href='/about') Xray Tracker Database + | Based on #[a(href='/about') Xray Tracker Database] if app.analysis.trackingDomains && app.analysis.trackingDomains.length > 0 .row.justify-content-sm-center.mb-5 @@ -244,8 +262,7 @@ block app p.text-truncate span(style='padding-left:28px') span(data-toggle='tooltip' data-placement='top' title='' data-original-title=`NS${permission}UsageDescription`) #{permission} - | Permissions are actions the application can do on your phone. - a(target='_blank' rel='noopener noreferrer' href='https://reports.exodus-privacy.eu.org/en/info/permissions/') Learn more... + | Permissions are actions the application can do on your phone. #[a(target='_blank' rel='noopener noreferrer' href='https://reports.exodus-privacy.eu.org/en/info/permissions/') Learn more...] if app.analysisFailure diff --git a/views/layout.pug b/views/layout.pug index b540824..597443c 100644 --- a/views/layout.pug +++ b/views/layout.pug @@ -4,6 +4,28 @@ html meta(charset='utf-8') meta(name='viewport' content='width=device-width, initial-scale=1, shrink-to-fit=no') title= `${title} | TrackerControl for iOS` + + if pageDescription + meta(name='description' content=pageDescription) + if canonicalUrl + link(rel='canonical' href=canonicalUrl) + + meta(property='og:site_name' content=siteName || 'TrackerControl for iOS') + meta(property='og:type' content=ogType || 'website') + meta(property='og:title' content=ogTitle || title) + if pageDescription + meta(property='og:description' content=pageDescription) + if canonicalUrl + meta(property='og:url' content=canonicalUrl) + if ogImage + meta(property='og:image' content=ogImage) + meta(name='twitter:card' content='summary') + meta(name='twitter:title' content=ogTitle || title) + if pageDescription + meta(name='twitter:description' content=pageDescription) + if ogImage + meta(name='twitter:image' content=ogImage) + link(rel='stylesheet', href='/css/bootstrap.min.css') link(rel='stylesheet', href='/css/exodus.css') link(rel='stylesheet', href='/css/styles.css') @@ -25,6 +47,8 @@ html ul.navbar-nav.mr-auto.mt-2.mt-lg-0 li.nav-item.mr-xl-3.ml-xl-3.mr-lg-2.ml-lg-2 a.nav-link(href='/') Home + li.nav-item.mr-xl-3.ml-xl-3.mr-lg-2.ml-lg-2 + a.nav-link(href='/trackers') Trackers li.nav-item.mr-xl-3.ml-xl-3.mr-lg-2.ml-lg-2 a.nav-link(href='/statistics') Statistics li.nav-item.mr-xl-3.ml-xl-3.mr-lg-2.ml-lg-2 @@ -52,3 +76,5 @@ html script(src='/js/jquery-3.4.1.slim.min.js') script(src='/js/popper.min.js') script(src='/js/bootstrap.min.js') + + block scripts diff --git a/views/lookup.pug b/views/lookup.pug new file mode 100644 index 0000000..468db10 --- /dev/null +++ b/views/lookup.pug @@ -0,0 +1,105 @@ +extends layout + +block content + - const isTracker = kind === 'tracker' + + h2= entry.name + p + if entry.countryName + span.badge.badge-pill(class=`region-${entry.region.toLowerCase()}`) #{entry.region} + =" " + span #{entry.flag} #{entry.countryName} + else if isTracker && entry.system + span.badge.badge-pill.region-unresolved System API + else + span.badge.badge-pill.region-unresolved Unattributed + if isTracker && entry.company + if entry.companySlug + | Operated by #[a.report-link(href=`/company/${entry.companySlug}`)= entry.company]. + else + | Operated by #{entry.company}. + + .jumbotron-stats.mb-4 + .row.text-center + .col-md-6.mb-3 + .stat-number= entry.appCount + .stat-label= entry.appCount === 1 ? 'app contains it' : 'apps contain it' + .col-md-6.mb-3 + .stat-number #{entry.pct}% + .stat-label of analysed apps + .stat-base.text-muted of #{totalApps} apps analysed + + if isTracker && entry.system + .alert.alert-info(role='alert') + p.mb-0 + | This signature identifies an Apple system API rather than a third-party tracker. It is reported for completeness and is excluded from the jurisdiction analysis. + + if isTracker && !entry.company && !entry.system + .alert.alert-warning(role='alert') + p.mb-0 + | This tracker could not be matched to a company in the tracker database, so no jurisdiction is shown for it. + + if !isTracker && entry.trackers && entry.trackers.length > 0 + h5 Trackers operated by #{entry.name} + p + each tracker in entry.trackers + a.badge.badge-pill.badge-outline-primary.analytics.mr-1(href=`/tracker/${tracker.slug}`) + | #{tracker.name} (#{tracker.appCount}) + p.text-muted.small Numbers in brackets are the apps each tracker was detected in. + + if isTracker && exodus + p + a.link(target='_blank' rel='noopener noreferrer' href=`https://reports.exodus-privacy.eu.org/en/trackers/${exodus.id}/`) Tracker profile on Exodus Privacy + if exodus.categories && exodus.categories.length > 0 + | + each category in exodus.categories + span.badge.badge-pill.badge-outline-primary.analytics.ml-1 #{category} + + h3.mt-4 Apps + p.text-muted.small + if pagination.total > 0 + | Showing #{pagination.from}–#{pagination.to} of #{pagination.total} apps, most reviewed first. + else + | No apps to show. + + .container + each app in pagination.apps + .row.position-relative.mb-2 + .col-3.col-sm-2.col-md-2.my-auto + if app.icon + img.rounded(src=`${app.icon}` width='50' height='50' alt=`${app.title}`) + .col-9.col-sm-10.col-md-10.text-truncate.position-static + div + a.stretched-link.report-link(href=`/analysis/${app.appid}`)= app.title + div + span.badge.badge-pill.badge-danger.reports #{app.trackerCount} + | trackers + - const meta = jurisdictionMeta[app.classification] + if meta + span.badge.badge-pill.ml-1(class=meta.cssClass) !{meta.icon} #{meta.label} + .text-muted.small + if app.category + span #{app.category} + if app.analysed + | · analysed #{new Date(app.analysed).toLocaleDateString('en-GB', {day:'numeric', month:'short', year:'numeric'})} + + if pagination.totalPages > 1 + nav.mt-4(aria-label='App list pages') + ul.pagination.justify-content-center + li.page-item(class=pagination.page <= 1 ? 'disabled' : '') + a.page-link(href=`?page=${pagination.page - 1}` rel='prev') Previous + li.page-item.disabled + span.page-link Page #{pagination.page} of #{pagination.totalPages} + li.page-item(class=pagination.page >= pagination.totalPages ? 'disabled' : '') + a.page-link(href=`?page=${pagination.page + 1}` rel='next') Next + + p.text-muted.small.mt-4 + | Detection means the tracker's software is built into the app. It does not by itself prove that data was transmitted while the app was in use. Percentages are shares of the #{totalApps} apps analysed so far, which are not a random sample of the App Store — see #[a(href='/about#results') About] before quoting these figures. + if latestAnalysis + | Data as of #{new Date(latestAnalysis).toLocaleDateString('en-GB', {day:'numeric', month:'short', year:'numeric'})}. + + p.mt-3 + a.btn.btn-outline-primary(href=isTracker ? '/trackers' : '/companies') + | ← All #{isTracker ? 'trackers' : 'companies'} + +block app diff --git a/views/statistics.pug b/views/statistics.pug index 8756332..4e1f4dc 100644 --- a/views/statistics.pug +++ b/views/statistics.pug @@ -44,8 +44,16 @@ block content each tracker, index in topTrackersEnriched tr td= index + 1 - td.font-weight-bold= tracker.name - td= tracker.company + td.font-weight-bold + if tracker.slug + a.report-link(href=`/tracker/${tracker.slug}`)= tracker.name + else + = tracker.name + td + if tracker.companySlug + a.report-link(href=`/company/${tracker.companySlug}`)= tracker.company + else + = tracker.company td #{tracker.flag} #{tracker.countryName} td span.badge.badge-pill(class=`region-${tracker.region.toLowerCase()}`) #{tracker.region} @@ -70,7 +78,11 @@ block content each company, index in jurisdictionStats.topCompaniesSorted tr td= index + 1 - td.font-weight-bold= company.name + td.font-weight-bold + if company.slug + a.report-link(href=`/company/${company.slug}`)= company.name + else + = company.name td #{company.flag} #{company.countryName} td span.badge.badge-pill(class=`region-${company.region.toLowerCase()}`) #{company.region} @@ -130,10 +142,14 @@ block content else span.badge.badge-pill.jurisdiction-us None + p.mt-4 + a.btn.btn-outline-primary.mr-2(href='/trackers') Browse all trackers → + a.btn.btn-outline-primary(href='/companies') Browse all companies → + p.text-muted small | Based on Xray Tracker Database (#{xrayCompanyCount} tracker companies). - | - a(href='/about') About this analysis + | + a(href='/about#how-it-works') How these numbers are produced block app