Improve crawl budget and path geo hubs

[kargig]

Summary

Google Search Console showed roughly 45 indexed URLs vs ~2600
discovered-not-indexed. This PR shrinks low-value sitemap noise,
adds indexable path-based dive-site geo hubs, prerenders /map,
and tightens breadcrumbs / curated-list inclusion so crawl budget
goes to pages that can actually rank.

Changes Made

Sitemap and crawl budget

• Exclude thin public dive logs from the sitemap (notes/media/profile
  quality gates via query_substantial_public_dives)
• Exclude empty or thin curated lists unless they have at least 3
  dive sites (MIN_SITEMAP_LIST_ITEMS, query_substantial_public_lists)
• Keep query-param filter URLs out of the sitemap; canonicalize geo
  filters to path hubs instead

Path geo hubs

• Add indexable /dive-sites/{country} and
  /dive-sites/{country}/{region} routes that reuse the existing
  Dive Sites list/search UI (DiveSitePathGate, geoHubs.js)
• Sync path segments ↔ country/region filters with a hydration gate
  so hubs do not snap back to /dive-sites after remount
• Return {region, country} from GET /api/v1/dive-sites/regions
  so a region-only selection can set both filters and build the
  correct hub URL
• Prerender geo hubs and /map in the SEO router; include map in
  nginx prerender paths

Breadcrumbs and labeling

• Align UI crumbs and JSON-LD with navbar names (Dive Sites, Dive Log,
  Dive Routes, etc.)
• Add current-page crumb on site, dive, route, center, and trip detail
• Prefer site hierarchy for route breadcrumbs when a dive site is known

Docs

• Design + implementation plan under docs/superpowers/

Breaking changes

• GET /api/v1/dive-sites/regions response shape changed from
  string[] to [{ region, country }]. Frontend consumers in this
  PR are updated; any external client of that endpoint must adapt.

Testing

Automated

• New/extended coverage in backend/tests/test_seo_geo.py (substantial
  dives/lists, geo helpers)
• Updates in test_seo_router.py (geo hub + map prerender, Dive Log
  labeling)
• Updates in test_dive_sites.py for regions response schema
• Run via isolated Docker only:
  cd backend && ./docker-test-github-actions.sh tests/test_seo_geo.py tests/test_seo_router.py tests/test_dive_sites.py

Manual

• Country/region filter on Dive Sites: URL becomes path hub and
  survives remount (no snap-back to /dive-sites)
• Region-only pick sets country + region and updates the hub path
• Breadcrumbs on Dive Site, Dive Log detail, Route, Center, Trip
  match navbar labels and include the current page
• After regenerating static content, sitemap list URLs should only
  include public profile lists with ≥3 sites
  (local sitemap.xml is stale until regenerated)

Related Issues

• Context: GSC Coverage export 2026-09-25 for divemap.blue
  (~45 indexed / ~2606 not indexed, mostly Discovered – currently
  not indexed)
• Design: docs/superpowers/specs/2026-09-25-seo-indexing-fixes-design.md
• Plan: docs/superpowers/plans/2026-09-25-seo-indexing-fixes.md

Additional Notes

Deployment

• Regenerating llm_content/sitemap.xml (static content job / R2
  publish) is required for thin-list and thin-dive exclusions to
  take effect in production
• Nginx prerender path updates need deploy for /map and geo hubs

Follow-up

• Regenerate local/prod sitemap so live XML matches the new list
  filter (validated: localhost still listed empty/thin favorites)
• Optional Chrome DevTools pass on search behavior for /dive-sites,
  country hub, and region hub (called out in the plan)

Notes for reviewers

• Two commits on this branch: crawl-budget + geo hubs, then geo sync /
  crumbs / thin-list sitemap filter
• Geo hubs intentionally keep in-app ? filters for UX while
  canonicalizing hub URLs to paths to avoid duplicate indexing

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit f0ebdde)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Issue The geo-hub branch is only reached when parts[1] is not a digit, but the elif parts[1].isdigit() guard means a path like /dive-sites/greece/ (trailing slash) or a country slug containing digits (e.g. a region/country name with a number) would be misrouted. More concretely, parts[1].isdigit() is checked before the geo-hub else, so any country slug that is purely numeric would be treated as a dive-site id. This is unlikely for real country names but the routing relies on the assumption that no country/region slug is all digits; if such a value exists it will 404 as an invalid site id instead of rendering the hub. elif parts[1].isdigit(): # Dive Site Detail (numeric id) try: site_id = int(parts[1]) except ValueError: raise HTTPException(status_code=404, detail="Invalid Dive Site ID") site = ( db.query(DiveSite) .filter(DiveSite.id == site_id, DiveSite.status == "approved", DiveSite.deleted_at.is_(None)) .options(joinedload(DiveSite.difficulty), joinedload(DiveSite.ratings), joinedload(DiveSite.media)) .first() ) if not site: raise HTTPException(status_code=404, detail="Dive Site not found") # Extract a random photo media URL if available (combining both direct site photos and public dive log photos at this site) from app.models import SiteMedia, DiveMedia site_photos = db.query(SiteMedia).filter( SiteMedia.dive_site_id == site.id, SiteMedia.media_type == MediaType.photo ).all() dive_photos = db.query(DiveMedia).join(Dive).filter( Dive.dive_site_id == site.id, Dive.is_private == False, DiveMedia.media_type == MediaType.photo ).all() all_photos = site_photos + dive_photos if all_photos: photo_media = random.choice(all_photos) # Order of preference: thumbnail, if it doesn't exist then medium, if it doesn't exist then original if photo_media.thumbnail_url: preview_path = photo_media.thumbnail_url image_width = 400 image_height = 400 elif photo_media.medium_url: preview_path = photo_media.medium_url image_width = 1200 image_height = 1200 else: preview_path = photo_media.url image_width = None image_height = None image_url = resolve_seo_image_url(preview_path, base_url) image_type = get_image_mime_type(image_url) slug = get_dive_site_slug(site) # Check for mismatched/missing slug and return 301 Redirect for canonicalization requested_slug = parts[2] if len(parts) >= 3 else "" if requested_slug != slug: redirect_path = f"/dive-sites/{site.id}/{slug}" if slug else f"/dive-sites/{site.id}" return RedirectResponse(url=f"{base_url}{redirect_path}", status_code=301) detail_path = f"/dive-sites/{site.id}/{slug}" if slug else f"/dive-sites/{site.id}" avg, total = _site_rating_stats(site) location_parts = [site.region, site.country] location_suffix = ", ".join(filter(None, location_parts)) page_title = f"Divemap - {site.name}" if location_suffix: page_title += f" - {location_suffix}" main_content = render_dive_site_main(site, avg, total) description = dive_site_meta_description(site, avg, total) json_ld = dive_site_schema(base_url, detail_path, site, avg, total) canonical = f"{base_url}{detail_path}" else: # Geo hub: /dive-sites/{country-slug}[/region-slug] countries = distinct_approved_countries(db) country = resolve_label_from_slug(countries, parts[1]) if not country: raise HTTPException(status_code=404, detail="Country not found") Possible Issue The legacy query-param migration effect runs whenever location.pathname === '/dive-sites' and a country query param is present, and it navigates to the path hub. However getInitialFilters still reads country/region from the query string on mount, so on the first render the filters are populated from the query params and the debounced/immediate URL effects may fire before the migration redirect completes, potentially issuing a redundant navigation or briefly rendering with stale filters. The hydration gate only guards the path-hub case (geoCountrySlug), not this query-param case. useEffect(() => { if (geoCountrySlug) return; if (location.pathname !== '/dive-sites') return; const qCountry = searchParams.get('country'); if (!qCountry) return; const qRegion = searchParams.get('region') || ''; const params = new URLSearchParams(searchParams); params.delete('country'); params.delete('region'); const qs = params.toString(); const path = buildGeoHubPath(qCountry, qRegion); navigate(qs ? `${path}?${qs}` : path, { replace: true }); }, [geoCountrySlug, location.pathname, searchParams, navigate]); Performance query_substantial_public_dives uses joinedload(Dive.media) together with a correlated EXISTS subquery on DiveMedia. For dives with many media rows this can produce a large cartesian result set during sitemap generation. Since only the existence of media matters for the filter, consider selectinload or omitting the media eager load if the caller does not need the media objects, to avoid loading all media rows for every substantial dive. def query_substantial_public_dives(db: Session) -> list[Dive]: """Public dives from enabled users that meet substance criteria (for sitemap). Substance filters run in SQL so sitemap generation does not load every public dive (plus media) into memory. """ has_media = ( db.query(DiveMedia.id) .filter(DiveMedia.dive_id == Dive.id) .exists() ) # Match is_substantial_public_dive: profile OR notes >= 100 chars OR media. # CHAR_LENGTH matches Python len() on Unicode text under MySQL. substantial = or_( Dive.profile_xml_path.isnot(None), Dive.profile_sample_count > 0, func.char_length(func.trim(Dive.dive_information)) >= 100, has_media, ) return ( db.query(Dive) .options(joinedload(Dive.media), joinedload(Dive.user), joinedload(Dive.dive_site)) .join(User, Dive.user_id == User.id) .filter( Dive.is_private == False, # noqa: E712 User.enabled == True, # noqa: E712 substantial, ) .all() )

---

⚠️ Review coverage: The following files were not included in this review because of the token budget:

• frontend/src/pages/TripDetail.jsx

[kargig]

/review

[github-actions]

Persistent review updated to latest commit f0ebdde