Skip to content

Fix mixed sitemap lastmod parsing - #39

Open
ziodave wants to merge 1 commit into
mainfrom
codex/fix-mixed-sitemap-lastmod
Open

ziodave wants to merge 1 commit into
mainfrom
codex/fix-mixed-sitemap-lastmod

Conversation

@ziodave

@ziodave ziodave commented Sep 24, 2026

Copy link
Copy Markdown
Member

Why

LendingTree's sitemap legally mixes W3C date-only values such as 2026-09-22
with full timestamps such as 2024-10-24T12:20:57+00:00. The SDK parsed the
column without an explicit mixed-format policy and then assumed every value
supported .to_pydatetime(). This caused sitemap ingestion to fail before all
URLs were processed.

Failing production run:
https://github.com/graph-sync/graph-sync-lendingtree-com/actions/runs/36007252436

Sitemap protocol:
https://www.sitemaps.org/protocol.html

What changed

  • Parse sitemap lastmod values with format="mixed" so Pandas handles each
    W3C representation independently.
  • Normalize parsed values with utc=True, yielding consistent timezone-aware
    timestamps before conversion to Python datetimes.
  • Apply the same behavior to both SitemapSourceAdapter and the public legacy
    SitemapUrlSource.
  • Add focused regression tests using the exact LendingTree date formats.

Important files

  • wordlift_sdk/ingestion/sources.py
  • wordlift_sdk/url_source/sitemap_url_source.py
  • tests/ingestion/test_sources.py
  • tests/url_provider/test_sitemap_url_provider.py

Verification

  • poetry run python tests/tools/run_slice_tests.py ingestion -- -q
    • 132 passed, 1 skipped
  • Focused source tests
    • 9 passed, 1 skipped
  • Affected-module coverage
    • 95% combined
  • Ruff check and format check
    • passed
  • git diff --check
    • passed

Risks and follow-up

  • Date-only values now serialize as midnight UTC with an explicit +00:00
    offset. Offset timestamps are normalized to the equivalent UTC instant.
  • Invalid or missing lastmod values remain omitted or None.
  • A patch release and a rerun of LendingTree's production_all workflow are
    still required after merge.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant