Skip to content

OffsetIncrement.next_page_token re-extracts the full page to count records, discarding the retriever's last_page_size #1136

Description

@bazarnov

Symptom

Every declarative stream paginated with DefaultPaginator + OffsetIncrement parses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.

Root cause

At airbyte-cdk 7.3.1, airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py (next_page_token, lines ~78-92):

decoded_response = next(self.decoder.decode(response))

if self.extractor:
    page_size_from_response = len(list(self.extractor.extract_records(response=response)))
    last_page_size = (
        page_size_from_response if page_size_from_response is not None else last_page_size
    )
  • self.extractor.extract_records(response=response) re-runs the full record extraction and materializes the whole page into a list just to take len().
  • The wiring is unconditional for declarative sources: model_to_component_factory.py passes extractor_model=model.record_selector.extractor into create_default_paginator >> create_offset_increment, so the paginator always holds a second instance of the stream's extractor.
  • The incoming last_page_size argument - already counted per emitted record by SimpleRetriever - is effectively always overwritten: len() never returns None, so the fallback branch is dead.

Impact

Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.

Reproduction

Any manifest with DefaultPaginator + OffsetIncrement; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once from next_page_token).

Suggestion

Count from what the retriever already extracted (it computes last_page_size per record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.

Precedent

Found while reviewing airbytehq/airbyte#83188 and airbytehq/airbyte#83318 (GA4 pagination fix: CursorPagination >> OffsetIncrement on 25k-100k row pages).

Metadata

Metadata

Assignees

No one assigned

    Labels

    communityPRs and issues from community contributors

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions