Symptom
Every declarative stream paginated with DefaultPaginator + OffsetIncrement parses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.
Root cause
At airbyte-cdk 7.3.1, airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py (next_page_token, lines ~78-92):
decoded_response = next(self.decoder.decode(response))
if self.extractor:
page_size_from_response = len(list(self.extractor.extract_records(response=response)))
last_page_size = (
page_size_from_response if page_size_from_response is not None else last_page_size
)
self.extractor.extract_records(response=response) re-runs the full record extraction and materializes the whole page into a list just to take len().
- The wiring is unconditional for declarative sources:
model_to_component_factory.py passes extractor_model=model.record_selector.extractor into create_default_paginator >> create_offset_increment, so the paginator always holds a second instance of the stream's extractor.
- The incoming
last_page_size argument - already counted per emitted record by SimpleRetriever - is effectively always overwritten: len() never returns None, so the fallback branch is dead.
Impact
Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.
Reproduction
Any manifest with DefaultPaginator + OffsetIncrement; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once from next_page_token).
Suggestion
Count from what the retriever already extracted (it computes last_page_size per record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.
Precedent
Found while reviewing airbytehq/airbyte#83188 and airbytehq/airbyte#83318 (GA4 pagination fix: CursorPagination >> OffsetIncrement on 25k-100k row pages).
Symptom
Every declarative stream paginated with
DefaultPaginator+OffsetIncrementparses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.Root cause
At airbyte-cdk 7.3.1,
airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py(next_page_token, lines ~78-92):self.extractor.extract_records(response=response)re-runs the full record extraction and materializes the whole page into a list just to takelen().model_to_component_factory.pypassesextractor_model=model.record_selector.extractorintocreate_default_paginator>>create_offset_increment, so the paginator always holds a second instance of the stream's extractor.last_page_sizeargument - already counted per emitted record bySimpleRetriever- is effectively always overwritten:len()never returnsNone, so the fallback branch is dead.Impact
Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.
Reproduction
Any manifest with
DefaultPaginator+OffsetIncrement; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once fromnext_page_token).Suggestion
Count from what the retriever already extracted (it computes
last_page_sizeper record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.Precedent
Found while reviewing airbytehq/airbyte#83188 and airbytehq/airbyte#83318 (GA4 pagination fix:
CursorPagination>>OffsetIncrementon 25k-100k row pages).