Skip to content

refactor: generate the sync SDK from the async source with unasync - #189

Merged
marcusschiesser merged 5 commits into
mainfrom
marcusschiesser/unasync-reviewable
Oct 21, 2026
Merged

marcusschiesser merged 5 commits into
mainfrom
marcusschiesser/unasync-reviewable

Conversation

@marcusschiesser

@marcusschiesser marcusschiesser commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Same change as #186, rebuilt from main as five commits that git and GitHub can follow, so the review is readable commit by commit. Since #186, commit 2 has been rebuilt so that helper functions stay in the _async/ module they came from. Only 6 modules keep a _<module>_common.py (see below), not 17.

Commits

  1. 02118eb refactor: move the mixed sync/async modules under _async/: pure git mv of the 25 mixed modules to introspection_sdk/_async/, no content change. GitHub shows every file as renamed with 0 changes; nothing to read. (Unchanged SHA.)
  2. 0f484bc refactor(_async): keep only the async twins: read this one closely. It deletes the sync classes from each moved module (+1,088 / −5,475 over 35 files). For 11 of the modules (tags, members, repositories, connectors, shares, tasks, files, connections, automations, _reads, proxy/_transport) the change is only deletions, plus the one-line module docstring (Re-exported by :mod:…``, which reads right in both twins) and the imports. It also renames _acp_post / `_apost_token_form` to `_cp_post_async` / `_post_token_form_async` and makes `_http` sleep through a hand-written `_sleep` / `_sleep_async` pair in `_backoff.py`, so the generator needs no special rule for either. The helpers stay where they were, and unasync copies them into `_sync/`. Because of commit 1, every file shows as an edit of the old module.
  3. 0eea1c3 refactor: re-export both twins from the public modules: the stubs at the old paths that re-export both twins and __all__.
  4. f1d15f7 build(unasync): generate the sync twins into _sync/ with scripts/gen_sync.py: read scripts/gen_sync.py (102 lines); skim _sync/. It is generated, and CI regenerates it and fails on any difference. Also adds the ruff per-file-ignores for _sync/ (UP028, C416).
  5. c5aebfe ci: check the generated sync code and re-exported API in CI: CI steps, the griffe extension, jscpd threshold, AGENTS.md, pre-commit, deptry, uv.lock.

Why 6 _common modules remain

Some objects have to be one object that both twins use, not two copies. A twin can't import them from its public module, because that module imports the twin. These are the only things the 6 modules hold:

Module Shared object Why it can't be copied
_auth_common.py OAuthToken, AuthSession, SignInSupersededError, _EmailCodeAuthBase Public classes that both twins return or raise (one class for isinstance / except). _EmailCodeAuthBase is the base of the hand-written EmailCodeAuth / AsyncEmailCodeAuth.
_http_common.py RawResponse Returned by both HTTP clients; public.
runner_resources/_events_common.py UnknownEventCounter, UNKNOWN_EVENT_SKIPS Process-wide skip counter: a copy per twin would split the count.
resources/_runtimes_common.py RuntimeHandle, AsyncRuntimeHandle Hand-written (runtime_id is a property against a coroutine), and both generated Runtimes classes return them.
runner_resources/_conversations_common.py ConversationExportParams Public TypedDict that both twins and both protocols take.
_streaming_common.py _SseEvent, _SseAccumulator The SSE parse state both stream parsers share.

Besides those classes, the modules keep only the private helpers the classes themselves call: _jwt_claims, _claim, _resolve_base_api_url, _utcnow, SessionListener, _is_uuid and the events _logger. _is_rejection, _rejected and _not_signed_in are used only by the hand-written EmailCodeAuth twins, so they moved into the public auth.py next to them. Every other helper and constant now has an async copy and a generated sync copy; the public modules re-export the async copy under the old name. Private names that tests import (_http._clean_params, files._materialise_upload, …) are still re-exported.

Commits 1 to 4 were made with --no-verify and don't import or pass tests on their own (commit 1 leaves the public modules missing until commit 3). Commit 5 is the full tree. On it I ran prek run --all-files (every hook passes, including ty and gen-sync), ruff format --check, ruff check, ty check, deptry, pytest (774 passed, coverage 92.4%), scripts/gen_sync.py + git diff --exit-code, griffe check --against v0.29.0 -e scripts/griffe_reexports.py (exit 0, and exit 0 against v0.29.1) and jscpd (16.22%). Merge with squash as usual. This replaces #186; close that one once this is merged.

Coverage note: total coverage is 92.36%, against 93.94% on the previous head and 93.6% on main. The fail_under = 80 ratchet holds and no test changed. The drop comes from the copied helpers: a test that calls a helper directly through its public name (parse_lookback, decode_arrow_page, _materialise_upload, …) covers the async copy, while the sync copy is covered only through the sync methods that call it. The same applies the other way round.


The description and evidence below are from #186 and still apply to this tree, except where a section says it was updated.

Every class or function with an Async* twin is now written once, as async code under introspection_sdk/_async/. scripts/gen_sync.py generates the sync twin into introspection_sdk/_sync/ with unasync, pinned to unasync==0.6.0 in the dev extra. httpx/httpcore and urllib3 work the same way.

Layout

  • introspection_sdk/_async/<path>.py: the hand-written async twins, the single source.
  • introspection_sdk/_sync/<path>.py: generated. Each file opens with a # Generated by scripts/gen_sync.py from … Do not edit header.
  • introspection_sdk/<path>.py (the public module): re-exports both twins (and __all__), so every import path and public name stays as it was. The few objects both twins must share as one object live in a sibling _<module>_common.py (6 modules, see above). Private names that tests import from the public modules (_http._HttpClient, _clean_params, files._materialise_upload, streaming._parse_sse, …) are still re-exported.
  • Generated: protocols, client, runner, _http, auth (token functions, CredentialProvider), streaming, proxy/{_transport,httpx,httpx2}, resources/{connectors,experiments,members,recipes,repositories,runtimes,tags}, runner_resources/{_reads,automations,connections,conversations,events,files,metrics,shares,tasks}. That is 55 sync classes and functions, about 2,800 generated lines.

The generator (scripts/gen_sync.py, updated for c5aebfe)

  • unasync's token pass drops async and await. Async* and _Async* lose the prefix, and async_* and *_async lose the affix. An explicit TOKENS map covers the rest: AsyncClient→Client, AsyncBaseTransport→BaseTransport, AsyncHTTPTransport→HTTPTransport, AsyncByteStream→SyncByteStream, aclose→close, aiter_bytes/aiter_lines/aiter_raw/aread, handle_async_request→handle_request, _async→_sync, plus unasync's defaults (AsyncIterator→Iterator, __aenter__→__enter__, …).
  • SUBSTITUTIONS handles what isn't a single token: Awaitable[X] becomes X, prose cross-references like :class:`AsyncPager` lose the prefix, Asynchronous becomes Synchronous, and an # unasync: async-only … # unasync: end fence marks a block the sync twin doesn't have. Today that fence covers the async client's __aenter__/__aexit__ (the sync client is no context manager) and 35 async docstrings that say something the sync one doesn't.
  • The script deletes introspection_sdk/_sync/ and generates into it in place. Then ruff runs once over the folder with the repo config: ruff check --fix --select I,F401 (sorts imports, drops the ones only the async code used; no --unsafe-fixes), then ruff format. The for x in y: yield x and [x for x in y] that async for leaves stay literal; pyproject.toml turns UP028 and C416 off for introspection_sdk/_sync/**.
  • There is no --check mode: the Lint & Type Check job and the gen-sync pre-commit hook run the script, then fail on git diff --exit-code or any untracked file under introspection_sdk/_sync.
  • deptry gets a DEP004 ignore scoped to unasync, with a reason; the import is a dev-only generator.

Not generated (hand-written, with reason)

Code Why
pagination.py: Pager / AsyncPager (+ cursor_paginate pair) Pager delegates attributes to the first page (__getattr__), AsyncPager is awaitable (__await__). The members differ, not just the tokens.
resumable.py: stream_resumable / stream_resumable_async tests/rest/test_resumable.py patches introspection_sdk.resumable._retry_delay. A generated _sync/resumable.py would look the name up in its own globals, so the patch would miss. Tests may not change.
auth.py: EmailCodeAuth / AsyncEmailCodeAuth The sync twin guards its state with threading locks (with self._state:) that the async twin, on one event loop, doesn't take.
resources/annotations.py: Annotations / AsyncAnnotations list() resolves email filters eagerly in sync (errors raise at call time) and lazily on the first fetch in async (a sync list() can't await). Generating it would change sync behaviour.
resources/runtimes.py: RuntimeHandle / AsyncRuntimeHandle runtime_id is a property on one and a coroutine on the other. Runtimes / AsyncRuntimes are generated.
proxy/httpx.py vs proxy/httpx2.py These two are copies across libraries (httpx vs httpx2), not async/sync twins. Each file's own sync class is generated from its async class, but the two async sources stay near-copies.

Public API

griffe check (uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py, as api-compat.yml now runs it): exit 0, no breakages, and no api-break label or ! needed.

⚠️ Before c637ad2 that result was weaker than it looked (see "API check: re-exported classes" below for the fix). When griffe compares an old class against a new re-export, it returns at its seen_paths guard before reaching the members (_type_based_yield → _alias_incompatibilities → _type_based_yield on the same old path). So every moved class passes without its members being compared. I checked this: changing a default in _async/runner_resources/shares.py still gave exit 0 against main. From the next release on, both sides are re-exports and the check catches changes again; I verified that by running --against HEAD with the same injected change, which griffe then reports. AGENTS.md now notes this. To cover this PR, I applied griffe's own breakage rules to the resolved targets of every public object (resolving aliases on both sides, with a fresh seen_paths per object). Against v0.29.0 and against main (86e42ba): 0 breakages. The same harness reports the injected default change, so it isn't vacuous.

griffe dump comparison (public objects resolved through aliases; 64 modules, 514 classes, 962 functions, 2,686 attributes compared, against both main and v0.29.0):

  • Public objects removed: 0. Added: 0.
  • Kind, signature (parameter names, kinds, annotations, defaults), return annotation, decorators, bases, attribute values and annotations: 0 differences.
  • Sync docstrings: 0 differences. Every generated sync class and function is also AST-identical to the hand-written one on main, docstrings included (checked per object). Comments match too, apart from the generated-file header. Re-checked on cd7ecd3: 182 sync top-level objects, helpers included, are AST-identical to main. The one exception is UnknownEventCounter, which logs through the shared _logger (same logger name) instead of the module-level logger. Since c5aebfe, four statements differ from main by construction: two for chunk in …: yield chunk loops where main has yield from, one [table for table in self] where main has list(self), and _sleep(delay) where main has time.sleep(delay). Docstrings are unchanged, and the private _async/ and _sync/ module docstrings now read the same in both trees.
  • Async docstrings: 44 changed, all on objects whose old docstring was a one-line stub such as Async twin of :meth:`Members.update`., or that had no docstring. They now carry the full text of the sync docstring. The 47 async docstrings with real content are byte-identical: 12 because they map to the sync text by prose substitution, 35 through the async-only fence. The 44 changed docstrings:
  • auth.async_authorization_code_token
  • auth.async_email_code_token
  • auth.async_refresh_access_token
  • auth.async_revoke_session
  • auth.async_send_email_code
  • auth.async_service_account_token
  • auth.async_token_exchange
  • auth.AsyncCredentialProvider
  • auth.AsyncCredentialProvider.authorization
  • auth.AsyncCredentialProvider.refresh_after_unauthorized
  • resources.connectors.AsyncConnectors
  • resources.members.AsyncMembers
  • resources.members.AsyncMembers.create
  • resources.members.AsyncMembers.update
  • resources.recipes.AsyncRecipes
  • resources.repositories.AsyncRepositories
  • resources.repositories.AsyncRepositories.commit
  • resources.repositories.AsyncRepositories.commits
  • resources.repositories.AsyncRepositories.list
  • resources.repositories.AsyncRepositoryContents
  • resources.repositories.AsyncRepositoryMerges
  • resources.repositories.AsyncRepositoryMerges.create
  • runner_resources.automations.AsyncAutomations
  • runner_resources.automations.AsyncAutomations.create
  • runner_resources.automations.AsyncAutomations.delete
  • runner_resources.automations.AsyncAutomations.get
  • runner_resources.automations.AsyncAutomations.trigger
  • runner_resources.automations.AsyncAutomations.update
  • runner_resources.connections.AsyncAppConnections
  • runner_resources.connections.AsyncAppConnections.create
  • runner_resources.connections.AsyncAppConnections.delete
  • runner_resources.connections.AsyncAppConnections.get
  • runner_resources.connections.AsyncAppConnections.list
  • runner_resources.connections.AsyncRunnerAppConnections
  • runner_resources.connections.AsyncRunnerAppConnections.create
  • runner_resources.conversations.AsyncConversations.export_arrow
  • runner_resources.conversations.AsyncConversations.export_json
  • runner_resources.conversations.AsyncConversations.export_stream
  • runner_resources.conversations.AsyncConversations.export_trajectory
  • runner_resources.events.AsyncEvents
  • runner_resources.events.AsyncEvents.get
  • runner_resources.metrics.AsyncMetrics
  • runner_resources.shares.AsyncShares.update
  • streaming.parse_ag_ui_events_async
  • 19 aliases to ag_ui.core don't resolve on either side (external package, unchanged).

Runtime-visible change outside griffe: the moved classes' __module__ is now introspection_sdk._sync.… / introspection_sdk._async.… (for example introspection_sdk.Runner.__module__). Import paths, isinstance and pickling-by-reference through the public paths are unaffected. The events module logger keeps its name (introspection_sdk.runner_resources.events).

Duplicate code (jscpd)

introspection_sdk/_sync/** goes in .jscpd.json ignore; it repeats _async/ by construction.

Duplicated lines Total lines Share
main (baseline) 5,788 19,813 29.2% (exact-only 15.8%)
previous head (17 _common modules) 2,819 16,951 16.63%
this PR (6 _common modules) 2,702 16,661 16.22%

threshold goes from 29.3 to 16.3 (updated for cd7ecd3).

Checks run locally

  • uv run pytest tests/: 774 passed, no test edits. Coverage 92.4% on c5aebfe (92.36% on cd7ecd3) (main 93.6%; see the coverage note above); --cov-fail-under=80 holds. The fenced async docstrings count as statements, which adds 3 uncovered lines in async methods no test calls.
  • ruff format --check, ruff check (incl. the complexity ratchet), ty check (pinned), mypy, deptry, jscpd (exit 0), actionlint, check_examples.py, scripts/gen_sync.py + git diff --exit-code: all pass.
  • scripts/check_api_contract.py (scheduled, not a PR gate): output is byte-identical to main's.
  • prek run --all-files: every hook passes on c5aebfe (and on cd7ecd3), the local ty hook included.

API check: re-exported classes

griffe check passed vacuously for every class this PR moves behind a re-export: _type_based_yield adds the old object's path to seen_paths, goes into _alias_incompatibilities, and recurses into the alias's target with that same old path, which returns at once. griffe has no CLI option for this, and griffe 2.4.0 (latest) has the same diff.py, so an upgrade doesn't help.

The fix stays on the plain CLI and uses griffe's extension API instead of a custom checker: scripts/griffe_reexports.py is a ~20-line griffe.Extension whose on_package hook replaces each public alias whose target lives in introspection_sdk with the target object. It runs on the old release and the new code alike (griffe loads both with the same extensions), so both sides hold real objects and griffe's own breakage rules compare them. api-compat.yml adds -e scripts/griffe_reexports.py; the api-break label and ! title override is unchanged. AGENTS.md documents the flag. pyproject.toml widens the scoped deptry DEP004 ignore to griffe (dev-only import in scripts/).

Evidence (the breaking change was applied locally only, never committed): remove name from AsyncRecipes.list in _async/resources/recipes.py and regenerate _sync/.

Clean HEAD (cd7ecd3; also exit 0 on c5aebfe):

$ uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py
exit=0

With the breaking change, the old command vs the new one:

$ uv run griffe check introspection_sdk --search . --against v0.29.0
exit=0

$ uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py
introspection_sdk/_async/resources/recipes.py:25: AsyncRecipes.list(name): Parameter was removed
introspection_sdk/_sync/resources/recipes.py:27: Recipes.list(name): Parameter was removed
exit=1

Re-run on cd7ecd3 and on c5aebfe with the same change in _async/ only (no regeneration): introspection_sdk/_async/resources/recipes.py:25: AsyncRecipes.list(name): Parameter was removed, exit 1.

Docs

AGENTS.md gains a "Sync code generation" section (workflow, layout, token map, fence, the hand-written list, adding a twin). The "Duplicate code" section now mentions the _sync/ exclusion, and "Public API compatibility" describes the griffe extension below.

🤖 Generated with Claude Code

Pure git mv of each module that holds both an Async* class and its sync
twin to its path under introspection_sdk/_async/, with no content
change, so the next commits show as edits of the moved files. The tree
does not import at this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T14:01:11.419294Z c5aebfe Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@marcusschiesser

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: cd7ecd38c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

marcusschiesser and others added 4 commits October 21, 2026 20:54
In each moved module, delete the sync classes and functions so the file
holds the async source alone; the helper functions and constants stay
where they were, and unasync copies them into the sync twin. The async
docstrings carry the full text.

Six modules get a sibling _<module>_common.py for what both twins must
share as one object, which cannot live in the public module that imports
the twins: auth (OAuthToken, AuthSession, SignInSupersededError,
_EmailCodeAuthBase), _http (RawResponse), events (UnknownEventCounter,
UNKNOWN_EVENT_SKIPS), runtimes (the hand-written RuntimeHandle /
AsyncRuntimeHandle), conversations (ConversationExportParams) and
streaming (_SseEvent, _SseAccumulator), plus the private helpers those
classes call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore each public module at its old path as a stub that re-exports
the async twin from _async/ and the sync twin from _sync/ (and
__all__), so every import path and public name stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sync.py

Add scripts/gen_sync.py, which deletes introspection_sdk/_sync/, writes
the sync twin of each _async/ module into it with unasync plus a token
map and a few substitutions, then runs ruff over the folder, and check in
its output. Every file there is generated. ruff's UP028 and C416 are off
for _sync/, so the loops an `async for` leaves stay as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run scripts/gen_sync.py in the Lint & Type Check job and as a pre-commit
hook and fail on any diff or untracked file under introspection_sdk/_sync/,
load scripts/griffe_reexports.py in the API compatibility check so
re-exported classes are compared member by member, pin unasync in the dev
extra (deptry DEP004 ignore scoped to it), lower the jscpd threshold to
16.3 for the new layout, and document the sync code generation in
AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marcusschiesser

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: c5aebfed73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@marcusschiesser
marcusschiesser merged commit bae82a5 into main Oct 21, 2026
10 checks passed
@marcusschiesser
marcusschiesser deleted the marcusschiesser/unasync-reviewable branch October 21, 2026 14:05

This branch was successfully deployed

1 active deployment
build — c5aebfed Deployed Oct 21, 2026 by marcusschiesser via Unit Tests #515
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant