Skip to content

refactor: generate the sync SDK from the async source with unasync - #186

Closed
marcusschiesser wants to merge 3 commits into
mainfrom
marcusschiesser/unasync-sync-codegen
Closed

marcusschiesser wants to merge 3 commits into
mainfrom
marcusschiesser/unasync-sync-codegen

Conversation

@marcusschiesser

@marcusschiesser marcusschiesser commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Every class or function with an Async* twin is now written once, as async code under introspection_sdk/_async/. scripts/unasync.py generates the sync twin into introspection_sdk/_sync/ with unasync, pinned to unasync==0.6.0 in the dev extra. httpx/httpcore and urllib3 work the same way.

Layout

  • introspection_sdk/_async/<path>.py: the hand-written async twins, the single source.
  • introspection_sdk/_sync/<path>.py: generated. Each file opens with a # Generated by scripts/unasync.py from … Do not edit header.
  • introspection_sdk/<path>.py (the public module): re-exports both twins (and __all__), so every import path and public name stays as it was. Code both twins share moves to a sibling _<module>_common.py, because a twin can't import its own public module, which imports the twin. Private names that tests import from the public modules (_http._HttpClient, _clean_params, files._materialise_upload, streaming._parse_sse, …) are still re-exported.
  • Generated: protocols, client, runner, _http, auth (token functions, CredentialProvider), streaming, proxy/{_transport,httpx,httpx2}, resources/{connectors,experiments,members,recipes,repositories,runtimes,tags}, runner_resources/{_reads,automations,connections,conversations,events,files,metrics,shares,tasks}. That is 55 sync classes and functions, about 2,800 generated lines.

The generator (scripts/unasync.py)

  • unasync's token pass drops async and await. Async* and _Async* lose the prefix, and async_* and *_async lose the affix. An explicit TOKENS map covers the rest: AsyncClient→Client, AsyncBaseTransport→BaseTransport, AsyncHTTPTransport→HTTPTransport, AsyncByteStream→SyncByteStream, aclose→close, aiter_bytes/aiter_lines/aiter_raw/aread, handle_async_request→handle_request, _async→_sync, _acp_post→_cp_post, _apost_token_form→_post_token_form, plus unasync's defaults (AsyncIterator→Iterator, __aenter__→__enter__, …).
  • SUBSTITUTIONS handles what isn't a single token: import asyncio/asyncio.sleep( become time/time.sleep(, Awaitable[X] becomes X, prose cross-references like :class:`AsyncPager` lose the prefix, Asynchronous becomes Synchronous, and an # unasync: async-only … # unasync: end fence marks a block the sync twin doesn't have. Today that fence covers the async client's __aenter__/__aexit__ (the sync client is no context manager) and 35 async docstrings that say something the sync one doesn't.
  • After that, ruff runs with the repo config: it sorts imports, drops imports only the async code used, and applies UP028/C416 to turn the for x in y: yield x and [x for x in y] left by async for into yield from and list(). Then it formats.
  • --check regenerates in memory and fails with a diff. It's a new step in the Lint & Type Check job and a new unasync pre-commit hook.
  • The script shares its name with the library, so it drops its own directory from sys.path before import unasync. deptry gets a DEP004 ignore scoped to unasync, with a reason; the import is a dev-only generator.

Not generated (hand-written, with reason)

Code Why
pagination.py: Pager / AsyncPager (+ cursor_paginate pair) Pager delegates attributes to the first page (__getattr__), AsyncPager is awaitable (__await__). The members differ, not just the tokens.
resumable.py: stream_resumable / stream_resumable_async tests/rest/test_resumable.py patches introspection_sdk.resumable._retry_delay. A generated _sync/resumable.py would look the name up in its own globals, so the patch would miss. Tests may not change.
auth.py: EmailCodeAuth / AsyncEmailCodeAuth The sync twin guards its state with threading locks (with self._state:) that the async twin, on one event loop, doesn't take.
resources/annotations.py: Annotations / AsyncAnnotations list() resolves email filters eagerly in sync (errors raise at call time) and lazily on the first fetch in async (a sync list() can't await). Generating it would change sync behaviour.
resources/runtimes.py: RuntimeHandle / AsyncRuntimeHandle runtime_id is a property on one and a coroutine on the other. Runtimes / AsyncRuntimes are generated.
proxy/httpx.py vs proxy/httpx2.py These two are copies across libraries (httpx vs httpx2), not async/sync twins. Each file's own sync class is generated from its async class, but the two async sources stay near-copies.

Public API

griffe check (uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py, as api-compat.yml now runs it): exit 0, no breakages, and no api-break label or ! needed.

⚠️ Before c637ad2 that result was weaker than it looked (see "API check: re-exported classes" below for the fix). When griffe compares an old class against a new re-export, it returns at its seen_paths guard before reaching the members (_type_based_yield → _alias_incompatibilities → _type_based_yield on the same old path). So every moved class passes without its members being compared. I checked this: changing a default in _async/runner_resources/shares.py still gave exit 0 against main. From the next release on, both sides are re-exports and the check catches changes again; I verified that by running --against HEAD with the same injected change, which griffe then reports. AGENTS.md now notes this. To cover this PR, I applied griffe's own breakage rules to the resolved targets of every public object (resolving aliases on both sides, with a fresh seen_paths per object). Against v0.29.0 and against main (86e42ba): 0 breakages. The same harness reports the injected default change, so it isn't vacuous.

griffe dump comparison (public objects resolved through aliases; 64 modules, 514 classes, 962 functions, 2,686 attributes compared, against both main and v0.29.0):

  • Public objects removed: 0. Added: 0.
  • Kind, signature (parameter names, kinds, annotations, defaults), return annotation, decorators, bases, attribute values and annotations: 0 differences.
  • Sync docstrings: 0 differences. Every generated sync class and function is also AST-identical to the hand-written one on main, docstrings included (checked per object). Comments match too, apart from the generated-file header.
  • Async docstrings: 44 changed, all on objects whose old docstring was a one-line stub such as Async twin of :meth:`Members.update`., or that had no docstring. They now carry the full text of the sync docstring. The 47 async docstrings with real content are byte-identical: 12 because they map to the sync text by prose substitution, 35 through the async-only fence. The 44 changed docstrings:
  • auth.async_authorization_code_token
  • auth.async_email_code_token
  • auth.async_refresh_access_token
  • auth.async_revoke_session
  • auth.async_send_email_code
  • auth.async_service_account_token
  • auth.async_token_exchange
  • auth.AsyncCredentialProvider
  • auth.AsyncCredentialProvider.authorization
  • auth.AsyncCredentialProvider.refresh_after_unauthorized
  • resources.connectors.AsyncConnectors
  • resources.members.AsyncMembers
  • resources.members.AsyncMembers.create
  • resources.members.AsyncMembers.update
  • resources.recipes.AsyncRecipes
  • resources.repositories.AsyncRepositories
  • resources.repositories.AsyncRepositories.commit
  • resources.repositories.AsyncRepositories.commits
  • resources.repositories.AsyncRepositories.list
  • resources.repositories.AsyncRepositoryContents
  • resources.repositories.AsyncRepositoryMerges
  • resources.repositories.AsyncRepositoryMerges.create
  • runner_resources.automations.AsyncAutomations
  • runner_resources.automations.AsyncAutomations.create
  • runner_resources.automations.AsyncAutomations.delete
  • runner_resources.automations.AsyncAutomations.get
  • runner_resources.automations.AsyncAutomations.trigger
  • runner_resources.automations.AsyncAutomations.update
  • runner_resources.connections.AsyncAppConnections
  • runner_resources.connections.AsyncAppConnections.create
  • runner_resources.connections.AsyncAppConnections.delete
  • runner_resources.connections.AsyncAppConnections.get
  • runner_resources.connections.AsyncAppConnections.list
  • runner_resources.connections.AsyncRunnerAppConnections
  • runner_resources.connections.AsyncRunnerAppConnections.create
  • runner_resources.conversations.AsyncConversations.export_arrow
  • runner_resources.conversations.AsyncConversations.export_json
  • runner_resources.conversations.AsyncConversations.export_stream
  • runner_resources.conversations.AsyncConversations.export_trajectory
  • runner_resources.events.AsyncEvents
  • runner_resources.events.AsyncEvents.get
  • runner_resources.metrics.AsyncMetrics
  • runner_resources.shares.AsyncShares.update
  • streaming.parse_ag_ui_events_async
  • 19 aliases to ag_ui.core don't resolve on either side (external package, unchanged).

Runtime-visible change outside griffe: the moved classes' __module__ is now introspection_sdk._sync.… / introspection_sdk._async.… (for example introspection_sdk.Runner.__module__). Import paths, isinstance and pickling-by-reference through the public paths are unaffected. The events module logger keeps its name (introspection_sdk.runner_resources.events).

Duplicate code (jscpd)

introspection_sdk/_sync/** goes in .jscpd.json ignore; it repeats _async/ by construction.

Duplicated lines Total lines Share
main (baseline) 5,788 19,813 29.2% (exact-only 15.8%)
this PR 2,819 16,951 16.63% (exact-only 3.00%)

threshold goes from 29.3 to 16.7.

Checks run locally

  • uv run pytest tests/: 774 passed, no test edits. Coverage 93.9% (main 93.6%); --cov-fail-under=80 holds. The fenced async docstrings count as statements, which adds 3 uncovered lines in async methods no test calls.
  • ruff format --check, ruff check (incl. the complexity ratchet), ty check (pinned), mypy, deptry, jscpd (exit 0), actionlint, check_examples.py, scripts/unasync.py --check: all pass.
  • scripts/check_api_contract.py (scheduled, not a PR gate): output is byte-identical to main's.
  • prek run --all-files: every hook passes except the local unpinned uvx ty. That gives the same 37 diagnostics on main (they move along with the code), while CI's pinned ty passes.

API check: re-exported classes

griffe check passed vacuously for every class this PR moves behind a re-export: _type_based_yield adds the old object's path to seen_paths, goes into _alias_incompatibilities, and recurses into the alias's target with that same old path, which returns at once. griffe has no CLI option for this, and griffe 2.4.0 (latest) has the same diff.py, so an upgrade doesn't help.

The fix stays on the plain CLI and uses griffe's extension API instead of a custom checker: scripts/griffe_reexports.py is a ~20-line griffe.Extension whose on_package hook replaces each public alias whose target lives in introspection_sdk with the target object. It runs on the old release and the new code alike (griffe loads both with the same extensions), so both sides hold real objects and griffe's own breakage rules compare them. api-compat.yml adds -e scripts/griffe_reexports.py; the api-break label and ! title override is unchanged. AGENTS.md documents the flag. pyproject.toml widens the scoped deptry DEP004 ignore to griffe (dev-only import in scripts/).

Evidence (the breaking change was applied locally only, never committed): remove name from AsyncRecipes.list in _async/resources/recipes.py and regenerate _sync/.

Clean HEAD (c637ad2):

$ uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py
exit=0

With the breaking change, the old command vs the new one:

$ uv run griffe check introspection_sdk --search . --against v0.29.0
exit=0

$ uv run griffe check introspection_sdk --search . --against v0.29.0 -e scripts/griffe_reexports.py
introspection_sdk/_async/resources/recipes.py:25: AsyncRecipes.list(name): Parameter was removed
introspection_sdk/_sync/resources/recipes.py:27: Recipes.list(name): Parameter was removed
exit=1

Docs

AGENTS.md gains a "Sync code generation" section (workflow, layout, token map, fence, the hand-written list, adding a twin). The "Duplicate code" section now mentions the _sync/ exclusion, and "Public API compatibility" describes the griffe extension below.

🤖 Generated with Claude Code

Every class or function with an Async* twin is now written once, as async
code under introspection_sdk/_async/, and scripts/unasync.py generates the
sync twin into introspection_sdk/_sync/ with unasync (pinned in the dev
extra), as httpx/httpcore and urllib3 do. The public modules re-export both
twins, so every import path and public name is unchanged; code both twins
share moves to sibling _<module>_common.py modules.

The generated files carry a "do not edit" header, and CI and a pre-commit
hook regenerate them and fail on any diff. Code that genuinely differs
between sync and async stays hand-written: Pager/AsyncPager,
stream_resumable, EmailCodeAuth, Annotations and RuntimeHandle.

The generated tree is excluded from jscpd, which drops the duplicated-lines
share of introspection_sdk/ from 29.2% to 16.63%; the threshold goes from
29.3 to 16.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T12:04:56.605041Z c637ad2 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

griffe check skips the members of a class that moved behind a re-export:
_type_based_yield records the old path in seen_paths, then recurses into
the alias's target with that same path and returns. Load a griffe
extension that swaps each public in-package alias for its target before
the check, so the breakage rules apply to the moved classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marcusschiesser

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: c637ad28da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@marcusschiesser

Copy link
Copy Markdown
Contributor Author

Superseded by #189, which rebuilds this change as five commits git can follow (commit 1 is 25 pure renames into _async/). Its final tree is byte-identical to this PR's head 1781225 (git diff 1781225 ea1fbf6 is empty). The branch marcusschiesser/unasync-sync-codegen is kept.

This branch was previously deployed

1 inactive deployment
build — 1781225c Deployed Oct 21, 2026 by marcusschiesser via Unit Tests #512
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant