Skip to content

partition schemas and model stubs - #871

Open
stobrien89 wants to merge 1 commit into
native-types-cli-integrationfrom
native-codegen-partitioning
Open

stobrien89 wants to merge 1 commit into
native-types-cli-integrationfrom
native-codegen-partitioning

Conversation

@stobrien89

Copy link
Copy Markdown
Contributor

Issue #, if available:
#797

Description

Partitions large standalone types packages to reduce generation memory and static analysis time.

Stacked on #866 and intended to be reviewed against native-types-cli-integration.

Changes

  • Partition schemas above 250 shapes using weighted source-size boundaries.
  • Keep recursive schema components in the same partition.
  • Partition model stubs above 250 declarations.
  • Preserve stable schemas.py and models.pyi facades.
  • Preserve single-file output for packages at or below each threshold.

EC2 benchmark

Measured on x86 with 4,065 selected shapes, five runs, and one warmup.

Metric Unpartitioned Partitioned Change
Generation time 3.557 s 3.555 s -0.1%
Peak RSS 264.5 MiB 122.4 MiB -53.7%
Peak Python allocation 222.6 MiB 84.7 MiB -61.9%
Generated bytes 4.78 MiB 5.33 MiB +11.6%
Model import 50.6 ms 40.4 ms -20.2%
First model access 12.3 ms 22.3 ms +10.0 ms
Strict Pyright 81.98 s 15.25 s -81.4%

Validation

  • Tests cover schema and stub output below, at, and above the 250-item threshold.
  • Tests verify recursive component placement, cross-partition references, facade caching, deterministic output, and stable public exports.
  • Partitioned generated packages pass Ruff and strict Pyright.

Compatibility

Public model imports, lazy model construction, schema identity, serde behavior, and typing remain unchanged. Partition files live under _private.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@stobrien89
stobrien89 added this pull request to stack #864 October 25, 2026 23:25
@stobrien89
stobrien89 requested a balanced review from Copilot October 25, 2026 23:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Generated-output compatibility and lint failures remain, alongside schema-planning and partition-weighting issues that undermine scalability.

4 open issues
What changed in this PR

This PR extends the native standalone types generator with private schema and model-stub partitions, reducing memory use and static analysis time while retaining stable facade modules.

Changes:

  • Adds configurable partition targets, defaulting to 250 items.
  • Balances schema partitions by estimated source size and keeps recursive components together.
  • Adds partitioning tests and documents output layout, compatibility, and benchmark results.
File Description
packages/​smithy-python/​tests/​unit/​test_schemas.py Tests thresholds, lazy loading, recursion, and size balancing.
packages/​smithy-python/​tests/​unit/​test_models.py Tests partitioned stubs, public models, and strict Pyright.
packages/​smithy-python/​tests/​unit/​test_generation.py Tests partition defaults and positive-value validation.
packages/​smithy-python/​src/​smithy_python/​schemas.py Adds weighted schema partitions and a lazy facade.
packages/​smithy-python/​src/​smithy_python/​models.py Adds private stub partitions and public re-exports.
packages/​smithy-python/​src/​smithy_python/​generation.py Defines and validates partition settings.
designs/​codegen/​types.md Updates settings and large-model design guidance.
designs/​codegen/​partitioning.md Documents partitioning behavior and performance measurements.
designs/​codegen/​index.md Links the partitioning design.

Review effort: Balanced

Comment thread packages/smithy-python/src/smithy_python/models.py
Comment thread packages/smithy-python/src/smithy_python/schemas.py
Comment thread packages/smithy-python/src/smithy_python/schemas.py Outdated
Comment thread packages/smithy-python/src/smithy_python/schemas.py

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review found no blocking issues in partitioning, facade behavior, or compatibility, and the supporting tests cover the key boundaries.

No open issues 🎉

4 resolved since last review

Review effort: Balanced

@stobrien89
stobrien89 marked this pull request as ready for review October 26, 2026 00:32
@stobrien89
stobrien89 requested a review from a team as a code owner October 26, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants