Skip to content

Ilongin/1910 preserve pydantic class identity - #1969

Open
ilongin wants to merge 10 commits into
mainfrom
ilongin/1910-preserve-pydantic-class-identity
Open

Ilongin/1910 preserve pydantic class identity#1969
ilongin wants to merge 10 commits into
mainfrom
ilongin/1910-preserve-pydantic-class-identity

Conversation

@ilongin

@ilongin ilongin commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Preserve the identity of plain Pydantic models when reading saved datasets in a new process.

Previously, only DataModel subclasses were registered automatically. A plain BaseModel was reconstructed from its stored schema during deserialization, even when the original class had already been imported. The reconstructed class had the same name and fields but different Python identity, causing isinstance() checks and nested Pydantic validation to fail.

Closes #1910.

Changes

  • Reuse an already-imported Pydantic class when its module, name, and complete model shape match the stored schema.
  • Compare nested model graphs recursively, including models inside optional, list, mapping, tuple, and union annotations.
  • Validate existing ModelStore matches before reusing them.
  • Fall back to a synthetic model when the original class is unavailable or its shape has changed.
  • Warn when an imported class has the expected identity but a different stored shape.
  • Resolve classes only from sys.modules; dataset metadata never triggers a module import.
  • Improve errors for malformed nested custom-type schemas.
  • Update the Pydantic model documentation to describe automatic resolution and synthetic fallback.

Tests

  • Added a cross-process regression test covering direct, list, and optional nested Pydantic models.
  • Verified that the restored objects use the imported classes and can be passed back into their parent model.
  • Verified that validators on the restored real class run during dataset reads.
  • Added unit coverage for matching and mismatched model graphs, recursive models, malformed nested schemas, safe module lookup, and ModelStore behavior.

Fixes #1910

@ilongin
ilongin marked this pull request as draft August 31, 2026 14:53
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 31, 2026

Copy link
Copy Markdown

Deploying datachain with  Cloudflare Pages  Cloudflare Pages

Latest commit: f032289
Status: ✅  Deploy successful!
Preview URL: https://13b00d32.datachain-2g6.pages.dev
Branch Preview URL: https://ilongin-1910-preserve-pydant.datachain-2g6.pages.dev

View logs

@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ilongin
ilongin marked this pull request as ready for review September 2, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested pydantic models lose their class identity in another process

1 participant