Skip to content

Always use spawn for data loader workers#782

Draft
jder wants to merge 1 commit into
mainfrom
codex/always-use-spawn
Draft

Always use spawn for data loader workers#782
jder wants to merge 1 commit into
mainfrom
codex/always-use-spawn

Conversation

@jder

@jder jder commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • Remove fork/spawn selection based on data source location and always use spawn when data loader workers are enabled.
  • Simplify DataContainer and location resolution by dropping fork-support checks that are no longer needed.
  • Update trainer tests to assert the multiprocessing context is spawn for worker-backed loaders and None when workers are disabled.

Testing

  • Added/updated unit coverage for trainer loader setup.
  • Not run (not requested).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant