Repository navigation
[Bugfix][Loader] Read index shards in subfolders; direct I/O U16/U32 - #55
Merged
Merged
Conversation
When model.safetensors.index.json names shards both at the top level and in a subfolder, the loader globbed only the top level and never read the subfolder shards; their parameters stayed unloaded without an error. The loader now adds the index's subfolder shards, and fails at load time when the index names a shard that would not be read (unless the model passes its own allow patterns). Direct I/O maps the safetensors U16 and U32 dtypes. Port of 1CatAI/1Cat-vLLM#1274 by @Peuqui (merged as 691b6597c9). One change from upstream: this tree has an index-only fallback that already lists every shard when the top level has none (the FP4-Local layout), so the subfolder pass adds only shards not yet selected; applied as is, the patch made FP4-Local read each of its 21 shards twice. Adds a test for that layout. Fixes #54 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
This was referenced Oct 30, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
When a checkpoint's
model.safetensors.index.jsonnames shards both at the top level and in a subfolder,DefaultModelLoaderglobbed only the top level and never read the subfolder shards. The parameters in those shards stayed unloaded and no error was raised. The loader now adds the index's subfolder shards to the files it reads, and fails at load time when the index names a shard that would still not be read (unless the model passes its own allow patterns). Direct I/O now maps the safetensorsU16andU32dtypes.Upstream fix: 1CatAI/1Cat-vLLM#1274 by @Peuqui (merged as 691b6597c9). It applies cleanly to this tree, with one change: this tree has an index-only fallback that lists every shard when the top level has none (the FP4-Local layout, all shards in
tensors/). Applied as is, the subfolder pass then added the same 21 shards a second time, so FP4-Local would have read every shard twice. The subfolder pass now adds only shards not yet selected, and a new test covers that layout.No model numerics change. None of the checkpoints the README lists use the mixed layout, so they load the same files as before.
Fixes #54
Validation
CPU, in the ReVolta build image:
pytest tests/model_executor/model_loader(without the tensorizer and Run:ai streamer suites, which need packages the container does not have): 192 passed with the change, 187 onmain. The 10 failures and 5 errors are the same set by node id onmain(reload and sharded-state tests that need a GPU or downloads). The 5 extra passes are the new tests: subfolder shards for three load strategies, the missing-shard error, and the all-subfolder layout.main, with no new errors. Before the de-duplication, FP4-Local selected 42.GPU: every engine start loads the checkpoint, so the 0.1.5 acceptance covers the reference checkpoint through its normal starts.
Attribution
The loader change and its tests are from 1CatAI/1Cat-vLLM#1274 (author @Peuqui, Apache-2.0). The de-duplication, its test and the port were prepared with AI assistance (Claude Code). Commits are signed off under the DCO.
🤖 Generated with Claude Code