Repository navigation
StorageClientConfig.from_file ignores all options in implicit profile creation #19
Closed as not planned
Description
Summary
Subject
Steps to Reproduce
- Create a config file named
msc_bug.yaml
profiles:
s3-my-bucket:
storage_provider:
type: s3
options:
endpoint_url: https://my-endpoint.s3.com
base_path: my-bucket- Create a script named
msc_bug.py
import multistorageclient as msc
def main():
client, _ = msc.resolve_storage_client("s3://my-bucket/my_file.txt")
assert client._storage_provider._endpoint_url == "https://my-endpoint.s3.com", f"{client._storage_provider._endpoint_url=}"
if __name__ == "__main__":
main()- Run
MSC_MSC_CONFIG=msc_bug.yaml python msc_bug.py Environment Details
- MultiStorageClient version
- Operating system and version
- Python version
Expected Behaviour
Url, as well as all other options, should be passed
Additional Context
Add any other context or screenshots about the bug here.
Activity
[-]StorageClientConfig.from_file ignore all options in implicit profile creation[/-][+]StorageClientConfig.from_file ignores all options in implicit profile creation[/+]pavelgein commentedon Mar 19, 2026
In this line https://github.com/NVIDIA/multi-storage-client/blob/main/multi-storage-client/src/multistorageclient/config.py#L77
only some of the options are passed
commiterate commentedon Mar 19, 2026
Reproduction with MSC 0.44.0: https://github.com/NVIDIA/multi-storage-client/tree/reproduction/19/reproduction
This is currently intended behavior since
multistorageclient.shortcuts.resolve_storage_client()(ref) doesn't know which profile to choose if the MSC config file looks like this:From an
s3://{bucket}/...URL, we only know the storage provider type (s3) and the bucket (which is used for thebase_path). If we filtered profiles for that, we'd get back{"s3-profile", "s3-emulator-profile"}.Since there currently isn't a tie breaking method, the implicit profiles feature instead creates a separate profile with minimum options that can be inferred from the URL (e.g.
_s3-my-bucketwithtype: s3+options.base_path: my-bucketset. This implicit profile naming scheme may change in the future).Today users need to use
msc://{profile name}instead to get an existing profile, somsc://s3-my-bucket/my_file.txt. The reproduction shows the differences in the resolved MSC storage clients.fix: reuse explicit profile options for matching implicit profiles (f…
commiterate commentedon Mar 19, 2026
Note that there's an existing path mapping feature that lets users map
s3://URLs tomsc://URLs so code can still continue to uses3://URLs.https://nvidia.github.io/multi-storage-client/references/configuration.html#path-mapping
Using the reproduction as the base, this would work:
msc_bug.yamlA path mapping of
s3://: msc://s3-no-base-path/, however, isn't supported yet.pavelgein commentedon Mar 20, 2026
Thanks, it works