Repository navigation
Interested to see some benchmark of objstore with PyTorch DataLoader #448
Description
Hey @kylebarron.
I am the main author of LitData: Lightning-AI/litData and we have lot of GeoSpatial users.
I am quite curious to see how objstore will behave with PyTorch DataLoader.
We have tried to build a rust backend to pull the data (our own or tried hf_transfer from Hugging Face).
It was faster in raw benchmark outside of PyTorch DataLoader but it became 10-30% slower otherwise. The CPU usage was pretty chaotic using a rust backend and constantly at 100% with boto3.
I would be pretty keen to connect and see if there is room for collaboration. I still believe that we could make a rush backend works well.
Best regards,
Thomas Chaton.
Activity
[-]Provide some benchmark of objstore with PyTorch DataLoader[/-][+]Interested to see some benchmark of objstore with PyTorch DataLoader[/+]kylebarron commentedon May 13, 2025
I haven't used PyTorch and I'm not familiar with its DataLoader. Is it async-compatible? The greatest performance improvements from obstore come when using it with high concurrency in async Python code.