distributed inference in a local cluster

[mfahsold]

Is it possible to distribute the extra across a kubernetes cluster? I have multiple machines in my Lab with different platforms, RAM and different GPU/NPU and would like to distribute the active exports across multiple machines.

[mfahsold]

*active experts not extras

[shuk777]

Thanks for your interest in our work! Currently we do not have distributed inference in our roadmap, you could work on this in your own fork for now.