Using the very fast CPU-GPU ram in the JUPITER nodes we can try to offload the optimizer states to do it. There should be a latency penalty but not so severe as with regular x86 nodes.
- Try CPU Offloading policy with the MixedPrecision that is used with FSDP2
Using the very fast CPU-GPU ram in the JUPITER nodes we can try to offload the optimizer states to do it. There should be a latency penalty but not so severe as with regular x86 nodes.