Bing found these results
zero3 and enable hybrid engine are not suitable for llama2
DeepSpeed ZeRO-3 Offload - DeepSpeed
ZeRO — DeepSpeed 0.14.3 documentation - Read the Docs
Zero3
deepspeed hybrid-engine support bloom model with zero3? #497 - GitHub
[BUG] When exec deepspeedchat step3_rlhf_finetuning training …
FullyShardedDataParallel — PyTorch 2.3 documentation
DeepSpeed ZeRO++: A leap in speed for LLM and chat model …
ZEROe - Low carbon aviation - Airbus
W-ZERO3 - Wikipedia