axolotl/examples/llama-4/README.md

Llama 4 by Meta AI

Available Examples

Llama 4 Scout 17Bx16Experts (109B)

• Multi-Modal/Vision QLoRA w/ FSDP1
• Text Single GPU (H100) QLoRA
• Text Multi GPU QLoRA w/ FSDP1

Our Single H100 implementation for Llama 4 Scout uses only 68.5GB VRAM for post-training with 4k context length @ 546 tokens/second. WandB logs here

Llama 4 Maverick 17Bx128Experts (400B)

• Text Multi GPU QLoRA w/FSDP1

Our 4xH100 implementation for Llama 4 Maverick uses 79.5GB VRAM/GPU for post-training with 4k context length @ 206 tokens/second. WandB logs here.