Files

Wing Lian 9d5c95db6f Add support for Accelerate CP, ND examples, and fix for parallel config w fsdp (#3019 )

* fix for parallelism config from trainer

* fix handling of parallelism_config w accelerate

* add todo for removal

* update to latest axolotl-contribs-mit for optimizer fix too

* synchronize training after checkpoint save

* dir spelling

* use latest accelerate main

* fix to not use partial state parallelism_config

* more fixeS

* use most recent accelerate fix

* fix cpu_ram_efficient_loading to meta devices from rank 0 to prevent CPU RAM oom

* improve handling of broadcasting fsdp2 state dict

* support for openai chat template with thinking key as the reasoning trace

* address PR feedback

* refactor to remove dependency on PartialState for parallelism config

* bump accelerate, gptoss fixes

* limit meta fixes to fsdp2 for now

* fixes for gpt oss

* fixup examples, don't use cpu-ram-efficient-loading for now

* remove problematic barrier

* patch parallelism config

* reorder comparison

* device mesh fixes

* make pure CP work

* lint

2025-08-07 21:22:15 -04:00

llama-3_1-8b-hdsp-tp.yaml

Add support for Accelerate CP, ND examples, and fix for parallel config w fsdp (#3019 )

2025-08-07 21:22:15 -04:00

qwen3-8b-fsdp-tp-cp.yaml

Add support for Accelerate CP, ND examples, and fix for parallel config w fsdp (#3019 )

2025-08-07 21:22:15 -04:00

README.md

Add support for Accelerate CP, ND examples, and fix for parallel config w fsdp (#3019 )

2025-08-07 21:22:15 -04:00

README.md

Distributed Parallel

See the accompanying blog post: Accelerate ND-Parallel: A guide to Efficient Multi-GPU Training

The examples provided are suitable for single node (8xGPU) SFT.

Qwen 3 8B w/ FSDP + TP + CP: YAML
Llama 3.1 8B w/ HSDP + TP: YAML