tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

feat(doc): explain deepspeed configs (#2514) [skip ci]

Browse Source 
* feat(doc): explain deepspeed configs

* fix: add fetch configs
This commit is contained in:
NanoCode012
2025-04-11 20:52:43 +07:00
committed by GitHub
parent 9a8e3e9c7b
commit 756a0559c1
1 changed files with 14 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/multi-gpu.qmd
View File

@@ -36,6 +36,9 @@ deepspeed: deepspeed_configs/zero1.json
### Usage {#sec-deepspeed-usage} ### Usage {#sec-deepspeed-usage}
```{.bash} ```{.bash}
# Fetch deepspeed configs (if not already present)
axolotl fetch deepspeed_configs
# Passing arg via config # Passing arg via config
axolotl train config.yml axolotl train config.yml
@@ -48,10 +51,20 @@ axolotl train config.yml --deepspeed deepspeed_configs/zero1.json
We provide default configurations for: We provide default configurations for:
- ZeRO Stage 1 (`zero1.json`) - ZeRO Stage 1 (`zero1.json`)
- ZeRO Stage 1 with torch compile (`zero1_torch_compile.json`)
- ZeRO Stage 2 (`zero2.json`) - ZeRO Stage 2 (`zero2.json`)
- ZeRO Stage 3 (`zero3.json`) - ZeRO Stage 3 (`zero3.json`)
- ZeRO Stage 3 with bf16 (`zero3_bf16.json`)
- ZeRO Stage 3 with bf16 and CPU offload params(`zero3_bf16_cpuoffload_params.json`)
- ZeRO Stage 3 with bf16 and CPU offload params and optimizer (`zero3_bf16_cpuoffload_all.json`)
Choose based on your memory requirements and performance needs. ::: {.callout-tip}
Choose the configuration that offloads the least amount to memory while still being able to fit on VRAM for best performance.
Start from Stage 1 -> Stage 2 -> Stage 3.
:::
## FSDP {#sec-fsdp} ## FSDP {#sec-fsdp}