tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[GPT-OSS] improve FSDP shard merging and documentation for GPT-OSS (#3073)

Browse Source 
* improve fsdp shard merging

* improve logging

* update information on merging and inferencing GPT-OSS

* cleanup readme

* automate cleanup of FSDP prefix

* import GRPO only if necessary

* only modify config.json on rank0

* merge final checkpoint at end of training

* prevent circular import

* Fix saving for sharded state dict

* devx, move merged to output dir

* move import back to top

* Fix stuck merge

* fix conditionals from pr feedback and add test
This commit is contained in:
Wing Lian
2025-08-15 21:25:01 -04:00
committed by GitHub
parent 130ef7c51a
commit ecbe8b2b61
7 changed files with 170 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
examples/gpt-oss/gpt-oss-120b-fft-fsdp2-offload.yaml
View File

@@ -20,6 +20,7 @@ datasets:
dataset_prepared_path: last_run_prepared
val_set_size: 0
output_dir: ./outputs/gpt-oss-out/
save_total_limit: 2 # the 120B model can use up to 720GB of disk space per checkpoint, so let's only keep the last 2
sequence_len: 4096
sample_packing: true