tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

simplify the example configs to be more minimal and less daunting (#2486) [skip ci]

Browse Source 
* simplify the example configs to be more minimal and less daunting

* drop empty s2_attention from example yamls
This commit is contained in:
Wing Lian
2025-04-04 13:47:26 -04:00
committed by GitHub
parent dd66fb163c
commit 9f824ef76a
101 changed files with 14 additions and 1140 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
examples/deepseek-v2/qlora-fsdp-2_5.yaml
View File

@@ -52,27 +52,19 @@ optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-5
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
special_tokens:
fsdp: