tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
2,664 Commits 319 Branches 32 Tags
c2bd75aff610f257e00a7f4e7e6649f39d770757
Commit Graph

12 Commits

Author SHA1 Message Date
Grant Holmes (Ren)
850c1a5f8d Add FSDP v2 swap memory support + QLoRA compatibility fixes (#3167) 
Co-authored-by: salman <salman.mohammadi@outlook.com>
 2025-09-26 10:23:59 +01:00
Wing Lian
22810c97b7 use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci] 
* use warmup_ratio as a better default than warmup steps since it's data dependent

* replace remainder of warmup_steps
 2025-07-30 06:44:06 -04:00
Wing Lian
af8d257aa2 make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci] 
* make pad_to_sequence_len default to the same value as sample_packing

* remove duplicate validation

* fix test

* update description meta

Co-authored-by: NanoCode012 <nano@axolotl.ai>

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
 2025-07-21 11:40:56 -04:00
Dan Saunders
10ba1622f7 checkpoint model on first step callback (#2906) 
* checkpoint model on first step callback

* remove debug

* add test cases; update existing tests not to save on first step

* move test out of solo

* delete

* default to False

* typo
 2025-07-15 15:00:48 -04:00
Wing Lian
dd8bad06d0 remove strict=false from example yamls [skip ci] (#2523) [skip ci] 2025-04-12 07:25:11 -07:00
Wing Lian
9f824ef76a simplify the example configs to be more minimal and less daunting (#2486) [skip ci] 
* simplify the example configs to be more minimal and less daunting

* drop empty s2_attention from example yamls
 2025-04-04 13:47:26 -04:00
salman
ac471a697a updating to fused (#2293) 2025-01-30 11:45:56 -05:00
Sunny Liu
1c14c4a15c Add hub model id config options to all example yml files (#2196) [skip ci] 
* added hub model_id in example yml

* add hub model id to example yml
 2024-12-17 11:24:30 -05:00
Wing Lian
4fde300e5f update outputs path so that we can mount workspace to /workspace/data (#1623) 
* update outputs path so that we can mount workspace to /workspace/data

* fix ln order
 2024-05-15 12:44:13 -04:00
Wing Lian
132eb740f0 DBRX Model Support (#1462) 
* wip for dbrx finetuning

* add fastcore for parallel loading of sharded weights

* fix dtype for load, use PartialState instead of accelerator to init process group, remove redundant wandb callback

* update to use v2 of the converted model

* more fixes for dbrx loras

* make sure to enable fsdp activation checkpointing

* fix support for 8bit loras too for dbrx

* apply z3 leaf moe fix for DBRX with deepspeed

* don't raise value error since child module searches could fail and be ok

* revert a previous change to fix fsdp

* update mistral/mistral qlora+fsdp yamls

* fix qlora+fsdp quant storage type

* more edge cases for qlora-fsdp

* fixes for fsdp+qlora w optimizer in 8bit

* add bigstral z3 config and make sure to use full_state_dict for fsdp
 2024-04-12 09:02:36 -04:00
Wing Lian
2a1589f6f6 strip out hacky qlora-fsdp workarounds now that qlora-fsdp fixes are upstreamed (#1428) 2024-03-21 11:56:13 -04:00
Wing Lian
9b6ee83a73 FDSP + QLoRA (#1378) 
* wip qlora + fsdp fixes

* more fixes

* make sure to load the lora 🤦

* only setup quantized meta on non-zero rank:

* only run setup_quantized_peft_meta_for_training for qlora+fsdp

* more fixes for qlora+fsdp

* chore: lint

* add example yml

* support mistral too

* fix for model_type and add mixtral support too

* set cpu_offload: false to reduce vram, constrain new accleerator logic to qlora + fsdp

* refactor for duplicate code
 2024-03-08 14:31:01 -05:00