tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
2,690 Commits 319 Branches 32 Tags
weight-scale-norm
Commit Graph

12 Commits

Author SHA1 Message Date
Wing Lian
22810c97b7 use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci] 
* use warmup_ratio as a better default than warmup steps since it's data dependent

* replace remainder of warmup_steps
 2025-07-30 06:44:06 -04:00
Wing Lian
af8d257aa2 make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci] 
* make pad_to_sequence_len default to the same value as sample_packing

* remove duplicate validation

* fix test

* update description meta

Co-authored-by: NanoCode012 <nano@axolotl.ai>

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
 2025-07-21 11:40:56 -04:00
Dan Saunders
10ba1622f7 checkpoint model on first step callback (#2906) 
* checkpoint model on first step callback

* remove debug

* add test cases; update existing tests not to save on first step

* move test out of solo

* delete

* default to False

* typo
 2025-07-15 15:00:48 -04:00
Wing Lian
dd8bad06d0 remove strict=false from example yamls [skip ci] (#2523) [skip ci] 2025-04-12 07:25:11 -07:00
Wing Lian
9f824ef76a simplify the example configs to be more minimal and less daunting (#2486) [skip ci] 
* simplify the example configs to be more minimal and less daunting

* drop empty s2_attention from example yamls
 2025-04-04 13:47:26 -04:00
salman
ac471a697a updating to fused (#2293) 2025-01-30 11:45:56 -05:00
Sunny Liu
1c14c4a15c Add hub model id config options to all example yml files (#2196) [skip ci] 
* added hub model_id in example yml

* add hub model id to example yml
 2024-12-17 11:24:30 -05:00
Wing Lian
4fde300e5f update outputs path so that we can mount workspace to /workspace/data (#1623) 
* update outputs path so that we can mount workspace to /workspace/data

* fix ln order
 2024-05-15 12:44:13 -04:00
Wing Lian
814aee6603 Phi2 multipack (#1173) 
* phi2 multipack

* update validation and examples for phi

* more updates to phi examples

* make sure to use the correct collator for phi multipack

* phi needs attention mask now for multipack

* if the special token already exists in the tokenizer, don't require in lora modules to save

* fix qlora yml for phi, fix phi test validation

* test qlora too

* make sure flash attention is enabled for the test

* don't use remote code for phi anymore

* reduce sequence len for sample packing phi
 2024-01-23 12:54:36 -05:00
Wing Lian
782b6a4216 set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci] 
* set fp16 to false if bf16, update bf16: auto in example YAMLs

* unset fp16 so that it fallsback properly if bf16 isn't available

* Update README.md [skip-ci]

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

* test that bf16 disables fp16

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>
 2024-01-22 18:44:01 -05:00
Wing Lian
c1b741d9fb pin model_revision for phi2 (#1123) 2024-01-14 17:31:51 -05:00
Wing Lian
732851f105 Phi2 rewrite (#1058) 
* restore to current phi modeling code from phi-2

* enable gradient checkpointing

* don't cast everything to float32 all the time

* gradient checkpointing for phi2 ParallelBlock module too

* fix enabling flash attn for phi2

* add comment about import

* fix phi2 example

* fix model type check for tokenizer

* revert float32 -> bf16 casting changes

* support fused dense flash attn

* fix the repo for flash-attn

* add package name for subdir pkg

* fix the data collator when not using sample packing

* install packaging for pytests in ci

* also fix setup to not install flash attn fused dense subdir if not extras

* split out the fused-dense-lib in extra requires

* don't train w group_by_length for phi

* update integration test to use phi2

* set max steps and save steps for phi e2e tests

* try to workaround ssave issue in ci

* skip phi2 e2e test for now
 2024-01-08 14:04:22 -05:00