tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
2,083 Commits 319 Branches 32 Tags
model-loader-refactor
Commit Graph

14 Commits

Author SHA1 Message Date
Wing Lian
dd8bad06d0 remove strict=false from example yamls [skip ci] (#2523) [skip ci] 2025-04-12 07:25:11 -07:00
Wing Lian
9f824ef76a simplify the example configs to be more minimal and less daunting (#2486) [skip ci] 
* simplify the example configs to be more minimal and less daunting

* drop empty s2_attention from example yamls
 2025-04-04 13:47:26 -04:00
Sunny Liu
1c14c4a15c Add hub model id config options to all example yml files (#2196) [skip ci] 
* added hub model_id in example yml

* add hub model id to example yml
 2024-12-17 11:24:30 -05:00
Wing Lian
4fde300e5f update outputs path so that we can mount workspace to /workspace/data (#1623) 
* update outputs path so that we can mount workspace to /workspace/data

* fix ln order
 2024-05-15 12:44:13 -04:00
Seungduk Kim
05bcc9ea56 Train parameters exclusively in specific ranges (#1390) 
* Train parameters exclusively in specific ranges

* Fix the style and update docs

* Update yaml example
 2024-03-14 11:05:42 -04:00
Leonardo Emili
5a5d47458d Add seq2seq eval benchmark callback (#1274) 
* Add CausalLMBenchEvalCallback for measuring seq2seq performance

* Fix code for pre-commit

* Fix typing and improve logging

* eval_sample_packing must be false with CausalLMBenchEvalCallback
 2024-02-13 08:24:30 -08:00
Wing Lian
54d2ac155b Mixtral fixes 20240124 (#1192) [skip ci] 
* mixtral nccl fixes

* make sure to patch for z3
 2024-01-24 14:59:57 -05:00
Wing Lian
782b6a4216 set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci] 
* set fp16 to false if bf16, update bf16: auto in example YAMLs

* unset fp16 so that it fallsback properly if bf16 isn't available

* Update README.md [skip-ci]

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

* test that bf16 disables fp16

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>
 2024-01-22 18:44:01 -05:00
Wing Lian
628b754824 set output_router_logits for mixtral config: (#995) 2023-12-22 12:57:02 -05:00
Wing Lian
5ea3aa31f0 Fix Deepspeed loading (#950) 
* add check for zero3

* freeze parameters

* fixes for deepspeed loading

* fix model parameter check

* unfrozen parameters in example mixtral and logging when unfreezing
 2023-12-13 16:03:23 -05:00
Wing Lian
5f79b8242f new evals_per_epoch and saves_per_epoch to make things cleaner (#944) 
* new evals_per_epoch and saves_per_epoch to make things cleaner

* update per PR feedback
 2023-12-12 15:35:23 -05:00
Wing Lian
7fabc4d95e Mixtral official (#942) 
* multipack support for official mixtral implementation

* fix patch to load multipack for mixtral

* chore: lint
 2023-12-11 23:44:33 -05:00
Wing Lian
35f9b0f149 update to latest transformers for mixstral support (#929) 
* update to latest transformers for mixstral support

* pin transformers

* fix typo
 2023-12-10 10:32:27 -05:00
Wing Lian
68b227a7d8 Mixtral multipack (#928) 
* mixtral multipack

* use mixtral model

* sample yml

* calculate cu_seqlens properly

* use updated flash ettention setting

* attn var checks

* force use of flash attention 2 for packing

* lint

* disable future fix for now

* update support table
 2023-12-09 21:26:30 -05:00