tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
2,664 Commits 319 Branches 32 Tags
c2bd75aff610f257e00a7f4e7e6649f39d770757
Commit Graph

23 Commits

Author SHA1 Message Date
Wing Lian
22810c97b7 use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci] 
* use warmup_ratio as a better default than warmup steps since it's data dependent

* replace remainder of warmup_steps
 2025-07-30 06:44:06 -04:00
Wing Lian
af8d257aa2 make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci] 
* make pad_to_sequence_len default to the same value as sample_packing

* remove duplicate validation

* fix test

* update description meta

Co-authored-by: NanoCode012 <nano@axolotl.ai>

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
 2025-07-21 11:40:56 -04:00
Dan Saunders
10ba1622f7 checkpoint model on first step callback (#2906) 
* checkpoint model on first step callback

* remove debug

* add test cases; update existing tests not to save on first step

* move test out of solo

* delete

* default to False

* typo
 2025-07-15 15:00:48 -04:00
Wing Lian
dd8bad06d0 remove strict=false from example yamls [skip ci] (#2523) [skip ci] 2025-04-12 07:25:11 -07:00
Wing Lian
9f824ef76a simplify the example configs to be more minimal and less daunting (#2486) [skip ci] 
* simplify the example configs to be more minimal and less daunting

* drop empty s2_attention from example yamls
 2025-04-04 13:47:26 -04:00
Sunny Liu
1c14c4a15c Add hub model id config options to all example yml files (#2196) [skip ci] 
* added hub model_id in example yml

* add hub model id to example yml
 2024-12-17 11:24:30 -05:00
Wing Lian
4fde300e5f update outputs path so that we can mount workspace to /workspace/data (#1623) 
* update outputs path so that we can mount workspace to /workspace/data

* fix ln order
 2024-05-15 12:44:13 -04:00
NanoCode012
f1ebaa07c6 chore(config): refactor old mistral config (#1435) 
* chore(config): refactor old mistral config

* chore: add link to colab on readme
 2024-03-25 12:00:44 +09:00
NanoCode012
a7a9a1433a fix(examples): remove is_*_derived as it's parsed automatically (#1297) 2024-02-22 00:52:46 +09:00
Leonardo Emili
5a5d47458d Add seq2seq eval benchmark callback (#1274) 
* Add CausalLMBenchEvalCallback for measuring seq2seq performance

* Fix code for pre-commit

* Fix typing and improve logging

* eval_sample_packing must be false with CausalLMBenchEvalCallback
 2024-02-13 08:24:30 -08:00
Wing Lian
782b6a4216 set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci] 
* set fp16 to false if bf16, update bf16: auto in example YAMLs

* unset fp16 so that it fallsback properly if bf16 isn't available

* Update README.md [skip-ci]

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

* test that bf16 disables fp16

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>
 2024-01-22 18:44:01 -05:00
mhenrichsen
93ebec1ac5 change val size (#992) 2023-12-22 16:18:16 +01:00
Wing Lian
5f79b8242f new evals_per_epoch and saves_per_epoch to make things cleaner (#944) 
* new evals_per_epoch and saves_per_epoch to make things cleaner

* update per PR feedback
 2023-12-12 15:35:23 -05:00
NanoCode012
a1da39cd48 Feat(wandb): Refactor to be more flexible (#767) 
* Feat: Update to handle wandb env better

* chore: rename wandb_run_id to wandb_name

* feat: add new recommendation and update config

* fix: indent and pop disabled env if project passed

* feat: test env set for wandb and recommendation

* feat: update to use wandb_name and allow id

* chore: add info to readme
 2023-12-04 22:17:25 +09:00
kallewoof
58ec8b1113 feature: loss watchdog for terminating training runs that are failing (#899) 
Co-authored-by: Karl-Johan Alm <kalle@gmail.com>
 2023-12-04 07:54:34 -05:00
Wing Lian
f544ab2bed don't compile deepspeed or bitsandbytes from source (#837) 2023-11-08 19:49:55 -05:00
Wing Lian
8b79ff0e94 fix eval_steps to be a sane default (#797) 
* fix eval_steps to be a sane default

* update docs for fractional eval_steps
 2023-10-27 22:36:30 -04:00
Wing Lian
9b43e7ea15 disable eval table w sample packing in examples (#778) 2023-10-23 09:18:44 -04:00
Wing Lian
2d8def68dc simplify by removing duplicate base_model_config (#772) 2023-10-23 01:42:38 -04:00
atgctg
ace70b33c6 Fix: lowercase True values in config (#713) 
* Fix: lowercase `True` values in config

* Fix: lowercase `True` values in config
 2023-10-10 21:32:20 +09:00
lukemarsden
295b2662e1 Get qlora mistral-7b fine tuning working on a single 4090 (#708) 2023-10-10 15:14:23 +09:00
NanoCode012
669f1d052c Fix: Higher vram usage for mistral and sample_packing (#691) 
* Fix: Higher vram usage for mistral and sample_packing

* chore: update comment

* chore: lint
 2023-10-06 12:33:43 -04:00
Abhishek Mishra
d4a88e4eca Adding qlora config for Mistral (#675) 
* Adding qlora config for Mistral

Contains fix for Mistral FA issue - ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call tokenizer.padding_side  = 'left' before tokenizing the input.

Fix for now is to set sample_packing: true and pad_to_sequence_len: true

* Renamed to qlora.yml
 2023-10-06 21:05:56 +09:00