axolotl

Files

Wing Lian 5ea3aa31f0 Fix Deepspeed loading (#950 )

* add check for zero3

* freeze parameters

* fixes for deepspeed loading

* fix model parameter check

* unfrozen parameters in example mixtral and logging when unfreezing

2023-12-13 16:03:23 -05:00

config.yml

new evals_per_epoch and saves_per_epoch to make things cleaner (#944 )

2023-12-12 15:35:23 -05:00

mixtral.yml

Fix Deepspeed loading (#950 )

2023-12-13 16:03:23 -05:00

qlora.yml

new evals_per_epoch and saves_per_epoch to make things cleaner (#944 )

2023-12-12 15:35:23 -05:00

README.md

Update mistral/README.md (#647 )

2023-09-28 10:24:56 -04:00

README.md

Mistral 7B is a language model with a total of 7.3 billion parameters, showcasing a notable performance across a variety of benchmarks.

Fine Tune:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml

If you run into CUDA OOM, use deepspeed with config zero2.json:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml --deepspeed deepspeed/zero2.json