axolotl

Files

Wing Lian 35f9b0f149 update to latest transformers for mixstral support (#929 )

* update to latest transformers for mixstral support

* pin transformers

* fix typo

2023-12-10 10:32:27 -05:00

config.yml

2023-12-04 22:17:25 +09:00

mixtral.yml

2023-12-10 10:32:27 -05:00

qlora.yml

2023-12-04 22:17:25 +09:00

README.md

2023-09-28 10:24:56 -04:00

Mistral 7B is a language model with a total of 7.3 billion parameters, showcasing a notable performance across a variety of benchmarks.

Fine Tune:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml

If you run into CUDA OOM, use deepspeed with config zero2.json:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml --deepspeed deepspeed/zero2.json