axolotl

Files

Wing Lian 7fabc4d95e Mixtral official (#942 )

* multipack support for official mixtral implementation

* fix patch to load multipack for mixtral

* chore: lint

2023-12-11 23:44:33 -05:00

config.yml

2023-12-04 22:17:25 +09:00

mixtral.yml

Mixtral official (#942 )

2023-12-11 23:44:33 -05:00

qlora.yml

2023-12-04 22:17:25 +09:00

README.md

2023-09-28 10:24:56 -04:00

Mistral 7B is a language model with a total of 7.3 billion parameters, showcasing a notable performance across a variety of benchmarks.

Fine Tune:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml

If you run into CUDA OOM, use deepspeed with config zero2.json:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml --deepspeed deepspeed/zero2.json