axolotl

Files

Wing Lian 782b6a4216 set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122 ) [skip ci]

* set fp16 to false if bf16, update bf16: auto in example YAMLs

* unset fp16 so that it fallsback properly if bf16 isn't available

* Update README.md [skip-ci]

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

* test that bf16 disables fp16

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

2024-01-22 18:44:01 -05:00

config.yml

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122 ) [skip ci]

2024-01-22 18:44:01 -05:00

mixtral.yml

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122 ) [skip ci]

2024-01-22 18:44:01 -05:00

qlora.yml

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122 ) [skip ci]

2024-01-22 18:44:01 -05:00

README.md

Update mistral/README.md (#647 )

2023-09-28 10:24:56 -04:00

README.md

Mistral 7B is a language model with a total of 7.3 billion parameters, showcasing a notable performance across a variety of benchmarks.

Fine Tune:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml

If you run into CUDA OOM, use deepspeed with config zero2.json:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml --deepspeed deepspeed/zero2.json