axolotl

Files

Motoki Wu 98c25e15cb Add ORPO example and e2e test (#1572 )

* add example for mistral orpo

* sample_packing: false for orpo

* go to load_dataset (since load_rl_datasets require a transfom_fn, which only dpo uses currently)

2024-04-27 12:07:06 -04:00

bigstral-ds-zero3.yaml

DBRX Model Support (#1462 )

2024-04-12 09:02:36 -04:00

config.yml

chore(config): refactor old mistral config (#1435 )

2024-03-25 12:00:44 +09:00

lora-mps.yml

Mps mistral lora (#1292 ) [skip ci]

2024-02-26 22:39:57 -05:00

lora.yml

chore(config): refactor old mistral config (#1435 )

2024-03-25 12:00:44 +09:00

mistral-qlora-fsdp.yml

DBRX Model Support (#1462 )

2024-04-12 09:02:36 -04:00

mistral-qlora-orpo.yml

Add ORPO example and e2e test (#1572 )

2024-04-27 12:07:06 -04:00

mixtral_22.yml

fix broken linting (#1541 )

2024-04-19 01:03:04 -04:00

mixtral-8x22b-qlora-fsdp.yml

DBRX Model Support (#1462 )

2024-04-12 09:02:36 -04:00

mixtral-qlora-fsdp.yml

DBRX Model Support (#1462 )

2024-04-12 09:02:36 -04:00

mixtral.yml

Train parameters exclusively in specific ranges (#1390 )

2024-03-14 11:05:42 -04:00

qlora.yml

chore(config): refactor old mistral config (#1435 )

2024-03-25 12:00:44 +09:00

README.md

Mixtral fixes 20240124 (#1192 ) [skip ci]

2024-01-24 14:59:57 -05:00

README.md

Mistral 7B is a language model with a total of 7.3 billion parameters, showcasing a notable performance across a variety of benchmarks.

Fine Tune:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml

If you run into CUDA OOM, use deepspeed with config zero2.json:

accelerate launch -m axolotl.cli.train examples/mistral/config.yml --deepspeed deepspeed_configs/zero2.json