Files

JinK 553c80f79a streaming multipack for pretraining dataset (#959 )

* [Feat] streaming multipack

* WIP make continued pretraining work w multipack

* fix up hadrcoding, lint

* fix dict check

* update test for updated pretraining multipack code

* fix hardcoded data collator fix for multipack pretraining

* fix the collator to be the max length for multipack pretraining

* don't bother with latest tag for test

* cleanup docker build/test

---------

Co-authored-by: jinwonkim93@github.com <jinwonkim>
Co-authored-by: Wing Lian <wing.lian@gmail.com>

2024-01-05 22:13:21 -05:00

lora.yml

fix: lint (#1037 )

2024-01-03 10:23:44 -05:00

pretrain.yml

streaming multipack for pretraining dataset (#959 )

2024-01-05 22:13:21 -05:00

qlora.yml

fix: lint (#1037 )

2024-01-03 10:23:44 -05:00

README.md

added tiny llama examples for lora and qlora (#1027 )

2024-01-02 20:00:37 -05:00

README.md

Overview

This is a simple example of how to finetune TinyLlama1.1B using either lora or qlora:

LoRa:

accelerate launch -m axolotl.cli.train examples/tiny-llama/lora.yml

qLoRa:

accelerate launch -m axolotl.cli.train examples/tiny-llama/qlora.yml

Both take about 10 minutes to complete on a 4090.