update openllama and clean up paths

README.md

@@ -16,14 +16,14 @@
 
 ## Axolotl supports
 
-|          | fp16/fp32 | fp16/fp32 w/ lora | qlora | gptq | gptq w/ lora | gptq w/flash attention | flash attention | xformers attention |
+|          | fp16/fp32 | lora | qlora | gptq | gptq w/ lora | gptq w/flash attn | flash attn | xformers attn |
-|----------|:----------|:------------------|-------|------|:-------------|------------------------|-----------------|--------------------|
+|----------|:----------|:-----|-------|------|:-------------|-------------------|------------|---------------|
-| llama    | ✅         | ✅                 | ✅     | ✅    | ❓            | ✅                      | ✅               | ✅                  |
+| llama    | ✅         | ✅    | ✅     | ✅    | ❓            | ✅                 | ✅          | ✅             |
-| Pythia   | ✅         | ✅                 | ✅     | ❌    | ❓            | ❌                      | ❌               | ❓                  |
+| Pythia   | ✅         | ✅    | ✅     | ❌    | ❓            | ❌                 | ❌          | ❓             |
-| cerebras | ✅         | ✅                 | ✅     | ❌    | ❓            | ❌                      | ❌               | ✅                  |
+| cerebras | ✅         | ✅    | ✅     | ❌    | ❓            | ❌                 | ❌          | ✅             |
-| mpt      | ✅         | ❌                 | ❓     | ❌    | ❓            | ❌                      | ❌               | ❓                  |
+| mpt      | ✅         | ❌    | ❓     | ❌    | ❓            | ❌                 | ❌          | ❓             |
-| falcon   | ✅         | ✅                 | ✅     | ❌    | ❓            | ❌                      | ❌               | ✅                  |
+| falcon   | ✅         | ✅    | ✅     | ❌    | ❓            | ❌                 | ❌          | ✅             |
-| gpt-j    | ✅         | ✅                 | ✅     | ❌    | ❓            | ❌                      | ❓               | ✅                  |
+| gpt-j    | ✅         | ✅    | ✅     | ❌    | ❓            | ❌                 | ❓          | ✅             |
 
 
 ## Quickstart ⚡

examples/openllama-3b/README.md

@@ -0,0 +1,16 @@
+# openllama-3b
+
+Basic full tune
+```shell
+accelerate launch scripts/finetune.py examples/qlora-openllama-3b/config.yml
+```
+
+LoRA
+```shell
+accelerate launch scripts/finetune.py examples/qlora-openllama-3b/lora.yml
+```
+
+QLoRA
+```shell
+accelerate launch scripts/finetune.py examples/qlora-openllama-3b/qlora.yml
+```

examples/openllama-3b/lora.yml

@@ -1,5 +1,5 @@
-base_model: openlm-research/open_llama_3b_600bt_preview
+base_model: openlm-research/open_llama_3b
-base_model_config: openlm-research/open_llama_3b_600bt_preview
+base_model_config: openlm-research/open_llama_3b
 model_type: LlamaForCausalLM
 tokenizer_type: LlamaTokenizer
 load_in_8bit: true

examples/openllama-3b/qlora.yml

@@ -1,5 +1,5 @@
-base_model: openlm-research/open_llama_3b_600bt_preview
+base_model: openlm-research/open_llama_3b
-base_model_config: openlm-research/open_llama_3b_600bt_preview
+base_model_config: openlm-research/open_llama_3b
 model_type: LlamaForCausalLM
 tokenizer_type: LlamaTokenizer
 load_in_8bit: false

examples/qlora-openllama-3b/README.md

@@ -1,6 +0,0 @@
-# qlora-openllama-3b
-
-```shell
-accelerate launch scripts/finetune.py examples/qlora-openllama-3b/config.yml
-
-```