tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Add Readme, Clean up comments

Browse Source
This commit is contained in:
jphillips
2023-05-29 14:35:58 -05:00
parent f1fbf666f7
commit ac85c0ed36
2 changed files with 10 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
examples/qlora-openllama-3b/README.md Normal file
View File

@@ -0,0 +1,6 @@
# qlora-openllama-3b
```shell
accelerate launch scripts/finetune.py examples/lora-openllama-3b
```

8
examples/qlora-openllama-3b.yml/config.yml → examples/qlora-openllama-3b/config.yml
View File

@@ -26,15 +26,15 @@ wandb_watch:
wandb_run_id: wandb_run_id:
wandb_log_model: wandb_log_model:
output_dir: ./qlora-out output_dir: ./qlora-out
batch_size: 1 batch_size: 4
micro_batch_size: 4 # Changed from 1 to 4 micro_batch_size: 4
num_epochs: 2 # Changed from 4 to 2 num_epochs: 2
optimizer: paged_adamw_32bit optimizer: paged_adamw_32bit
torchdistx_path: torchdistx_path:
lr_scheduler: cosine lr_scheduler: cosine
learning_rate: 0.0002 learning_rate: 0.0002
train_on_inputs: false train_on_inputs: false
group_by_length: true # Changed from false to true group_by_length: true
bf16: true bf16: true
fp16: false fp16: false
tf32: true tf32: true