tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

fix lora target module, require explicit flash attention, fix min logging steps, don't use adam8bit for int4, hash prepared datasets, support hf hub datasets

Browse Source
This commit is contained in:
Wing Lian
2023-04-17 18:01:12 -04:00
parent 4131183115
commit 87e073d0de
4 changed files with 93 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
configs/llama_65B_alpaca.yml
View File

@@ -21,7 +21,7 @@ lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
- q_proj
- w_proj
- v_proj
lora_fan_in_fan_out: false
wandb_project: llama-65b-lora
wandb_watch: