tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Fix Deepspeed loading (#950)

Browse Source 
* add check for zero3

* freeze parameters

* fixes for deepspeed loading

* fix model parameter check

* unfrozen parameters in example mixtral and logging when unfreezing
This commit is contained in:
Wing Lian
2023-12-13 16:03:23 -05:00
committed by GitHub
parent f1f60cb5b2
commit 5ea3aa31f0
7 changed files with 96 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
examples/mistral/mixtral.yml
View File

@@ -14,6 +14,15 @@ dataset_prepared_path: last_run_prepared
val_set_size: 0.0
output_dir: ./qlora-out
## You can optionally freeze the entire model and unfreeze a subset of parameters
unfrozen_parameters:
# - lm_head.*
# - model.embed_tokens.*
# - model.layers.2[0-9]+.block_sparse_moe.gate.*
# - model.layers.2[0-9]+.block_sparse_moe.experts.*
# - model.layers.3[0-9]+.block_sparse_moe.gate.*
# - model.layers.3[0-9]+.block_sparse_moe.experts.*
adapter: qlora
lora_model_dir: