tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

ensure that the bias is also in the correct dtype (#1848) [skip ci]

Browse Source 
* ensure that the bias is also in the correct dtype

* add nightly for dpo-qlora-fsdp
This commit is contained in:
Wing Lian
2024-08-22 11:45:00 -04:00
committed by GitHub
parent c3fc529bfc
commit 5b0b774e38
4 changed files with 115 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
examples/qwen2/qlora-fsdp.yaml
View File

@@ -72,4 +72,5 @@ fsdp_config:
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer
fsdp_state_dict_type: FULL_STATE_DICT
fsdp_sharding_strategy: FULL_SHARD
special_tokens: