tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

feat: expose bnb kwargs (#1018)

Browse Source 
* feat: expose bnb kwargs

* chore: added examples and link per suggestion

* Uncomment defaults per suggestion for readability

Co-authored-by: Hamel Husain <hamel.husain@gmail.com>

---------

Co-authored-by: Hamel Husain <hamel.husain@gmail.com>
This commit is contained in:
NanoCode012
2023-12-29 18:16:26 +09:00
committed by GitHub
parent f6ecf14dd4
commit 41353d2ea0
2 changed files with 21 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -520,6 +520,14 @@ model_config:
type: # linear | dynamic
factor: # float
# optional overrides to the bnb 4bit quantization configuration
# https://huggingface.co/docs/transformers/main/main_classes/quantization#transformers.BitsAndBytesConfig
bnb_config_kwargs:
# These are default values
llm_int8_has_fp16_weight: false
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: true
# Whether you are training a 4-bit GPTQ quantized model
gptq: true