tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

add tp_size in config doc

Browse Source
This commit is contained in:
Sunny Liu
2025-02-20 00:01:59 -05:00
committed by Sung Ching Liu
parent 64adbf1a15
commit 984be14147
2 changed files with 3 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/config.qmd
View File

@@ -78,6 +78,9 @@ tf32: true # require >=ampere
bfloat16: true # require >=ampere
float16: true
# Tensor parallel
tp_size: 1 # should be set to the number of cuda devices available
# Limit the memory for all available GPUs to this amount (if an integer, expressed in gigabytes); default: unset
gpu_memory_limit: 20GiB
# Do the LoRA/PEFT loading on CPU -- this is required if the base model is so large it takes up most or all of the available GPU VRAM, e.g. during a model and LoRA merge