tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

load model faster w low_cpu_mem_usage

Browse Source
This commit is contained in:
Wing Lian
2023-09-08 02:06:12 -04:00
parent fb8ee37ca6
commit 9c52a83403
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/axolotl/utils/models.py
View File

@@ -340,6 +340,7 @@ def load_model(
base_model, base_model,
config=config, config=config,
trust_remote_code=cfg.trust_remote_code or False, trust_remote_code=cfg.trust_remote_code or False,
low_cpu_mem_usage=True,
).half() ).half()
model = tp.tensor_parallel(model, sharded=False) model = tp.tensor_parallel(model, sharded=False)
else: else: