tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Add validation for TP with models with tied embeddings (#2999)

Browse Source 
* add validation for tp + tied embeddings models

* fix logic and messaging

* add additional guard for null tp size
This commit is contained in:
Wing Lian
2025-08-01 13:58:16 -04:00
committed by GitHub
parent 01a6bd1a0e
commit 7c3b428f23
1 changed files with 11 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
src/axolotl/loaders/utils.py
View File

@@ -131,6 +131,17 @@ def check_model_config(cfg: DictDefault, model_config: PretrainedConfig):
f"Please include [{lora_modules_to_save_joined}] in `lora_modules_to_save`." f"Please include [{lora_modules_to_save_joined}] in `lora_modules_to_save`."
) )
if (
cfg.tensor_parallel_size
and cfg.tensor_parallel_size > 1
and hasattr(model_config, "tie_word_embeddings")
and model_config.tie_word_embeddings
):
raise ValueError(
"Tensor parallelism is incompatible with models configured with `tie_word_embeddings` enabled. "
"Please use a model without `tie_word_embeddings`, or disable tensor parallelism."
)
def load_model_config(cfg: DictDefault) -> PretrainedConfig | addict.Dict: def load_model_config(cfg: DictDefault) -> PretrainedConfig | addict.Dict:
"""Loads and configures a model configuration from HuggingFace or local sources. """Loads and configures a model configuration from HuggingFace or local sources.