tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

don't resize embeddings if it's already large enough (#577)

Browse Source 
* don't resize embeddings if it's already large enough

* make sure to tie weights, even if we aren't resizing
This commit is contained in:
Wing Lian
2023-09-15 15:47:09 -04:00
committed by GitHub
parent 12a2dbbc2c
commit 360788296a
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/axolotl/utils/models.py
View File

@@ -302,7 +302,10 @@ def load_model(
if cfg.resize_token_embeddings_to_32x if cfg.resize_token_embeddings_to_32x
else len(tokenizer) else len(tokenizer)
) )
model.resize_token_embeddings(embeddings_len) if model.get_input_embeddings().num_embeddings < embeddings_len:
model.resize_token_embeddings(embeddings_len)
else:
model.tie_weights()
if ( if (
hasattr(model.config, "max_position_embeddings") hasattr(model.config, "max_position_embeddings")