tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

fix: switch to using the HuggingFace Transformers NEFT implementation (#941)

Browse Source 
* fix: switch to using the HuggingFace Transformers NEFT implementation

* linter

* add support for noisy_embedding_alpha with a warning about it being renamed

* restore pre/posttrain_hooks

* move validation of NEFT noise alpha into validate_config()

* linter
This commit is contained in:
kallewoof
2023-12-14 07:15:34 +09:00
committed by GitHub
parent 5ea3aa31f0
commit ef24342538
5 changed files with 23 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
src/axolotl/train.py
View File

@@ -16,7 +16,6 @@ from transformers.deepspeed import is_deepspeed_zero3_enabled
from axolotl.common.cli import TrainerCliArgs
from axolotl.logging_config import configure_logging
from axolotl.monkeypatch import neft_embeddings
from axolotl.utils.dict import DictDefault
from axolotl.utils.freeze import freeze_parameters_except
from axolotl.utils.models import load_model, load_tokenizer
@@ -180,21 +179,19 @@ def train(
return model, tokenizer
def pretrain_hooks(cfg, trainer):
def pretrain_hooks(_cfg, _trainer):
"""
Run hooks right before kicking off the training
:param cfg:
:param trainer:
:return:
"""
neft_embeddings.pretrain_hook(cfg, trainer)
def post_train_hooks(cfg, trainer):
def post_train_hooks(_cfg, _trainer):
"""
Run hooks right after training completes
:param cfg:
:param trainer:
:return:
"""
neft_embeddings.post_train_hook(cfg, trainer)