tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Feat/chatml add system message (#1117)

Browse Source 
* add system message to template

* readme update

* added code to register new system message

* register chatml template for test

---------

Co-authored-by: Mads Henrichsen <mads@BrbartiendeMads.lan>
Co-authored-by: Wing Lian <wing.lian@gmail.com>
This commit is contained in:
mhenrichsen
2024-01-25 08:24:27 +01:00
committed by GitHub
parent ee0b5f60e5
commit 98b4762077
7 changed files with 42 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tests/prompt_strategies/test_sharegpt.py
View File

@@ -7,9 +7,14 @@ from tokenizers import AddedToken
from transformers import AutoTokenizer
from axolotl.datasets import TokenizedPromptDataset
from axolotl.prompt_strategies.sharegpt import SimpleShareGPTPromptTokenizingStrategy
from axolotl.prompt_strategies.sharegpt import (
SimpleShareGPTPromptTokenizingStrategy,
register_chatml_template,
)
from axolotl.prompters import ShareGPTPrompterV2
register_chatml_template()
@pytest.fixture(name="sharegpt_dataset")
def fixture_sharegpt_dataset():