tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Ignore generation/endgeneration tags when analyzing Jinja chat template (#2787)

Browse Source 
* ignore generation/endgeneration tags

Axolotl handles calculating the mask for assistant turns on its own, and as such these tags are not needed, however currently the analyzer does not recognize them at all and throws an error.

* feat: add phi4 tokenizer test and unblock gemma2

* fix: improve template

* chore: refactor

* chore: lint

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
Co-authored-by: Wing Lian <wing@axolotl.ai>
This commit is contained in:
Carsten Kragelund Jørgensen
2025-06-18 21:59:07 +02:00
committed by GitHub
parent 34da391391
commit eb3a57eb17
5 changed files with 46 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
tests/prompt_strategies/conftest.py
View File

@@ -143,6 +143,12 @@ def fixture_phi35_tokenizer():
return tokenizer
@pytest.fixture(name="phi4_tokenizer", scope="session", autouse=True)
def fixture_phi4_tokenizer():
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-reasoning")
return tokenizer
@pytest.fixture(name="gemma2_tokenizer", scope="session", autouse=True)
def fixture_gemma2_tokenizer():
tokenizer = AutoTokenizer.from_pretrained("mlx-community/gemma-2-9b-it-4bit")