Carsten Kragelund Jørgensen eb3a57eb17 Ignore generation/endgeneration tags when analyzing Jinja chat template (#2787) ...

* ignore generation/endgeneration tags

Axolotl handles calculating the mask for assistant turns on its own, and as such these tags are not needed, however currently the analyzer does not recognize them at all and throws an error.

* feat: add phi4 tokenizer test and unblock gemma2

* fix: improve template

* chore: refactor

* chore: lint

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
Co-authored-by: Wing Lian <wing@axolotl.ai>

2025-06-18 15:59:07 -04:00

..

cli

add mocks for loading datasets in cli train tests (#2497) [skip ci]

2025-04-11 09:51:59 -04:00

core

Data loader refactor (#2707)

2025-06-10 19:53:07 -04:00

e2e

bump transformers==4.52.4 (#2800) [skip ci]

2025-06-18 15:46:14 -04:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

remove deprecated wandb env var (#2751)

2025-06-03 14:04:15 -07:00

monkeypatch

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

patched

remove deprecated wandb env var (#2751)

2025-06-03 14:04:15 -07:00

prompt_strategies

Ignore generation/endgeneration tags when analyzing Jinja chat template (#2787)

2025-06-18 15:59:07 -04:00

__init__.py

fix: minor patches for multimodal (#2441)

2025-03-31 13:40:12 +07:00

conftest.py

repop cache (#2639)

2025-05-06 11:09:07 -04:00

constants.py

Add Exact Deduplication Feature to Preprocessing Pipeline (#2072)

2024-12-02 08:47:10 -05:00

hf_offline_utils.py

fix tokenizer overrides w gemma3 (#2488)

2025-04-05 01:25:44 -04:00

test_data.py

fix: minor patches for multimodal (#2441)

2025-03-31 13:40:12 +07:00

test_datasets.py

Data loader refactor (#2707)

2025-06-10 19:53:07 -04:00

test_dict.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_exact_deduplication.py

Data loader refactor (#2707)

2025-06-10 19:53:07 -04:00

test_expand_mask.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_loaders.py

models.py -> loaders/ module refactor (#2680)

2025-05-23 15:51:11 -04:00

test_lora.py

models.py -> loaders/ module refactor (#2680)

2025-05-23 15:51:11 -04:00

test_normalize_config.py

replace references to random 68m model w 135m smollm2 (#2570) [skip ci]

2025-04-28 10:08:07 -04:00

test_packed_batch_sampler.py

KD fix w/ online distillation (#2700) [skip ci]

2025-06-17 12:09:13 -04:00

test_packed_dataset.py

bump hf deps (#2735) [skip ci]

2025-06-05 07:20:33 -07:00

test_packed_pretraining.py

fix streaming packing test (#2454)

2025-03-29 08:30:06 -04:00

test_perplexity.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_prompt_tokenizers.py

remove deprecated wandb env var (#2751)

2025-06-03 14:04:15 -07:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_tokenizers.py

models.py -> loaders/ module refactor (#2680)

2025-05-23 15:51:11 -04:00

test_validation_dataset.py

Validation for Muon optimizer with DS/FSDP (#2464)

2025-04-01 09:39:12 -04:00