transformers v5 upgrade (#3272)

* Prepare for transformers v5 upgrade * fix hf cli * update for hf hub changes * fix tokenizer apply_chat_template args * remap include_tokens_per_second * fix tps * handle migration for warmup * use latest hf hub * Fix scan -> ls * fix import * fix for renaming of mistral common tokenizer -> backend * update for fixed tokenziation for llama * Skip phi35 tests for now * remove mistral patch fixed upstream in huggingface/transformers#41439 * use namespacing for patch * don't rely on sdist for e2e tests for now * run modal ci without waiting too * Fix dep for ci * fix imports * Fix fp8 check * fsdp2 fixes * fix version handling * update fsdp version tests for new v5 behavior * Fail multigpu tests after 3 failures * skip known v5 broken tests for now and cleanup * bump deps * unmark skipped test * re-enable test_fsdp_qlora_prequant_packed test * increase multigpu ci timeout * skip broken gemma3 test * reduce timout back to original 120min now that the hanging test is skipped * fix for un-necessary collator for pretraining with bsz=1 * fix: safe_serialization deprecated in transformers v5 rc01 (#3318) * torch_dtype deprecated * load model in float32 for consistency with tests * revert some test fixtures back * use hf cache ls instead of scan * don't strip fsdp_version more fdsp_Version fixes for v5 fix version in fsdp_config fix aliasing fix fsdp_version check check fsdp_version is 2 in both places * Transformers v5 rc2 (#3347) * bump dep * use latest fbgemm, grab model config as part of fixture, un-skip test * import AutoConfig * don't need more problematic autoconfig when specifying config.json manually * add fixtures for argilla ultrafeedback datasets * download phi4-reasoning * fix arg * update tests for phi fast tokenizer changes * use explicit model types for gemma3 --------- Co-authored-by: Wing Lian <wing@axolotl.ai> * fix: AutoModelForVision2Seq -> AutoModelForImageTextToText * chore: remove duplicate * fix: attempt fix gemma3 text mode * chore: lint * ga release of v5 * need property setter for name_or_path for mistral tokenizer * vllm not compatible with transformers v5 * setter for chat_template w mistral too --------- Co-authored-by: NanoCode012 <nano@axolotl.ai> Co-authored-by: salman <salman.mohammadi@outlook.com>
2026-01-27 17:08:24 -05:00
parent a531e9d946
commit fc4e37920b
74 changed files with 262 additions and 309 deletions
--- a/tests/monkeypatch/test_mistral_tokenizer_patch.py
+++ b/tests/monkeypatch/test_mistral_tokenizer_patch.py
@@ -1,35 +0,0 @@
-"""Integration tests for MistralCommonTokenizer patches."""
-
-import pytest
-
-
-class TestMistralTokenizerPatchIntegration:
-    """Test MistralCommonTokenizer patch integration."""
-
-    @pytest.mark.integration
-    def test_mistral_tokenizer_image_patch(self):
-        """Test that MistralCommonTokenizer image patch can be applied."""
-        try:
-            from transformers.tokenization_mistral_common import MistralCommonTokenizer
-        except ImportError:
-            pytest.skip("MistralCommonTokenizer not available")
-
-        from axolotl.monkeypatch.models.mistral3.mistral_common_tokenizer import (
-            apply_mistral_tokenizer_image_patch,
-        )
-
-        # Store original method
-        original_apply_chat_template = MistralCommonTokenizer.apply_chat_template
-
-        # Apply patch
-        apply_mistral_tokenizer_image_patch()
-
-        # Verify patch was applied
-        assert (
-            MistralCommonTokenizer.apply_chat_template != original_apply_chat_template
-        ), "apply_chat_template was not patched"
-
-        # Verify the method is still callable
-        assert callable(MistralCommonTokenizer.apply_chat_template), (
-            "Patched method is not callable"
-        )