fc4e37920b
* Prepare for transformers v5 upgrade * fix hf cli * update for hf hub changes * fix tokenizer apply_chat_template args * remap include_tokens_per_second * fix tps * handle migration for warmup * use latest hf hub * Fix scan -> ls * fix import * fix for renaming of mistral common tokenizer -> backend * update for fixed tokenziation for llama * Skip phi35 tests for now * remove mistral patch fixed upstream in huggingface/transformers#41439 * use namespacing for patch * don't rely on sdist for e2e tests for now * run modal ci without waiting too * Fix dep for ci * fix imports * Fix fp8 check * fsdp2 fixes * fix version handling * update fsdp version tests for new v5 behavior * Fail multigpu tests after 3 failures * skip known v5 broken tests for now and cleanup * bump deps * unmark skipped test * re-enable test_fsdp_qlora_prequant_packed test * increase multigpu ci timeout * skip broken gemma3 test * reduce timout back to original 120min now that the hanging test is skipped * fix for un-necessary collator for pretraining with bsz=1 * fix: safe_serialization deprecated in transformers v5 rc01 (#3318) * torch_dtype deprecated * load model in float32 for consistency with tests * revert some test fixtures back * use hf cache ls instead of scan * don't strip fsdp_version more fdsp_Version fixes for v5 fix version in fsdp_config fix aliasing fix fsdp_version check check fsdp_version is 2 in both places * Transformers v5 rc2 (#3347) * bump dep * use latest fbgemm, grab model config as part of fixture, un-skip test * import AutoConfig * don't need more problematic autoconfig when specifying config.json manually * add fixtures for argilla ultrafeedback datasets * download phi4-reasoning * fix arg * update tests for phi fast tokenizer changes * use explicit model types for gemma3 --------- Co-authored-by: Wing Lian <wing@axolotl.ai> * fix: AutoModelForVision2Seq -> AutoModelForImageTextToText * chore: remove duplicate * fix: attempt fix gemma3 text mode * chore: lint * ga release of v5 * need property setter for name_or_path for mistral tokenizer * vllm not compatible with transformers v5 * setter for chat_template w mistral too --------- Co-authored-by: NanoCode012 <nano@axolotl.ai> Co-authored-by: salman <salman.mohammadi@outlook.com>
48 lines
2.8 KiB
Python
48 lines
2.8 KiB
Python
"""unit tests for perplexity eval callback"""
|
|
|
|
from pytest import fixture
|
|
from transformers.models.auto.modeling_auto import AutoModelForCausalLM
|
|
from transformers.models.auto.tokenization_auto import AutoTokenizer
|
|
|
|
from axolotl.utils.callbacks.perplexity import Perplexity
|
|
|
|
MODEL_NAME = "HuggingFaceTB/SmolLM2-135M"
|
|
|
|
|
|
@fixture()
|
|
def metric(tokenizer):
|
|
return Perplexity(tokenizer=tokenizer, max_seq_len=512)
|
|
|
|
|
|
@fixture()
|
|
def model():
|
|
return AutoModelForCausalLM.from_pretrained(
|
|
MODEL_NAME, trust_remote_code=True, dtype="float32"
|
|
)
|
|
|
|
|
|
@fixture()
|
|
def tokenizer():
|
|
tokenizer_ = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
|
|
tokenizer_.add_special_tokens({"pad_token": "<|endoftext|>"})
|
|
return tokenizer_
|
|
|
|
|
|
def test_perplexity_longer_than_stride(model, metric):
|
|
# taken from https://huggingface.co/datasets/roneneldan/TinyStories
|
|
sample_text = """
|
|
Once upon a time, there was a little car named Beep. Beep loved to go fast and play in the sun. Beep was a healthy car because he always had good fuel. Good fuel made Beep happy and strong. One day, Beep was driving in the park when he saw a big tree. The tree had many leaves that were falling. Beep liked how the leaves fall and wanted to play with them. Beep drove under the tree and watched the leaves fall on him. He laughed and beeped his horn. Beep played with the falling leaves all day. When it was time to go home, Beep knew he needed more fuel. He went to the fuel place and got more healthy fuel. Now, Beep was ready to go fast and play again the next day. And Beep lived happily ever after.
|
|
One day, a little fish named Fin was swimming near the shore. He saw a big crab and wanted to be friends. "Hi, I am Fin. Do you want to play?" asked the little fish. The crab looked at Fin and said, "No, I don't want to play. I am cold and I don't feel fine." Fin felt sad but wanted to help the crab feel better. He swam away and thought of a plan. He remembered that the sun could make things warm. So, Fin swam to the top of the water and called to the sun, "Please, sun, help my new friend feel fine and not freeze!" The sun heard Fin's call and shone its warm light on the shore. The crab started to feel better and not so cold. He saw Fin and said, "Thank you, little fish, for making me feel fine. I don't feel like I will freeze now. Let's play together!" And so, Fin and the crab played and became good friends.
|
|
"""
|
|
result = metric.compute(model, [sample_text])
|
|
ppl = result["score"]
|
|
assert round(ppl, 2) == 7.41
|
|
|
|
|
|
def test_perplexity_short(model, metric):
|
|
# taken from https://huggingface.co/datasets/roneneldan/TinyStories
|
|
sample_text = "Once upon a time, there was a little car named Beep. Beep loved to go fast and play in the sun."
|
|
result = metric.compute(model, [sample_text])
|
|
ppl = result["score"]
|
|
assert round(ppl, 2) == 10.33
|