JohanWork 5439707489 Feat(test): Add tests for alpaca chatml prompt tokenizer (#1088) ...

* draft for adding test for tokenizer

* clean up

* clean up

* fix pre commit

* fix pylint

* Revert "fix pylint"

This reverts commit cd2cda3cda.

* add pylint exception for pytest fixture

* update comments

* Apply suggestions from code review

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

* update spelling and import promptstyle

* reaname, restrucure

* clean up

* add fmt:on

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

2024-01-23 13:30:26 +09:00

..

core

add gptneox embeddings, fix phi2 inputs, also fix the casting (#1083)

2024-01-10 22:32:43 -05:00

e2e

Falcon embeddings (#1149) [skip docker]

2024-01-22 21:01:42 -05:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

monkeypatch

Multipack simplify for Mixtral (#1142)

2024-01-18 16:23:49 -05:00

prompt_strategies

Feat(test): Add tests for alpaca chatml prompt tokenizer (#1088)

2024-01-23 13:30:26 +09:00

utils

Add shifted sparse attention (#973) [skip-ci]

2024-01-18 10:16:07 -05:00

test_data.py

Fix pretraining with iterable/streaming Dataset (#556)

2023-09-13 00:16:40 -04:00

test_dict.py

fix DefaultDict.__or__

2023-08-13 01:15:50 +00:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_normalize_config.py

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

streaming multipack for pretraining dataset (#959)

2024-01-05 22:13:21 -05:00

test_prompt_tokenizers.py

fix mistral prompt assembly (#982)

2023-12-21 08:00:55 -08:00

test_prompters.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_tokenizers.py

Feat: Warns to add to modules_to_save when adding tokens or switching special_tokens (#787)

2023-12-22 21:49:07 +09:00

test_validation.py

Deprecate max packed sequence len (#1141)

2024-01-20 05:11:50 -05:00