Wing Lian f30afe4544 misc sharegpt fixes (#723) ...

* support for sharegpt with assistant talking first, better masking of assistant token, allow remap of roles from dataset

* invalid role is actually not possible

* update tokenized fixture for corrected labels

2023-10-13 11:04:39 -04:00

..

e2e

Feat: Allow usage of native Mistral FA when no sample_packing (#669)

2023-10-04 20:40:47 +09:00

fixtures

misc sharegpt fixes (#723)

2023-10-13 11:04:39 -04:00

monkeypatch

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_data.py

Fix pretraining with iterable/streaming Dataset (#556)

2023-09-13 00:16:40 -04:00

test_dict.py

fix DefaultDict.__or__

2023-08-13 01:15:50 +00:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_prompt_tokenizers.py

misc sharegpt fixes (#723)

2023-10-13 11:04:39 -04:00

test_prompters.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_tokenizers.py

simplify load_tokenizer

2023-08-12 18:55:06 -07:00

test_validation.py

Fix(cfg): Add validation for save_strategy and eval_strategy (#633)

2023-09-28 10:14:41 +09:00