Wing Lian eea2731a5e add streaming dataset support for pretraining datasets

2023-06-10 14:23:56 -04:00

..

fixtures

fix packing so that concatenated sequences reset the attention

2023-05-31 11:38:52 -04:00

test_dict.py

Ignore unsupported-binary-operation

2023-05-31 02:53:53 +09:00

test_packed_dataset.py

don't worry about duplicate code here

2023-05-31 12:05:43 -04:00

test_prompt_tokenizers.py

don't worry about duplicate code here

2023-05-31 12:05:43 -04:00

test_prompters.py

Lint test_prompters

2023-05-31 02:53:53 +09:00

test_validation.py

add streaming dataset support for pretraining datasets

2023-06-10 14:23:56 -04:00