Hamel Husain f1de29dd1e Respect sequence_len in config for type: llama2_chat (#926) ...

* Respect sequence_len in config for `type: llama2_chat`

It was hardcoded to `4096` I am not sure why?  This updates it to pull from the config. 

cc: @winglian

* Update llama2_chat.py

* apply black formatting

* fix tokenizer

* update test data

* lint fixtures

2023-12-12 09:39:22 -08:00

..

alpaca

fix packing so that concatenated sequences reset the attention

2023-05-31 11:38:52 -04:00

conversation.json

add unit test for sharegpt tokenization

2023-05-30 10:28:17 -04:00

conversation.missingturns.json

better handling and logging of empty sharegpt turns (#603)

2023-09-22 16:13:42 -04:00

conversation.tokenized_llama2chat.json

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

conversation.tokenized.json

misc sharegpt fixes (#723)

2023-10-13 11:04:39 -04:00