Wing Lian 85cf4f8e2c better handling of empty input ids when tokenizing (#395) ...

* better handling of empty input ids when tokenizing

* Add warning if tokenizer resulted in empty result

* fix len comparison for linter

2023-08-15 01:09:59 -04:00

..

monkeypatch

fix check for flash attn branching (#377)

2023-08-12 22:48:08 -04:00

prompt_strategies

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

utils

add utils.data.prepare_dataset

2023-08-14 21:28:29 -07:00

__init__.py

WIP for axolotl trainer

2023-04-14 00:20:05 -04:00

convert.py

Lint convert.py

2023-05-31 02:53:53 +09:00

datasets.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

logging_config.py

misc fixes

2023-07-17 03:00:27 -04:00

prompt_tokenizers.py

better handling of empty input ids when tokenizing (#395)

2023-08-15 01:09:59 -04:00

prompters.py

Error msg for sharegpt if conv has less than 2 msg (#379)

2023-08-14 17:40:40 +09:00