Wing Lian 651b7a31fc fix double eos token for chatml (#1054) [skip ci] ...

* fix double eos token for chatml

* isolate fix to chatml conversation

* fix add special tokens to include rstrip

* add test for train_on_inputs for sharegpt

* don't use rstrip for chatml

2024-01-09 09:33:38 -05:00

..

test_sharegpt.py

fix double eos token for chatml (#1054) [skip ci]

2024-01-09 09:33:38 -05:00