3 Commits

Author	SHA1	Message	Date
Johan Hansson	0d1d00a363	draft for adding test for tokenizer	2024-01-10 22:42:58 +01:00
NanoCode012	043c3860cd	fix: train_on_inputs: true ignored for sharegpt (#1045) [skip ci] ... * fix: `train_on_inputs: true` ignored for sharegpt * enable unit test for train_on_inputs for sharegpt --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 23:00:09 -05:00
Wing Lian	651b7a31fc	fix double eos token for chatml (#1054) [skip ci] ... * fix double eos token for chatml * isolate fix to chatml conversation * fix add special tokens to include rstrip * add test for train_on_inputs for sharegpt * don't use rstrip for chatml	2024-01-09 09:33:38 -05:00