7 Commits

Author	SHA1	Message	Date
Johan Hansson	cd2cda3cda	fix pylint	2024-01-11 16:54:03 +01:00
Johan Hansson	4fa557889c	fix pre commit	2024-01-10 23:04:12 +01:00
Johan Hansson	45d82b7b86	clean up	2024-01-10 22:57:28 +01:00
Johan Hansson	37a934bdb3	clean up	2024-01-10 22:50:05 +01:00
Johan Hansson	0d1d00a363	draft for adding test for tokenizer	2024-01-10 22:42:58 +01:00
NanoCode012	043c3860cd	fix: train_on_inputs: true ignored for sharegpt (#1045) [skip ci] ... * fix: `train_on_inputs: true` ignored for sharegpt * enable unit test for train_on_inputs for sharegpt --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 23:00:09 -05:00
Wing Lian	651b7a31fc	fix double eos token for chatml (#1054) [skip ci] ... * fix double eos token for chatml * isolate fix to chatml conversation * fix add special tokens to include rstrip * add test for train_on_inputs for sharegpt * don't use rstrip for chatml	2024-01-09 09:33:38 -05:00