Wing Lian f30afe4544 misc sharegpt fixes (#723) ...

* support for sharegpt with assistant talking first, better masking of assistant token, allow remap of roles from dataset

* invalid role is actually not possible

* update tokenized fixture for corrected labels

2023-10-13 11:04:39 -04:00

28 KiB

Raw Permalink Blame History

View Raw