5 Commits

Author	SHA1	Message	Date
Wing Lian	4d09b42ee3	plain input/output prompt strategy w/o chat templates (#1346) ... * plain input/output prompt strategy w/o chat templates * disable duplicate code check * make sure to add an eos/eot token to the end of the output so it will stop * multi turn segement support and test	2024-03-04 16:25:16 -05:00
mhenrichsen	98b4762077	Feat/chatml add system message (#1117) ... * add system message to template * readme update * added code to register new system message * register chatml template for test --------- Co-authored-by: Mads Henrichsen <mads@BrbartiendeMads.lan> Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-25 08:24:27 +01:00
JohanWork	5439707489	Feat(test): Add tests for alpaca chatml prompt tokenizer (#1088) ... * draft for adding test for tokenizer * clean up * clean up * fix pre commit * fix pylint * Revert "fix pylint" This reverts commit cd2cda3cda. * add pylint exception for pytest fixture * update comments * Apply suggestions from code review Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> * update spelling and import promptstyle * reaname, restrucure * clean up * add fmt:on --------- Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>	2024-01-23 13:30:26 +09:00
NanoCode012	043c3860cd	fix: train_on_inputs: true ignored for sharegpt (#1045) [skip ci] ... * fix: `train_on_inputs: true` ignored for sharegpt * enable unit test for train_on_inputs for sharegpt --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 23:00:09 -05:00
Wing Lian	651b7a31fc	fix double eos token for chatml (#1054) [skip ci] ... * fix double eos token for chatml * isolate fix to chatml conversation * fix add special tokens to include rstrip * add test for train_on_inputs for sharegpt * don't use rstrip for chatml	2024-01-09 09:33:38 -05:00