6 Commits

Author	SHA1	Message	Date
Wing Lian	d5dcf9c350	fix test fixture b/c hf trainer tokenization changed (#464)	2023-08-23 04:04:49 -04:00
Wing Lian	8cace80175	fix fixture for new tokenizer handling in transformers (#428)	2023-08-17 17:01:52 -04:00
Jan Philipp Harries	3392270544	experimental llama 2 chat support (#296) ... * experimental llama 2 chat support * few small fixes * llama2_chat * small fix to follow original implementation * small fixes and added fixtures/tests * fix -mixed up inference and finetuning conversations * args - small fix * small fix * small adjustment and warning * fix with pre-commit --------- Co-authored-by: Jan Philipp Harries <jpdus@users.noreply.github.com>	2023-08-06 17:40:52 -04:00
Wing Lian	9b8585dc70	fix packing so that concatenated sequences reset the attention	2023-05-31 11:38:52 -04:00
NanoCode012	b81c97ff76	Fix pre-commit for rebased files	2023-05-31 03:01:38 +09:00
Wing Lian	e6fdeb087f	add unit test for sharegpt tokenization	2023-05-30 10:28:17 -04:00