Commit Graph

3 Commits

Author SHA1 Message Date
Wing Lian
9b8585dc70 fix packing so that concatenated sequences reset the attention 2023-05-31 11:38:52 -04:00
NanoCode012
b81c97ff76 Fix pre-commit for rebased files 2023-05-31 03:01:38 +09:00
Wing Lian
e6fdeb087f add unit test for sharegpt tokenization 2023-05-30 10:28:17 -04:00