3 Commits

Author	SHA1	Message	Date
Wing Lian	61f44f311e	fix packing for tokenizers that don't use a bos_token when the bos token and eos token are both the same	2023-06-13 21:26:13 -04:00
Wing Lian	0136f510f2	don't worry about duplicate code here	2023-05-31 12:05:43 -04:00
Wing Lian	9b8585dc70	fix packing so that concatenated sequences reset the attention	2023-05-31 11:38:52 -04:00