add unit test for sharegpt tokenization

Browse Source

...

This commit is contained in:

Wing Lian

2023-05-30 10:28:17 -04:00

parent e43bcc6c4f

commit e6fdeb087f

3 changed files with 98 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1

tests/fixtures/conversation.tokenized.json vendored Normal file

File diff suppressed because one or more lines are too long