Merge branch 'main' into cj_tokenizer_default_prompt_template

2024-10-10 16:11:07 +07:00
parent 88658c0570
commit b8056d04d9
30 changed files with 1815 additions and 318 deletions
--- a/docs/input_output.qmd
+++ b/docs/input_output.qmd
@@ -205,7 +205,7 @@ ds = load_from_disk(f'last_run_prepared/{directory[0]}/')
    hi there!.  goodbye  farewell</s>
 ```

-We can check that the right tokens are ingored by comparing the labels
+We can check that the right tokens are ignored by comparing the labels
 to each token:

 ```python