NanoCode012 7099343c56 feat: add eos_tokens and train_on_eot for chat_template EOT parsing (#2364) ...

* feat: add eos_tokens and train_on_eot for chat_template EOT parsing

* fix: comments

* chore: add some examples of tokens

* feat: add new potential errors for chat_template to faq

* feat: add examples for EOT handling

* fix: change error to warning for missing EOS

* fix: warning typo

* feat: add tests for eot token handling

* fix: remove broken caplog capture in test

* fix: chattemplate strategy with kd missing eot changes

2025-04-28 10:11:20 -04:00

6.2 KiB

Raw Blame History

View Raw