mhenrichsen 575e5f28ec Update Tokenizer Overrides Handling in models.py (#1549) ...

* override special tokens mock code

* fix(doc): remove duplicate config

* feat: replace added_tokens in tokenizer and add test

* make sure to run tokenizer modification on rank 0 only

* use is local main process instead

* feat: rename config

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
Co-authored-by: Wing Lian <wing@axolotl.ai>

2025-03-05 11:15:12 -05:00

27 KiB

Raw Permalink Blame History

View Raw