tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

update docs for tokenizer_legacy (#401)

Browse Source 
* update docs for tokenizer_legacy

* add default info
This commit is contained in:
Wing Lian
2023-08-15 09:34:42 -04:00
committed by GitHub
parent 7ad37cb6d7
commit 47961fdb8b
1 changed files with 2 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -326,6 +326,8 @@ tokenizer_type: AutoTokenizer
trust_remote_code: trust_remote_code:
# use_fast option for tokenizer loading from_pretrained, default to True # use_fast option for tokenizer loading from_pretrained, default to True
tokenizer_use_fast: tokenizer_use_fast:
# Whether to use the legacy tokenizer setting, defaults to True
tokenizer_legacy:
# resize the model embeddings when new tokens are added to multiples of 32 # resize the model embeddings when new tokens are added to multiples of 32
# this is reported to improve training speed on some models # this is reported to improve training speed on some models
resize_token_embeddings_to_32x: resize_token_embeddings_to_32x: