tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

don't strip the prompt for check since we don't strip to tokenize anymore (#650)

Browse Source
This commit is contained in:
Wing Lian
2023-09-28 12:21:51 -04:00
committed by GitHub
parent b2edaaeff6
commit 8662e8ffe8
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/axolotl/prompt_tokenizers.py
View File

@@ -81,7 +81,7 @@ class PromptTokenizingStrategy(abc.ABC):
self, prompt: str, add_eos_token: bool = True, strip_bos_token: bool = False self, prompt: str, add_eos_token: bool = True, strip_bos_token: bool = False
) -> BatchEncoding: ) -> BatchEncoding:
result: BatchEncoding result: BatchEncoding
if not prompt.strip(): if not prompt:
LOG.warning("Empty text requested for tokenization.") LOG.warning("Empty text requested for tokenization.")
result = BatchEncoding(data={"input_ids": [], "attention_mask": []}) result = BatchEncoding(data={"input_ids": [], "attention_mask": []})
else: else: