tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Correct typos in datasets.py (#639)

Browse Source
This commit is contained in:
Felix Yan
2023-09-27 19:12:10 +03:00
committed by GitHub
parent 895f0a0723
commit d1236f2c41
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/axolotl/datasets.py
View File

@@ -22,7 +22,7 @@ class TokenizedPromptDataset(Dataset):
""" """
Dataset that returns tokenized prompts from a stream of text files. Dataset that returns tokenized prompts from a stream of text files.
Args: Args:
prompt_tokenizer (PromptTokenizingStrategy): The prompt tokenizing method for proccessing the data. prompt_tokenizer (PromptTokenizingStrategy): The prompt tokenizing method for processing the data.
dataset (dataset.Dataset): Dataset with text files. dataset (dataset.Dataset): Dataset with text files.
""" """
@@ -55,7 +55,7 @@ class ConstantLengthDataset(IterableDataset):
""" """
Iterable dataset that returns constant length chunks of tokens from stream of text files. Iterable dataset that returns constant length chunks of tokens from stream of text files.
Args: Args:
tokenizer (Tokenizer): The processor used for proccessing the data. tokenizer (Tokenizer): The processor used for processing the data.
dataset (dataset.Dataset): Dataset with text files. dataset (dataset.Dataset): Dataset with text files.
seq_length (int): Length of token sequences to return. seq_length (int): Length of token sequences to return.
""" """