45 lines
1.5 KiB
Plaintext
45 lines
1.5 KiB
Plaintext
# datasets { #axolotl.datasets }
|
|
|
|
`datasets`
|
|
|
|
Module containing Dataset functionality
|
|
|
|
## Classes
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [ConstantLengthDataset](#axolotl.datasets.ConstantLengthDataset) | Iterable dataset that returns constant length chunks of tokens from stream of text files. |
|
|
| [TokenizedPromptDataset](#axolotl.datasets.TokenizedPromptDataset) | Dataset that returns tokenized prompts from a stream of text files. |
|
|
|
|
### ConstantLengthDataset { #axolotl.datasets.ConstantLengthDataset }
|
|
|
|
```python
|
|
datasets.ConstantLengthDataset(self, tokenizer, datasets, seq_length=2048)
|
|
```
|
|
|
|
Iterable dataset that returns constant length chunks of tokens from stream of text files.
|
|
Args:
|
|
tokenizer (Tokenizer): The processor used for processing the data.
|
|
dataset (dataset.Dataset): Dataset with text files.
|
|
seq_length (int): Length of token sequences to return.
|
|
|
|
### TokenizedPromptDataset { #axolotl.datasets.TokenizedPromptDataset }
|
|
|
|
```python
|
|
datasets.TokenizedPromptDataset(
|
|
self,
|
|
prompt_tokenizer,
|
|
dataset,
|
|
process_count=None,
|
|
keep_in_memory=False,
|
|
**kwargs,
|
|
)
|
|
```
|
|
|
|
Dataset that returns tokenized prompts from a stream of text files.
|
|
Args:
|
|
prompt_tokenizer (PromptTokenizingStrategy): The prompt tokenizing method for processing the data.
|
|
dataset (dataset.Dataset): Dataset with text files.
|
|
process_count (int): Number of processes to use for tokenizing.
|
|
keep_in_memory (bool): Whether to keep the tokenized dataset in memory.
|