# common.datasets

`common.datasets`

Dataset loading utilities.

## Classes

Name | Description
--- | ---
TrainDatasetMeta | Dataclass with fields for training and validation datasets and metadata.

### TrainDatasetMeta

    common.datasets.TrainDatasetMeta(
        train_dataset,
        eval_dataset=None,
        total_num_steps=None,
    )

Dataclass with fields for training and validation datasets and metadata.

## Functions

Name | Description
--- | ---
load_datasets | Loads one or more training or evaluation datasets, calling
load_preference_datasets | Loads one or more training or evaluation datasets for RL training using paired
sample_dataset | Randomly sample `num_samples` samples with replacement from `dataset`.

### load_datasets

    common.datasets.load_datasets(cfg, cli_args=None, debug=False)

Loads one or more training or evaluation datasets, calling `axolotl.utils.data.prepare_datasets`. Optionally, logs out debug information.

#### Parameters

Name | Type | Description | Default
--- | --- | --- | ---
cfg | DictDefault | Dictionary mapping `axolotl` config keys to values. | required
cli_args | PreprocessCliArgs | TrainerCliArgs | None | Command-specific CLI arguments. | `None`
debug | bool | Whether to print out tokenization of sample. This is duplicated in `cfg` and `cli_args`, but is kept due to use in our Colab notebooks. | `False`

#### Returns

Name | Type | Description
--- | --- | ---
| TrainDatasetMeta | Dataclass with fields for training and evaluation datasets and the computed `total_num_steps`.

### load_preference_datasets

    common.datasets.load_preference_datasets(cfg, cli_args=None)

Loads one or more training or evaluation datasets for RL training using paired preference data, calling `axolotl.utils.data.rl.prepare_preference_datasets`. Optionally, logs out debug information.

#### Parameters

Name | Type | Description | Default
--- | --- | --- | ---
cfg | DictDefault | Dictionary mapping `axolotl` config keys to values. | required
cli_args | PreprocessCliArgs | TrainerCliArgs | None | Command-specific CLI arguments. | `None`

#### Returns

Name | Type | Description
--- | --- | ---
| TrainDatasetMeta | Dataclass with fields for training and evaluation datasets and the computed
| TrainDatasetMeta | `total_num_steps`.

### sample_dataset

    common.datasets.sample_dataset(dataset, num_samples)

Randomly sample `num_samples` samples with replacement from `dataset`.