# Pre-training

Data format for a pre-training completion task.

For pretraining, there is no prompt template or roles. The only required field is `text`:

    data.jsonl

    {"text": "first row"}
    {"text": "second row"}
    ...

NoteStreaming is recommended for large datasets

Axolotl usually loads the entire dataset into memory. This will be challenging for large datasets. Use the following config to enable streaming:

    config.yaml

    pretraining_dataset:
      - name:
        path:
        split:
        text_column: # column in dataset with the data, usually `text`
        type: pretrain
        trust_remote_code:
        skip: # number of rows of data to skip over from the beginning