Disable datasets caching when preparing dataset for packing

pin model_revision for phi2 (#1123 )
update PR template so we can capture twitter or discord handles (#1121 ) [skip ci]
2024-01-15 23:48:24 +01:00 · 2024-01-14 17:31:51 -05:00 · 2024-01-14 16:19:01 -05:00 · 2024-01-14 12:06:56 -05:00 · 2024-01-13 10:17:40 -05:00 · 2024-01-13 10:13:35 +01:00
66 changed files with 2762 additions and 453 deletions
--- a/.github/FUNDING.yml
+++ b/.github/FUNDING.yml
@@ -3,11 +3,11 @@
 github: OpenAccess-AI-Collective # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
 patreon: # Replace with a single Patreon username
 open_collective: # Replace with a single Open Collective username
-ko_fi: # Replace with a single Ko-fi username
+ko_fi: axolotl_ai # Replace with a single Ko-fi username
 tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
 community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
 liberapay: # Replace with a single Liberapay username
 issuehunt: # Replace with a single IssueHunt username
 otechie: # Replace with a single Otechie username
 lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
-custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
+custom: ['https://quickchart.io/qr?text=bitcoin%3Abc1qxlgwlqwfea5s2cxm42xqsfmwjct0rj8w8ea5np&size=480&centerImageUrl=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2Fthumb%2F4%2F46%2FBitcoin.svg%2F64px-Bitcoin.svg.png'] # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
--- a/.github/PULL_REQUEST_TEMPLATE/pull_request_template_simple.md
+++ b/.github/PULL_REQUEST_TEMPLATE/pull_request_template_simple.md
@@ -20,3 +20,8 @@
 ## Types of changes

 <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
+
+## Social Handles (Optional)
+
+<!-- Thanks for submitting a bugfix or enhancement. -->
+<!-- We'd love to show our thanks to you on Twitter & Discord if you provide your handle -->
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -0,0 +1,22 @@
+name: lint
+on:
+  # check on PRs, and manual triggers
+  pull_request:
+      paths:
+       - '**.py'
+       - 'requirements.txt'
+       - '.github/workflows/*.yml'
+       - "*.md"
+  workflow_dispatch:
+
+jobs:
+  pre-commit:
+    name: pre-commit
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+          cache: 'pip' # caching pip dependencies
+      - uses: pre-commit/action@v3.0.0
--- a/.github/workflows/pypi.yml
+++ b/.github/workflows/pypi.yml
@@ -34,11 +34,11 @@ jobs:
        run: echo ::set-output name=TAG_NAME::$(echo $GITHUB_REF | cut -d / -f 3)

      - name: Update version in setup.py
-        run: >-
+        run: |
          sed -i -E 's/version="([0-9.]+)",/version="${{ steps.tag.outputs.TAG_NAME }}",/g' setup.py

      - name: Build a binary wheel
-        run: >-
+        run: |
          python setup.py sdist bdist_wheel

      - name: Publish package distributions to PyPI
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -7,10 +7,12 @@ on:
    paths:
      - '**.py'
      - 'requirements.txt'
+      - '.github/workflows/*.yml'
  pull_request:
      paths:
       - '**.py'
       - 'requirements.txt'
+       - '.github/workflows/*.yml'
  workflow_dispatch:

 jobs:
@@ -31,7 +33,7 @@ jobs:
    strategy:
      fail-fast: false
      matrix:
-        python_version: ["3.9", "3.10"]
+        python_version: ["3.9", "3.10", "3.11"]
    timeout-minutes: 10

    steps:
@@ -53,29 +55,54 @@ jobs:
        run: |
          pytest --ignore=tests/e2e/ tests/

-  e2e-test:
-    name: E2E Tests
-    runs-on: [self-hosted, gpu]
-    timeout-minutes: 20
+  docker-e2e-tests:
+    if: github.repository_owner == 'OpenAccess-AI-Collective'
+    # this job needs to be run on self-hosted GPU runners...
+    runs-on: [self-hosted, gpu, docker]
+    timeout-minutes: 30
    needs: [pre-commit, pytest]

+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - cuda: 118
+            cuda_version: 11.8.0
+            python_version: "3.10"
+            pytorch: 2.0.1
+          - cuda: 121
+            cuda_version: 12.1.0
+            python_version: "3.10"
+            pytorch: 2.1.1
    steps:
-      - name: Check out repository code
-        uses: actions/checkout@v3
-
-      - name: Setup Python
-        uses: actions/setup-python@v4
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Docker metadata
+        id: metadata
+        uses: docker/metadata-action@v5
        with:
-          python-version: "3.10"
-#          cache: 'pip' # caching pip dependencies
-
-      - name: Install dependencies
+          images: winglian/axolotl-tests
+      - name: Build Docker image
        run: |
-          pip3 install --extra-index-url https://download.pytorch.org/whl/cu118 -U torch==2.0.1
-          pip3 uninstall -y transformers accelerate
-          pip3 install -U -e .[flash-attn,mamba-ssm]
-          pip3 install -r requirements-tests.txt
-
-      - name: Run e2e tests
+          # Set up build arguments
+          BASE_TAG="main-base-py${{ matrix.python_version }}-cu${{ matrix.cuda }}-${{ matrix.pytorch }}"
+          CUDA="${{ matrix.cuda }}"
+          PYTORCH_VERSION="${{ matrix.pytorch }}"
+          # Build the Docker image
+          docker build . \
+            --file ./docker/Dockerfile-tests \
+            --build-arg BASE_TAG=$BASE_TAG \
+            --build-arg CUDA=$CUDA \
+            --build-arg GITHUB_REF=$GITHUB_REF \
+            --build-arg PYTORCH_VERSION=$PYTORCH_VERSION \
+            --tag ${{ steps.metadata.outputs.tags }}-py${{ matrix.python_version }}-cu${{ matrix.cuda }}-${{ matrix.pytorch }} \
+            --no-cache
+      - name: Unit Tests w docker image
        run: |
-          pytest tests/e2e/
+          docker run --rm ${{ steps.metadata.outputs.tags }}-py${{ matrix.python_version }}-cu${{ matrix.cuda }}-${{ matrix.pytorch }} pytest --ignore=tests/e2e/ /workspace/axolotl/tests/
+      - name: GPU Unit Tests w docker image
+        run: |
+          docker run --privileged --gpus "all" --env WANDB_DISABLED=true --rm ${{ steps.metadata.outputs.tags }}-py${{ matrix.python_version }}-cu${{ matrix.cuda }}-${{ matrix.pytorch }} pytest --ignore=tests/e2e/patched/ /workspace/axolotl/tests/e2e/
+      - name: GPU Unit Tests monkeypatched w docker image
+        run: |
+          docker run --privileged --gpus "all" --env WANDB_DISABLED=true --rm ${{ steps.metadata.outputs.tags }}-py${{ matrix.python_version }}-cu${{ matrix.cuda }}-${{ matrix.pytorch }} pytest /workspace/axolotl/tests/e2e/patched/
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,7 @@
 **/axolotl.egg-info
 configs
+last_run_prepared/
+.vscode

 # Byte-compiled / optimized / DLL files
 __pycache__/
--- a/.vscode/README.md
+++ b/.vscode/README.md
@@ -0,0 +1 @@
+See [docs/debugging.md](../docs/debugging.md) for guidance on how to modify these files to debug axolotl with VSCode.
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -0,0 +1,34 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Debug axolotl prompt - sharegpt",
+            "type": "python",
+            "module": "accelerate.commands.launch",
+            "request": "launch",
+            "args": [
+                "-m", "axolotl.cli.train", "dev_sharegpt.yml",
+                // The flags below simplify debugging by overriding the axolotl config
+                // with the debugging tips above.  Modify as needed.
+                "--dataset_processes=1",      // limits data preprocessing to one process
+                "--max_steps=1",              // limits training to just one step
+                "--batch_size=1",             // minimizes batch size
+                "--micro_batch_size=1",       // minimizes batch size
+                "--val_set_size=0",           // disables validation
+                "--sample_packing=False",     // disables sample packing which is necessary for small datasets
+                "--eval_sample_packing=False",// disables sample packing on eval set
+                "--dataset_prepared_path=temp_debug/axolotl_outputs/data", // send data outputs to a temp folder
+                "--output_dir=temp_debug/axolotl_outputs/model" // send model outputs to a temp folder
+                ],
+            "console": "integratedTerminal",      // show output in the integrated terminal
+            "cwd": "${workspaceFolder}/devtools", // set working directory to devtools from the root of the project
+            "justMyCode": true,                   // step through only axolotl code
+            "env": {"CUDA_VISIBLE_DEVICES": "0",  // Since we aren't doing distributed training, we need to limit to one GPU
+                    "HF_HOME": "${workspaceFolder}/devtools/temp_debug/.hf-cache"}, // send HF cache to a temp folder
+            "preLaunchTask": "cleanup-for-dataprep", // delete temp folders (see below)
+        }
+    ]
+}
--- a/.vscode/tasks.json
+++ b/.vscode/tasks.json
@@ -0,0 +1,27 @@
+//this file is used by launch.json
+{
+  "version": "2.0.0",
+  "tasks": [
+    // this task changes into the devtools directory and deletes the temp_debug/axolotl_outputs folder
+    {
+      "label": "delete-outputs",
+      "type": "shell",
+      "command": "rm -rf temp_debug/axolotl_outputs",
+      "options":{ "cwd": "${workspaceFolder}/devtools"},
+      "problemMatcher": []
+    },
+    // this task changes into the devtools directory and deletes the `temp_debug/.hf-cache/datasets` folder
+    {
+      "label": "delete-temp-hf-dataset-cache",
+      "type": "shell",
+      "command": "rm -rf temp_debug/.hf-cache/datasets",
+      "options":{ "cwd": "${workspaceFolder}/devtools"},
+      "problemMatcher": []
+    },
+      // this task combines the two tasks above
+    {
+     "label": "cleanup-for-dataprep",
+     "dependsOn": ["delete-outputs", "delete-temp-hf-dataset-cache"],
+    }
+  ]
+}
--- a/README.md
+++ b/README.md
@@ -10,7 +10,7 @@ Features:
 - Integrated with xformer, flash attention, rope scaling, and multipacking
 - Works with single GPU or multiple GPUs via FSDP or Deepspeed
 - Easily run with Docker locally or on the cloud
- Log results and optionally checkpoints to wandb
+- Log results and optionally checkpoints to wandb or mlflow
 - And more!


@@ -39,10 +39,12 @@ Features:
  - [Special Tokens](#special-tokens)
 - [Common Errors](#common-errors-)
  - [Tokenization Mismatch b/w Training & Inference](#tokenization-mismatch-bw-inference--training)
+- [Debugging Axolotl](#debugging-axolotl)
 - [Need Help?](#need-help-)
 - [Badge](#badge-)
 - [Community Showcase](#community-showcase)
 - [Contributing](#contributing-)
+- [Sponsors](#sponsors-)

 </td>
 <td>
@@ -130,6 +132,9 @@ accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
  docker compose up -d
  ```

+>[!Tip]
+> If you want to debug axolotl or prefer to use Docker as your development environment, see the [debugging guide's section on Docker](docs/debugging.md#debugging-with-docker).
+
  <details>

  <summary>Docker advanced</summary>
@@ -137,7 +142,7 @@ accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
  A more powerful Docker command to run would be this:

  ```bash
-  docker run --privileged --gpus '"all"' --shm-size 10g --rm -it --name axolotl --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --mount type=volume,src=axolotl,target=/workspace/axolotl -v ${HOME}/.cache/huggingface:/root/.cache/huggingface winglian/axolotl:main-py3.10-cu118-2.0.1
+docker run --privileged --gpus '"all"' --shm-size 10g --rm -it --name axolotl --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --mount type=bind,src="${PWD}",target=/workspace/axolotl -v ${HOME}/.cache/huggingface:/root/.cache/huggingface winglian/axolotl:main-py3.10-cu118-2.0.1
  ```

  It additionally:
@@ -550,6 +555,11 @@ tf32: true # require >=ampere
 bfloat16: true # require >=ampere
 float16: true

+# Limit the memory for all available GPUs to this amount (if an integer, expressed in gigabytes); default: unset
+gpu_memory_limit: 20GiB
+# Do the LoRA/PEFT loading on CPU -- this is required if the base model is so large it takes up most or all of the available GPU VRAM, e.g. during a model and LoRA merge
+lora_on_cpu: true
+
 # A list of one or more datasets to finetune the model with
 datasets:
  # HuggingFace dataset repo | s3://,gs:// path | "json" for local dataset, make sure to fill data_files
@@ -589,6 +599,9 @@ datasets:
      # For `completion` datsets only, uses the provided field instead of `text` column
      field:

+# use RL training: dpo, ipo, kto_pair
+rl:
+
 # Saves the desired chat template to the tokenizer_config.json for easier inferencing
 # Currently supports chatml and inst (mistral/mixtral)
 chat_template: chatml
@@ -643,7 +656,8 @@ max_memory:
 # If you want to use 'lora' or 'qlora' or leave blank to train all parameters in original model
 adapter: lora
 # If you already have a lora model trained that you want to load, put that here.
-# This means after training, if you want to test the model, you should set this to the value of `lora_out_dir`.
+# This means after training, if you want to test the model, you should set this to the value of `output_dir`.
+# Note that if you merge an adapter to the base model, a new subdirectory `merged` will be created under the `output_dir`.
 lora_model_dir:

 # LoRA hyperparameters
@@ -670,10 +684,6 @@ lora_modules_to_save:
 #  - embed_tokens
 #  - lm_head

-# Once you complete training, the model will be saved to the following directory.
-# If you merge the adapter to the base model, a subdirectory `merged` will be created under this directory.
-# Make sure `lora_model_dir` points to this directory if you want to use the trained model.
-lora_out_dir:
 lora_fan_in_fan_out: false

 # ReLoRA configuration
@@ -692,6 +702,10 @@ wandb_name: # Set the name of your wandb run
 wandb_run_id: # Set the ID of your wandb run
 wandb_log_model: # "checkpoint" to log model to wandb Artifacts every `save_steps` or "end" to log only at the end of training

+# mlflow configuration if you're using it
+mlflow_tracking_uri: # URI to mlflow
+mlflow_experiment_name: # Your experiment name
+
 # Where to save the full-finetuned model to
 output_dir: ./completed-model

@@ -741,6 +755,9 @@ group_by_length: false

 # Whether to use gradient checkpointing https://huggingface.co/docs/transformers/v4.18.0/en/performance#gradient-checkpointing
 gradient_checkpointing: false
+# additional kwargs to pass to the trainer for gradient checkpointing
+# gradient_checkpointing_kwargs:
+#   use_reentrant: false

 # Stop training after this many evaluation losses have increased in a row
 # https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
@@ -749,6 +766,7 @@ early_stopping_patience: 3
 # Specify a scheduler and kwargs to use with the optimizer
 lr_scheduler: # 'one_cycle' | 'log_sweep' | empty for cosine
 lr_scheduler_kwargs:
+cosine_min_lr_ratio: # decay lr to some percentage of the peak lr, e.g. cosine_min_lr_ratio=0.1 for 10% of peak lr

 # For one_cycle optim
 lr_div_factor: # Learning rate div factor
@@ -932,8 +950,9 @@ accelerate launch -m axolotl.cli.train your_config.yml
 You can optionally pre-tokenize dataset with the following before finetuning.
 This is recommended for large datasets.

- Set `push_dataset_to_hub: hf_user/repo` to push it to Huggingface.
- Use `--debug` to see preprocessed examples.
+- Set `dataset_prepared_path:` to a local folder for saving and loading pre-tokenized dataset.
+- (Optional): Set `push_dataset_to_hub: hf_user/repo` to push it to Huggingface.
+- (Optional): Use `--debug` to see preprocessed examples.

 ```bash
 python -m axolotl.cli.preprocess your_config.yml
@@ -1035,21 +1054,23 @@ Please use `--sample_packing False` if you have it on and receive the error simi

 ### Merge LORA to base

-Add below flag to train command above
+The following command will merge your LORA adapater with your base model.  You can optionally pass the argument `--lora_model_dir` to specify the directory where your LORA adapter was saved, otherwhise, this will be inferred from `output_dir` in your axolotl config file.  The merged model is saved in the sub-directory `{lora_model_dir}/merged`.

 ```bash
-python3 -m axolotl.cli.merge_lora examples/your_config.yml --lora_model_dir="./completed-model"
+python3 -m axolotl.cli.merge_lora your_config.yml --lora_model_dir="./completed-model"
 ```

-If you run out of CUDA memory, you can try to merge in system RAM with
+You may need to use the `gpu_memory_limit` and/or `lora_on_cpu` config options to avoid running out of memory. If you still run out of CUDA memory, you can try to merge in system RAM with

 ```bash
 CUDA_VISIBLE_DEVICES="" python3 -m axolotl.cli.merge_lora ...
 ```

+although this will be very slow, and using the config options above are recommended instead.
+
 ## Common Errors 🧰

-See also the [FAQ's](./docs/faq.md).
+See also the [FAQ's](./docs/faq.md) and [debugging guide](docs/debugging.md).

 > If you encounter a 'Cuda out of memory' error, it means your GPU ran out of memory during the training process. Here's how to resolve it:

@@ -1099,6 +1120,10 @@ If you decode a prompt constructed by axolotl, you might see spaces between toke

 Having misalignment between your prompts during training and inference can cause models to perform very poorly, so it is worth checking this.  See [this blog post](https://hamel.dev/notes/llm/05_tokenizer_gotchas.html) for a concrete example.

+## Debugging Axolotl
+
+See [this debugging guide](docs/debugging.md) for tips on debugging Axolotl, along with an example configuration for debugging with VSCode.
+
 ## Need help? 🙋♂️

 Join our [Discord server](https://discord.gg/HhrNrHJPRb) where we can help you
@@ -1141,3 +1166,33 @@ pre-commit install
 # test
 pytest tests/
 ```
+
+## Sponsors 🤝❤
+
+OpenAccess AI Collective is run by volunteer contributors such as [winglian](https://github.com/winglian),
+[NanoCode012](https://github.com/NanoCode012), [tmm1](https://github.com/tmm1),
+[mhenrichsen](https://github.com/mhenrichsen), [casper-hansen](https://github.com/casper-hansen),
+[hamelsmu](https://github.com/hamelsmu) and many more who help us accelerate forward by fixing bugs, answering
+community questions and implementing new features. Axolotl needs donations from sponsors for the compute needed to
+run our unit & integration tests, troubleshooting community issues, and providing bounties. If you love axolotl,
+consider sponsoring the project via [GitHub Sponsors](https://github.com/sponsors/OpenAccess-AI-Collective),
+[Ko-fi](https://ko-fi.com/axolotl_ai) or reach out directly to
+[wing@openaccessaicollective.org](mailto:wing@openaccessaicollective.org).
+
+---
+
+#### 💎 Diamond Sponsors - [Contact directly](mailto:wing@openaccessaicollective.org)
+
+---
+
+#### 🥇 Gold Sponsors - $5000/mo
+
+---
+
+#### 🥈 Silver Sponsors - $1000/mo
+
+---
+
+#### 🥉 Bronze Sponsors - $500/mo
+
+---
--- a/deepspeed/zero3_cpu.json
+++ b/deepspeed/zero3_cpu.json
@@ -1,47 +0,0 @@
-{
-  "zero_optimization": {
-    "stage": 3,
-    "offload_optimizer": {
-      "device": "cpu",
-      "pin_memory": true
-    },
-    "offload_param": {
-      "device": "cpu",
-      "pin_memory": true
-    },
-    "overlap_comm": true,
-    "contiguous_gradients": true,
-    "sub_group_size": 0,
-    "reduce_bucket_size": "auto",
-    "stage3_prefetch_bucket_size": "auto",
-    "stage3_param_persistence_threshold": "auto",
-    "stage3_max_live_parameters": 0,
-    "stage3_max_reuse_distance": 0,
-    "stage3_gather_16bit_weights_on_model_save": true
-  },
-  "bf16": {
-    "enabled": "auto"
-  },
-  "fp16": {
-    "enabled": "auto",
-    "auto_cast": false,
-    "loss_scale": 0,
-    "initial_scale_power": 32,
-    "loss_scale_window": 1000,
-    "hysteresis": 2,
-    "min_loss_scale": 1
-  },
-  "optimizer": {
-    "type": "AdamW",
-    "params": {
-      "lr": "auto",
-      "betas": "auto",
-      "eps": "auto",
-      "weight_decay": "auto"
-    }
-  },
-  "gradient_accumulation_steps": "auto",
-  "train_batch_size": "auto",
-  "train_micro_batch_size_per_gpu": "auto",
-  "wall_clock_breakdown": false
-}
--- a/devtools/README.md
+++ b/devtools/README.md
@@ -0,0 +1 @@
+This directory contains example config files that might be useful for debugging. Please see [docs/debugging.md](../docs/debugging.md) for more information.
--- a/devtools/dev_sharegpt.yml
+++ b/devtools/dev_sharegpt.yml
@@ -0,0 +1,49 @@
+# Example config for debugging the sharegpt prompt format
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+model_type: LlamaForCausalLM
+tokenizer_type: LlamaTokenizer
+is_llama_derived_model: true
+
+load_in_8bit: true
+load_in_4bit: false
+
+datasets:
+  - path: philschmid/guanaco-sharegpt-style
+    type: sharegpt
+    shards: 10
+val_set_size: 0
+output_dir: temp_debug/axolotl_outputs/model
+dataset_prepared_path: temp_debug/axolotl_outputs/data
+dataset_processes: 1
+
+sequence_len: 4096
+sample_packing: false
+pad_to_sequence_len: true
+
+adapter: lora
+lora_model_dir:
+lora_r: 32
+lora_alpha: 16
+lora_dropout: 0.05
+lora_target_linear: true
+lora_fan_in_fan_out:
+
+micro_batch_size: 1
+num_epochs: 1
+max_steps: 10
+optimizer: adamw_bnb_8bit
+lr_scheduler: cosine
+learning_rate: 0.0002
+
+train_on_inputs: false
+group_by_length: false
+bf16: false
+fp16: true
+tf32: false
+
+gradient_checkpointing: true
+logging_steps: 1
+flash_attention: true
+
+warmup_steps: 10
+weight_decay: 0.0
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -20,9 +20,9 @@ WORKDIR /workspace/axolotl

 # If AXOLOTL_EXTRAS is set, append it in brackets
 RUN if [ "$AXOLOTL_EXTRAS" != "" ] ; then \
-        pip install -e .[deepspeed,flash-attn,$AXOLOTL_EXTRAS]; \
+        pip install -e .[deepspeed,flash-attn,mamba-ssm,$AXOLOTL_EXTRAS]; \
    else \
-        pip install -e .[deepspeed,flash-attn]; \
+        pip install -e .[deepspeed,flash-attn,mamba-ssm]; \
    fi

 # So we can test the Docker image
--- a/docker/Dockerfile-runpod
+++ b/docker/Dockerfile-runpod
@@ -5,6 +5,7 @@ ENV HF_DATASETS_CACHE="/workspace/data/huggingface-cache/datasets"
 ENV HUGGINGFACE_HUB_CACHE="/workspace/data/huggingface-cache/hub"
 ENV TRANSFORMERS_CACHE="/workspace/data/huggingface-cache/hub"
 ENV HF_HOME="/workspace/data/huggingface-cache/hub"
+ENV HF_HUB_ENABLE_HF_TRANSFER="1"

 COPY scripts/runpod-entrypoint.sh /root/runpod-entrypoint.sh

--- a/docker/Dockerfile-tests
+++ b/docker/Dockerfile-tests
@@ -0,0 +1,40 @@
+ARG BASE_TAG=main-base
+FROM winglian/axolotl-base:$BASE_TAG
+
+ARG TORCH_CUDA_ARCH_LIST="7.0 7.5 8.0 8.6+PTX"
+ARG AXOLOTL_EXTRAS=""
+ARG CUDA="118"
+ENV BNB_CUDA_VERSION=$CUDA
+ARG PYTORCH_VERSION="2.0.1"
+ARG GITHUB_REF="main"
+
+ENV PYTORCH_VERSION=$PYTORCH_VERSION
+
+RUN apt-get update && \
+    apt-get install -y --allow-change-held-packages vim curl nano libnccl2 libnccl-dev
+
+WORKDIR /workspace
+
+RUN git clone --depth=1 https://github.com/OpenAccess-AI-Collective/axolotl.git
+
+WORKDIR /workspace/axolotl
+
+RUN git fetch origin +$GITHUB_REF && \
+    git checkout FETCH_HEAD
+
+# If AXOLOTL_EXTRAS is set, append it in brackets
+RUN if [ "$AXOLOTL_EXTRAS" != "" ] ; then \
+        pip install -e .[deepspeed,flash-attn,mamba-ssm,$AXOLOTL_EXTRAS]; \
+    else \
+        pip install -e .[deepspeed,flash-attn,mamba-ssm]; \
+    fi
+
+# So we can test the Docker image
+RUN pip install pytest
+
+# fix so that git fetch/pull from remote works
+RUN git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" && \
+    git config --get remote.origin.fetch
+
+# helper for huggingface-login cli
+RUN git config --global credential.helper store
--- a/docs/debugging.md
+++ b/docs/debugging.md
@@ -0,0 +1,242 @@
+# Debugging Axolotl
+
+This document provides some tips and tricks for debugging Axolotl.  It also provides an example configuration for debugging with VSCode.  A good debugging setup is essential to understanding how Axolotl code works behind the scenes.
+
+## Table of Contents
+
+- [General Tips](#general-tips)
+- [Debugging with VSCode](#debugging-with-vscode)
+    - [Background](#background)
+    - [Configuration](#configuration)
+    - [Customizing your debugger](#customizing-your-debugger)
+    - [Video Tutorial](#video-tutorial)
+- [Debugging With Docker](#debugging-with-docker)
+    - [Setup](#setup)
+    - [Attach To Container](#attach-to-container)
+    - [Video - Attaching To Docker On Remote Host](#video---attaching-to-docker-on-remote-host)
+
+## General Tips
+
+While debugging it's helpful to simplify your test scenario as much as possible.  Here are some tips for doing so:
+
+> [!Important]
+> All of these tips are incorporated into the [example configuration](#configuration) for debugging with VSCode below.
+
+1. **Make sure you are using the latest version of axolotl**:  This project changes often and bugs get fixed fast.  Check your git branch and make sure you have pulled the latest changes from `main`.
+1. **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing:
+    - Set `CUDA_VISIBLE_DEVICES` to a single GPU, ex: `export CUDA_VISIBLE_DEVICES=0`.
+    - Set `dataset_processes: 1` in your axolotl config or run the training command with `--dataset_processes=1`.
+2. **Use a small dataset**: Construct or use a small dataset from HF Hub. When using a small dataset, you will often have to make sure `sample_packing: False` and `eval_sample_packing: False` to avoid errors.  If you are in a pinch and don't have time to construct a small dataset but want to use from the HF Hub, you can shard the data (this will still tokenize the entire dataset, but will only use a fraction of the data for training.  For example, to shard the dataset into 20 pieces, add the following to your axolotl config):
+    ```yaml
+    dataset:
+        ...
+        shards: 20
+    ```
+3. **Use a small model**: A good example of a small model is [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0).
+4. **Minimize iteration time**: Make sure the training loop finishes as fast as possible, with these settings.
+    - `micro_batch_size: 1`
+    - `max_steps: 1`
+    - `val_set_size: 0`
+5. **Clear Caches:** Axolotl caches certain steps and so does the underlying HuggingFace trainer.  You may want to clear some of these caches when debugging.
+    - Data preprocessing: When debugging data preprocessing, which includes prompt template formation, you may want to delete the directory set in `dataset_prepared_path:` in your axolotl config.  If you didn't set this value, the default is `last_run_prepared`.
+    - HF Hub: If you are debugging data preprocessing, you should clear the relevant HF cache [HuggingFace cache](https://huggingface.co/docs/datasets/cache), by deleting the appropriate `~/.cache/huggingface/datasets/...` folder(s).
+    - **The recommended approach is to redirect all outputs and caches to a temporary folder and delete selected subfolders before each run.  This is demonstrated in the example configuration below.**
+
+
+## Debugging with VSCode
+
+### Background
+
+The below example shows how to configure VSCode to debug data preprocessing of the `sharegpt` format.  This is the format used when you have the following in your axolotl config:
+
+```yaml
+datasets:
+  - path: <path to your sharegpt formatted dataset> # example on HF Hub: philschmid/guanaco-sharegpt-style
+    type: sharegpt
+```
+
+>[!Important]
+> If you are already familiar with advanced VSCode debugging, you can skip the below explanation and look at the files [.vscode/launch.json](../.vscode/launch.json) and [.vscode/tasks.json](../.vscode/tasks.json) for an example configuration.
+
+>[!Tip]
+> If you prefer to watch a video, rather than read, you can skip to the [video tutorial](#video-tutorial) below (but doing both is recommended).
+
+### Setup
+
+Make sure you have an [editable install](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) of Axolotl, which ensures that changes you make to the code are reflected at runtime.  Run the following commands from the root of this project:
+
+```bash
+pip3 install packaging
+pip3 install -e '.[flash-attn,deepspeed]'
+```
+
+#### Remote Hosts
+
+If you developing on a remote host, you can easily use VSCode to debug remotely.  To do so, you will need to follow this [remote - SSH guide](https://code.visualstudio.com/docs/remote/ssh).  You can also see the video below on [Docker and Remote SSH debugging](#video---attaching-to-docker-on-remote-host).
+
+```bash
+
+### Configuration
+
+The easiest way to get started is to modify the [.vscode/launch.json](../.vscode/launch.json) file in this project.  This is just an example configuration, so you may need to modify or copy it to suit your needs.
+
+For example, to mimic the command `cd devtools && CUDA_VISIBLE_DEVICES=0 accelerate launch -m axolotl.cli.train dev_sharegpt.yml`, you would use the below configuration[^1].  Note that we add additional flags that override the axolotl config and incorporate the tips above (see the comments). We also set the working directory to `devtools` and set the `env` variable `HF_HOME` to a temporary folder that is later partially deleted.  This is because we want to delete the HF dataset cache before each run in order to ensure that the data preprocessing code is run from scratch.
+
+```jsonc
+// .vscode/launch.json
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Debug axolotl prompt - sharegpt",
+            "type": "python",
+            "module": "accelerate.commands.launch",
+            "request": "launch",
+            "args": [
+                "-m", "axolotl.cli.train", "dev_sharegpt.yml",
+                // The flags below simplify debugging by overriding the axolotl config
+                // with the debugging tips above.  Modify as needed.
+                "--dataset_processes=1",      // limits data preprocessing to one process
+                "--max_steps=1",              // limits training to just one step
+                "--batch_size=1",             // minimizes batch size
+                "--micro_batch_size=1",       // minimizes batch size
+                "--val_set_size=0",           // disables validation
+                "--sample_packing=False",     // disables sample packing which is necessary for small datasets
+                "--eval_sample_packing=False",// disables sample packing on eval set
+                "--dataset_prepared_path=temp_debug/axolotl_outputs/data", // send data outputs to a temp folder
+                "--output_dir=temp_debug/axolotl_outputs/model" // send model outputs to a temp folder
+                ],
+            "console": "integratedTerminal",      // show output in the integrated terminal
+            "cwd": "${workspaceFolder}/devtools", // set working directory to devtools from the root of the project
+            "justMyCode": true,                   // step through only axolotl code
+            "env": {"CUDA_VISIBLE_DEVICES": "0",  // Since we aren't doing distributed training, we need to limit to one GPU
+                    "HF_HOME": "${workspaceFolder}/devtools/temp_debug/.hf-cache"}, // send HF cache to a temp folder
+            "preLaunchTask": "cleanup-for-dataprep", // delete temp folders (see below)
+        }
+    ]
+}
+```
+
+**Additional notes about this configuration:**
+
+- The argument `justMyCode` is set to `true` such that you step through only the axolotl code.  If you want to step into dependencies, set this to `false`.
+- The `preLaunchTask`: `cleanup-for-dataprep` is defined in [.vscode/tasks.json](../.vscode/tasks.json) and is used to delete the following folders before debugging, which is essential to ensure that the data pre-processing code is run from scratch:
+    -  `./devtools/temp_debug/axolotl_outputs`
+    - `./devtools/temp_debug/.hf-cache/datasets`
+
+>[!Tip]
+> You may not want to delete these folders. For example, if you are debugging model training instead of data pre-processing, you may NOT want to delete the cache or output folders. You may also need to add additional tasks to the `tasks.json` file depending on your use case.
+
+Below is the [./vscode/tasks.json](../.vscode/tasks.json) file that defines the `cleanup-for-dataprep` task.  This task is run before each debugging session when you use the above configuration.  Note how there are two tasks that delete the two folders mentioned above.  The third task `cleanup-for-dataprep` is a composite task that combines the two tasks.  A composite task is necessary because VSCode does not allow you to specify multiple tasks in the `preLaunchTask` argument of the `launch.json` file.
+
+```jsonc
+// .vscode/tasks.json
+// this file is used by launch.json
+{
+    "version": "2.0.0",
+    "tasks": [
+      // this task changes into the devtools directory and deletes the temp_debug/axolotl_outputs folder
+      {
+        "label": "delete-outputs",
+        "type": "shell",
+        "command": "rm -rf temp_debug/axolotl_outputs",
+        "options":{ "cwd": "${workspaceFolder}/devtools"},
+        "problemMatcher": []
+      },
+      // this task changes into the devtools directory and deletes the `temp_debug/.hf-cache/datasets` folder
+      {
+        "label": "delete-temp-hf-dataset-cache",
+        "type": "shell",
+        "command": "rm -rf temp_debug/.hf-cache/datasets",
+        "options":{ "cwd": "${workspaceFolder}/devtools"},
+        "problemMatcher": []
+      },
+        // this task combines the two tasks above
+      {
+       "label": "cleanup-for-dataprep",
+       "dependsOn": ["delete-outputs", "delete-temp-hf-dataset-cache"],
+      }
+    ]
+}
+```
+
+### Customizing your debugger
+
+Your debugging use case may differ from the example above.  The easiest thing to do is to put your own axolotl config in the `devtools` folder and modify the `launch.json` file to use your config.  You may also want to modify the `preLaunchTask` to delete different folders or not delete anything at all.
+
+### Video Tutorial
+
+The following video tutorial walks through the above configuration and demonstrates how to debug with VSCode, (click the image below to watch):
+
+<div style="text-align: center; line-height: 0;">
+
+<a href="https://youtu.be/xUUB11yeMmc" target="_blank"
+title="How to debug Axolotl (for fine tuning LLMs)"><img
+src="https://i.ytimg.com/vi/xUUB11yeMmc/maxresdefault.jpg"
+style="border-radius: 10px; display: block; margin: auto;" width="560" height="315" /></a>
+
+<figcaption style="font-size: smaller;"><a href="https://hamel.dev">Hamel Husain's</a> tutorial: <a href="https://www.youtube.com/watch?v=xUUB11yeMmc">Debugging Axolotl w/VSCode</a></figcaption>
+
+</div>
+<br>
+
+## Debugging With Docker
+
+Using [official Axolotl Docker images](https://hub.docker.com/r/winglian/axolotl/tags) is a great way to debug your code, and is a very popular way to use Axolotl.  Attaching VSCode to Docker takes a few more steps.
+
+### Setup
+
+On the host that is running axolotl (ex: if you are using a remote host), clone the axolotl repo and change your current directory to the root:
+
+```bash
+git clone https://github.com/OpenAccess-AI-Collective/axolotl
+cd axolotl
+```
+
+>[!Tip]
+> If you already have axolotl cloned on your host, make sure you have the latest changes and change into the root of the project.
+
+Next, run the desired docker image and mount the current directory. Below is a docker command you can run to do this:[^2]
+
+```bash
+docker run --privileged --gpus '"all"' --shm-size 10g --rm -it --name axolotl --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --mount type=bind,src="${PWD}",target=/workspace/axolotl -v ${HOME}/.cache/huggingface:/root/.cache/huggingface winglian/axolotl:main-py3.10-cu118-2.0.1
+```
+
+>[!Tip]
+> To understand which containers are available, see the [Docker section of the README](../README.md#docker) and the [DockerHub repo](https://hub.docker.com/r/winglian/axolotl/tags).  For details of how the Docker containers are built, see axolotl's [Docker CI builds](../.github/workflows/main.yml).
+
+You will now be in the container.  Next, perform an editable install of Axolotl:
+
+```bash
+pip3 install packaging
+pip3 install -e '.[flash-attn,deepspeed]'
+```
+
+### Attach To Container
+
+Next, if you are using a remote host, [Remote into this host with VSCode](https://code.visualstudio.com/docs/remote/ssh).  If you are using a local host, you can skip this step.
+
+Next, select `Dev Containers: Attach to Running Container...` using the command palette (`CMD + SHIFT + P`) in VSCode.  You will be prompted to select a container to attach to.  Select the container you just created.  You will now be in the container with a working directory that is at the root of the project.  Any changes you make to the code will be reflected both in the container and on the host.
+
+Now you are ready to debug as described above (see [Debugging with VSCode](#debugging-with-vscode)).
+
+### Video - Attaching To Docker On Remote Host
+
+Here is a short video that demonstrates how to attach to a Docker container on a remote host:
+
+<div style="text-align: center; line-height: 0;">
+
+<a href="https://youtu.be/0AuoR7QnHR0" target="_blank"
+title="Debugging Axolotl Part 2: Attaching to Docker on a Remote Host"><img
+src="https://i.ytimg.com/vi/0AuoR7QnHR0/hqdefault.jpg"
+style="border-radius: 10px; display: block; margin: auto;" width="560" height="315" /></a>
+
+<figcaption style="font-size: smaller;"><a href="https://hamel.dev">Hamel Husain's</a> tutorial: <a href="https://youtu.be/0AuoR7QnHR0">Debugging Axolotl Part 2: Attaching to Docker on a Remote Host
+</a></figcaption>
+
+</div>
+<br>
+
+[^1]: The config actually mimics the command `CUDA_VISIBLE_DEVICES=0 python -m accelerate.commands.launch -m axolotl.cli.train devtools/sharegpt.yml`, but this is the same thing.
+
+[^2]: Many of the below flags are recommended best practices by Nvidia when using nvidia-container-toolkit.  You can read more about these flags [here](https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html).
--- a/docs/rlhf.md
+++ b/docs/rlhf.md
@@ -0,0 +1,44 @@
+# RLHF (Beta)
+
+### Overview
+
+Reinforcement Learning from Human Feedback is a method whereby a language model is optimized from data using human
+feedback. Various methods include, but not limited to:
+
+- Proximal Policy Optimization (PPO) (not yet supported in axolotl)
+- Direct Preference Optimization (DPO)
+- Identity Preference Optimization (IPO)
+
+
+### RLHF using Axolotl
+
+[!IMPORTANT]
+This is a BETA feature and many features are not fully implemented. You are encouraged to open new PRs to improve the integration and functionality.
+
+The various RL training methods are implemented in trl and wrapped via axolotl. Below are various examples with how you can use various preference datasets to train models that use ChatML
+
+#### DPO
+```yaml
+rl: true
+datasets:
+  - path: Intel/orca_dpo_pairs
+    split: train
+    type: intel_apply_chatml
+  - path: argilla/ultrafeedback-binarized-preferences
+    split: train
+    type: argilla_apply_chatml
+```
+
+#### IPO
+```yaml
+rl: ipo
+```
+
+#### Trl autounwrap for peft
+
+Trl supports autounwrapping peft models, so that a ref model does not need to be additionally loaded, leading to less VRAM needed. This is on by default. To turn it off, pass the following config.
+
+```yaml
+# load ref model when adapter training.
+rl_adapter_ref_model: true
+```
--- a/examples/yayi2-30b/fft.yml
+++ b/examples/yayi2-30b/fft.yml
@@ -1,25 +1,35 @@
-base_model: models/yayi2-30b
+base_model: microsoft/phi-2
+model_revision:  834565c  # pin model repo to the previous architecture
 model_type: AutoModelForCausalLM
 tokenizer_type: AutoTokenizer
-is_mistral_derived_model: false
 trust_remote_code: true
-model_revision: refs/pr/5

 load_in_8bit: false
 load_in_4bit: false
 strict: false

 datasets:
-  - path: mhenrichsen/alpaca_2k_test
+  - path: garage-bAInd/Open-Platypus
    type: alpaca
+
 dataset_prepared_path:
 val_set_size: 0.05
-output_dir: ./out
+output_dir: ./phi-sft-out

 sequence_len: 2048
-sample_packing: false
-pad_to_sequence_len: false
-eval_sample_packing: false
+sample_packing: false  # currently unsupported
+pad_to_sequence_len:
+
+adapter:
+lora_model_dir:
+lora_r: 16
+lora_alpha: 32
+lora_dropout: 0.1
+lora_target_linear: true
+lora_fan_in_fan_out:
+lora_modules_to_save:
+  - embd
+  - lm_head

 wandb_project:
 wandb_entity:
@@ -29,16 +39,19 @@ wandb_log_model:

 gradient_accumulation_steps: 1
 micro_batch_size: 1
-num_epochs: 1
-optimizer: adamw_bnb_8bit
+num_epochs: 4
+optimizer: paged_adamw_8bit
+adam_beta2: 0.95
+adam_epsilon: 0.00001
+max_grad_norm: 1.0
 lr_scheduler: cosine
-learning_rate: 0.000005
+learning_rate: 1e-5

 train_on_inputs: false
 group_by_length: false
 bf16: true
 fp16: false
-tf32: false
+tf32: true

 gradient_checkpointing: true
 early_stopping_patience:
@@ -48,17 +61,14 @@ logging_steps: 1
 xformers_attention:
 flash_attention: true

-warmup_steps: 10
+warmup_steps: 100
 evals_per_epoch: 4
-eval_table_size:
-eval_table_max_new_tokens: 128
 saves_per_epoch: 1
 debug:
-deepspeed: deepspeed/zero3_cpu.json
-weight_decay: 0.0
+deepspeed:
+weight_decay: 0.1
 fsdp:
 fsdp_config:
+resize_token_embeddings_to_32x: true
 special_tokens:
-  bos_token: "<s>"
-  eos_token: "</s>"
-  unk_token: "<unk>"
+  pad_token: "<|endoftext|>"
--- a/examples/tiny-llama/README.md
+++ b/examples/tiny-llama/README.md
@@ -0,0 +1,17 @@
+# Overview
+
+This is a simple example of how to finetune TinyLlama1.1B using either lora or qlora:
+
+LoRa:
+
+```
+accelerate launch -m axolotl.cli.train examples/tiny-llama/lora.yml
+```
+
+qLoRa:
+
+```
+accelerate launch -m axolotl.cli.train examples/tiny-llama/qlora.yml
+```
+
+Both take about 10 minutes to complete on a 4090.
--- a/examples/llama-2/tiny-llama.yml
+++ b/examples/llama-2/tiny-llama.yml
@@ -1,5 +1,4 @@
-base_model: PY007/TinyLlama-1.1B-intermediate-step-715k-1.5T
-
+base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
 model_type: LlamaForCausalLM
 tokenizer_type: LlamaTokenizer
 is_llama_derived_model: true
@@ -17,6 +16,7 @@ output_dir: ./lora-out

 sequence_len: 4096
 sample_packing: true
+pad_to_sequence_len: true

 adapter: lora
 lora_model_dir:
@@ -55,7 +55,6 @@ flash_attention: true

 warmup_steps: 10
 evals_per_epoch: 4
-eval_table_size:
 saves_per_epoch: 1
 debug:
 deepspeed:
@@ -63,6 +62,3 @@ weight_decay: 0.0
 fsdp:
 fsdp_config:
 special_tokens:
-  bos_token: "<s>"
-  eos_token: "</s>"
-  unk_token: "<unk>"
--- a/examples/tiny-llama/pretrain.yml
+++ b/examples/tiny-llama/pretrain.yml
@@ -0,0 +1,58 @@
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+
+model_type: LlamaForCausalLM
+tokenizer_type: LlamaTokenizer
+is_llama_derived_model: true
+
+load_in_8bit: false
+load_in_4bit: false
+strict: false
+
+max_steps: 200
+pretraining_dataset:
+  path: c4
+  name: en
+dataset_prepared_path:
+val_set_size: 0.0
+output_dir: ./model-out
+
+sequence_len: 2048
+sample_packing: true
+
+wandb_project:
+wandb_entity:
+wandb_watch:
+wandb_name:
+wandb_log_model:
+
+gradient_accumulation_steps: 4
+micro_batch_size: 2
+num_epochs: 4
+optimizer: adamw_bnb_8bit
+lr_scheduler: cosine
+learning_rate: 0.0002
+
+train_on_inputs: false
+group_by_length: false
+bf16: true
+fp16: false
+tf32: false
+
+gradient_checkpointing: true
+early_stopping_patience:
+resume_from_checkpoint:
+local_rank:
+logging_steps: 1
+xformers_attention:
+flash_attention: true
+
+warmup_steps: 10
+evals_per_epoch:
+eval_table_size:
+saves_per_epoch: 1
+debug:
+deepspeed:
+weight_decay: 0.0
+fsdp:
+fsdp_config:
+special_tokens:
--- a/examples/tiny-llama/qlora.yml
+++ b/examples/tiny-llama/qlora.yml
@@ -1,9 +1,7 @@
-base_model: wenge-research/yayi2-30b
-model_type: AutoModelForCausalLM
-tokenizer_type: AutoTokenizer
-is_mistral_derived_model: false
-trust_remote_code: true
-model_revision: refs/pr/5
+base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
+model_type: LlamaForCausalLM
+tokenizer_type: LlamaTokenizer
+is_llama_derived_model: true

 load_in_8bit: false
 load_in_4bit: true
@@ -12,36 +10,36 @@ strict: false
 datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
-dataset_prepared_path: last_run_prepared
-val_set_size: 0.1
+dataset_prepared_path:
+val_set_size: 0.05
 output_dir: ./qlora-out

 adapter: qlora
 lora_model_dir:

-sequence_len: 2048 # Fits in 40gb VRAM. Can easily do 4096 in A100 80 or a A6000
-sample_packing: false
-pad_to_sequence_len: false
+sequence_len: 4096
+sample_packing: true
+pad_to_sequence_len: true

 lora_r: 32
 lora_alpha: 16
 lora_dropout: 0.05
+lora_target_modules:
 lora_target_linear: true
 lora_fan_in_fan_out:
-lora_target_modules:

-wandb_project: yayi2
+wandb_project:
 wandb_entity:
 wandb_watch:
 wandb_name:
 wandb_log_model:

-gradient_accumulation_steps: 1
-micro_batch_size: 1
-num_epochs: 1
-optimizer: adamw_bnb_8bit
+gradient_accumulation_steps: 4
+micro_batch_size: 2
+num_epochs: 4
+optimizer: paged_adamw_32bit
 lr_scheduler: cosine
-learning_rate: 0.0005
+learning_rate: 0.0002

 train_on_inputs: false
 group_by_length: false
@@ -55,15 +53,10 @@ resume_from_checkpoint:
 local_rank:
 logging_steps: 1
 xformers_attention:
-flash_attention: false
-
-loss_watchdog_threshold: 5.0
-loss_watchdog_patience: 3
+flash_attention: true

 warmup_steps: 10
 evals_per_epoch: 4
-eval_table_size:
-eval_table_max_new_tokens: 128
 saves_per_epoch: 1
 debug:
 deepspeed:
@@ -71,6 +64,3 @@ weight_decay: 0.0
 fsdp:
 fsdp_config:
 special_tokens:
-  bos_token: "<s>"
-  eos_token: "</s>"
-  unk_token: "<unk>"
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,11 +1,10 @@
 --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
-auto-gptq==0.5.1
-packaging
-peft==0.6.0
-transformers==4.36.2
+packaging==23.2
+peft==0.7.0
+transformers @ git+https://github.com/huggingface/transformers.git@3cefac1d974db5e2825a0cb2b842883a628be7a0
 tokenizers==0.15.0
 bitsandbytes>=0.41.1
-accelerate==0.24.1
+accelerate @ git+https://github.com/huggingface/accelerate.git@0d2280dadc6a93413a5496613b7fdda3a4d2551b
 deepspeed
 addict
 fire
@@ -21,6 +20,7 @@ hf_transfer
 colorama
 numba
 numpy>=1.24.4
+mlflow
 # qlora things
 bert-score==0.3.13
 evaluate==0.4.0
@@ -33,7 +33,11 @@ fschat==0.2.34
 gradio==3.50.2
 tensorboard

+mamba-ssm==1.1.1
+
 # remote filesystems
 s3fs
 gcsfs
 # adlfs
+
+trl>=0.7.9
--- a/setup.py
+++ b/setup.py
@@ -11,16 +11,17 @@ def parse_requirements():
    with open("./requirements.txt", encoding="utf-8") as requirements_file:
        lines = [r.strip() for r in requirements_file.readlines()]
        for line in lines:
+            is_extras = (
+                "flash-attn" in line
+                or "flash-attention" in line
+                or "deepspeed" in line
+                or "mamba-ssm" in line
+            )
            if line.startswith("--extra-index-url"):
                # Handle custom index URLs
                _, url = line.split()
                _dependency_links.append(url)
-            elif (
-                "flash-attn" not in line
-                and "deepspeed" not in line
-                and line
-                and line[0] != "#"
-            ):
+            elif not is_extras and line and line[0] != "#":
                # Handle standard packages
                _install_requires.append(line)

@@ -51,11 +52,17 @@ setup(
        "flash-attn": [
            "flash-attn==2.3.3",
        ],
+        "fused-dense-lib": [
+            "fused-dense-lib  @ git+https://github.com/Dao-AILab/flash-attention@v2.3.3#subdirectory=csrc/fused_dense_lib",
+        ],
        "deepspeed": [
            "deepspeed",
        ],
        "mamba-ssm": [
            "mamba-ssm==1.0.1",
        ],
+        "auto-gptq": [
+            "auto-gptq==0.5.1",
+        ],
    },
 )
--- a/src/axolotl/cli/init.py
+++ b/src/axolotl/cli/init.py
@@ -2,6 +2,7 @@

 import importlib
 import logging
+import math
 import os
 import random
 import sys
@@ -16,6 +17,7 @@ import yaml
 # add src to the pythonpath so we don't need to pip install this
 from accelerate.commands.config import config_args
 from art import text2art
+from datasets import concatenate_datasets, load_dataset
 from huggingface_hub import HfApi
 from huggingface_hub.utils import LocalTokenNotFoundError
 from transformers import GenerationConfig, TextIteratorStreamer, TextStreamer
@@ -23,10 +25,15 @@ from transformers import GenerationConfig, TextIteratorStreamer, TextStreamer
 from axolotl.common.cli import TrainerCliArgs, load_model_and_tokenizer
 from axolotl.logging_config import configure_logging
 from axolotl.train import TrainDatasetMeta
-from axolotl.utils.config import normalize_config, validate_config
+from axolotl.utils.config import (
+    normalize_cfg_datasets,
+    normalize_config,
+    validate_config,
+)
 from axolotl.utils.data import prepare_dataset
 from axolotl.utils.dict import DictDefault
 from axolotl.utils.distributed import is_main_process
+from axolotl.utils.mlflow_ import setup_mlflow_env_vars
 from axolotl.utils.models import load_tokenizer
 from axolotl.utils.tokenization import check_dataset_labels
 from axolotl.utils.trainer import prepare_optim_env
@@ -71,7 +78,7 @@ def do_merge_lora(
    safe_serialization = cfg.save_safetensors is True

    LOG.info("running merge of LoRA with base model")
-    model = model.merge_and_unload()
+    model = model.merge_and_unload(progressbar=True)
    model.to(dtype=cfg.torch_dtype)

    if cfg.local_rank == 0:
@@ -79,6 +86,7 @@ def do_merge_lora(
        model.save_pretrained(
            str(Path(cfg.output_dir) / "merged"),
            safe_serialization=safe_serialization,
+            progressbar=True,
        )
        tokenizer.save_pretrained(str(Path(cfg.output_dir) / "merged"))

@@ -285,7 +293,12 @@ def load_cfg(config: Path = Path("examples/"), **kwargs):

    normalize_config(cfg)

+    normalize_cfg_datasets(cfg)
+
    setup_wandb_env_vars(cfg)
+
+    setup_mlflow_env_vars(cfg)
+
    return cfg


@@ -325,6 +338,94 @@ def load_datasets(
    )


+def load_rl_datasets(
+    *,
+    cfg: DictDefault,
+    cli_args: TrainerCliArgs,  # pylint: disable=unused-argument
+) -> TrainDatasetMeta:
+    train_datasets: List[Any] = []
+    for i, ds_cfg in enumerate(cfg.datasets):
+        train_datasets.insert(i, load_dataset(ds_cfg["path"], split=ds_cfg["split"]))
+    # eval_dataset = load_dataset(
+    #     cfg.test_datasets[0]["path"], split=cfg.test_datasets[0]["split"]
+    # )
+    eval_dataset = None
+
+    def argilla_apply_chatml(sample):  # pylint: disable=possibly-unused-variable
+        if "system" in sample and sample["system"]:
+            sample["prompt"] = (
+                f"<|im_start|>system\n{sample['system']}<|im_end|>\n"
+                f"<|im_start|>user\n{sample['instruction']}<|im_end|>\n<|im_start|>assistant\n"
+            )
+        else:
+            sample[
+                "prompt"
+            ] = f"<|im_start|>user\n{sample['instruction']}<|im_end|>\n<|im_start|>assistant\n"
+        sample["chosen"] = f"{sample['chosen_response']}<|im_end|>"
+        sample["rejected"] = f"{sample['rejected_response']}<|im_end|>"
+        return sample
+
+    def intel_apply_chatml(sample):  # pylint: disable=possibly-unused-variable
+        if "system" in sample and sample["system"]:
+            sample["prompt"] = (
+                f"<|im_start|>system\n{sample['system']}<|im_end|>\n"
+                f"<|im_start|>user\n{sample['question']}<|im_end|>\n<|im_start|>assistant\n"
+            )
+        else:
+            sample[
+                "prompt"
+            ] = f"<|im_start|>user\n{sample['question']}<|im_end|>\n<|im_start|>assistant\n"
+        sample["chosen"] = f"{sample['chosen']}<|im_end|>"
+        sample["rejected"] = f"{sample['rejected']}<|im_end|>"
+        return sample
+
+    def apply_chatml(sample):  # pylint: disable=possibly-unused-variable
+        if "system" in sample and sample["system"]:
+            sample["prompt"] = (
+                f"<|im_start|>system\n{sample['system']}<|im_end|>\n"
+                f"<|im_start|>user\n{sample['prompt']}<|im_end|>\n<|im_start|>assistant\n"
+            )
+        else:
+            sample[
+                "prompt"
+            ] = f"<|im_start|>user\n{sample['prompt']}<|im_end|>\n<|im_start|>assistant\n"
+        sample["chosen"] = f"{sample['chosen']}<|im_end|>"
+        sample["rejected"] = f"{sample['rejected']}<|im_end|>"
+        return sample
+
+    def ultra_apply_chatml(sample):  # pylint: disable=possibly-unused-variable
+        if "system" in sample and sample["system"]:
+            sample["prompt"] = (
+                f"<|im_start|>system\n{sample['system']}<|im_end|>\n"
+                f"<|im_start|>user\n{sample['prompt']}<|im_end|>\n<|im_start|>assistant\n"
+            )
+        else:
+            sample[
+                "prompt"
+            ] = f"<|im_start|>user\n{sample['prompt']}<|im_end|>\n<|im_start|>assistant\n"
+        sample["chosen"] = f"{sample['chosen'][1]['content']}<|im_end|>"
+        sample["rejected"] = f"{sample['rejected'][1]['content']}<|im_end|>"
+        return sample
+
+    for i, data_set in enumerate(train_datasets):
+        _type = cfg.datasets[i]["type"]
+        ds_type_fn = locals()[_type]
+        train_datasets[i] = data_set.map(ds_type_fn)
+    train_dataset = concatenate_datasets(train_datasets)
+
+    # eval_dataset = eval_dataset.map(intel_apply_chatml)
+
+    total_num_steps = int(
+        math.ceil(len(train_dataset) * cfg.num_epochs / cfg.batch_size)
+    )
+
+    return TrainDatasetMeta(
+        train_dataset=train_dataset,
+        eval_dataset=eval_dataset,
+        total_num_steps=total_num_steps,
+    )
+
+
 def check_accelerate_default_config():
    if Path(config_args.default_yaml_config_file).exists():
        LOG.warning(
--- a/src/axolotl/cli/merge_lora.py
+++ b/src/axolotl/cli/merge_lora.py
@@ -25,9 +25,20 @@ def do_cli(config: Path = Path("examples/"), **kwargs):
        load_in_8bit=False,
        load_in_4bit=False,
        flash_attention=False,
-        **kwargs
+        **kwargs,
    )

+    if not parsed_cfg.lora_model_dir and parsed_cfg.output_dir:
+        parsed_cfg.lora_model_dir = parsed_cfg.output_dir
+    if not Path(parsed_cfg.lora_model_dir).exists():
+        raise ValueError(
+            f"Target directory for merge: `{parsed_cfg.lora_model_dir}` does not exist."
+        )
+
+    parsed_cfg.load_in_4bit = False
+    parsed_cfg.load_in_8bit = False
+    parsed_cfg.flash_attention = False
+
    do_merge_lora(cfg=parsed_cfg, cli_args=parsed_cli_args)


--- a/src/axolotl/cli/preprocess.py
+++ b/src/axolotl/cli/preprocess.py
@@ -31,6 +31,7 @@ def do_cli(config: Path = Path("examples/"), **kwargs):
    parsed_cli_args, _ = parser.parse_args_into_dataclasses(
        return_remaining_strings=True
    )
+
    if not parsed_cfg.dataset_prepared_path:
        msg = (
            Fore.RED
--- a/src/axolotl/cli/train.py
+++ b/src/axolotl/cli/train.py
@@ -12,6 +12,7 @@ from axolotl.cli import (
    check_user_token,
    load_cfg,
    load_datasets,
+    load_rl_datasets,
    print_axolotl_text_art,
 )
 from axolotl.common.cli import TrainerCliArgs
@@ -30,7 +31,11 @@ def do_cli(config: Path = Path("examples/"), **kwargs):
    parsed_cli_args, _ = parser.parse_args_into_dataclasses(
        return_remaining_strings=True
    )
-    dataset_meta = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
+
+    if parsed_cfg.rl:
+        dataset_meta = load_rl_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
+    else:
+        dataset_meta = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)


--- a/src/axolotl/core/trainer_builder.py
+++ b/src/axolotl/core/trainer_builder.py
@@ -1,3 +1,4 @@
+# pylint: disable=too-many-lines
 """
 Builder for the training args and trainer
 """
@@ -20,6 +21,7 @@ from torch.optim.lr_scheduler import OneCycleLR
 from torch.utils.data import BatchSampler, DataLoader, RandomSampler, SequentialSampler
 from transformers import EarlyStoppingCallback, Trainer, TrainingArguments
 from transformers.trainer_utils import seed_worker
+from trl import DPOTrainer

 from axolotl.monkeypatch.relora import ReLoRACallback, ReLoRAScheduler
 from axolotl.utils.callbacks import (
@@ -33,10 +35,14 @@ from axolotl.utils.callbacks import (
 )
 from axolotl.utils.collators import (
    BatchSamplerDataCollatorForSeq2Seq,
+    DataCollatorForSeq2Seq,
    MambaDataCollator,
 )
-from axolotl.utils.samplers import MultipackBatchSampler
-from axolotl.utils.schedulers import get_cosine_schedule_with_quadratic_warmup
+from axolotl.utils.samplers import MultipackBatchSampler, get_dataset_lengths
+from axolotl.utils.schedulers import (
+    get_cosine_schedule_with_min_lr,
+    get_cosine_schedule_with_quadratic_warmup,
+)

 try:
    import torch._dynamo  # pylint: disable=ungrouped-imports
@@ -59,6 +65,12 @@ class AxolotlTrainingArguments(TrainingArguments):
        default=False,
        metadata={"help": "Use quadratic warmup for cosine scheduling."},
    )
+    pretraining: bool = field(
+        default=False,
+        metadata={
+            "help": "Indicates to trainer whether we are doing continued pretraining."
+        },
+    )
    sample_packing: bool = field(
        default=False,
        metadata={"help": "Use sample packing for efficient training."},
@@ -112,6 +124,10 @@ class AxolotlTrainingArguments(TrainingArguments):
        default=None,
        metadata={"help": "prefetch_factor argument to the dataloader"},
    )
+    cosine_min_lr_ratio: Optional[float] = field(
+        default=None,
+        metadata={"help": "Minimum learning rate is min_lr_ratio * learning_rate"},
+    )


 class AxolotlTrainer(Trainer):
@@ -122,10 +138,19 @@ class AxolotlTrainer(Trainer):
    args = None  # type: AxolotlTrainingArguments
    tag_names = ["axolotl"]

-    def __init__(self, *args, num_epochs=1, bench_data_collator=None, **kwargs):
+    def __init__(
+        self,
+        *_args,
+        num_epochs=1,
+        bench_data_collator=None,
+        eval_data_collator=None,
+        **kwargs
+    ):
        self.num_epochs = num_epochs
        self.bench_data_collator = bench_data_collator
-        super().__init__(*args, **kwargs)
+        self.eval_data_collator = eval_data_collator
+        super().__init__(*_args, **kwargs)
+        self.train_data_collator = self.data_collator

    def create_scheduler(
        self, num_training_steps: int, optimizer: torch.optim.Optimizer = None
@@ -151,23 +176,29 @@ class AxolotlTrainer(Trainer):
                    num_warmup_steps=self.args.get_warmup_steps(num_training_steps),
                    num_training_steps=num_training_steps,
                )
+            elif self.args.lr_scheduler_type == "cosine" and self.args.cosine_min_lr_ratio is not None:
+                assert 0 <= self.args.cosine_min_lr_ratio <= 1.0, "cosine_min_lr_ratio must be between 0.0 and 1.0"
+                if self.args.deepspeed:
+                    LOG.warning("Using cosine scheduler with deepspeed. This may be ignored if a scheduler is set \
+                                in the deepspeed JSON")
+                self.lr_scheduler = get_cosine_schedule_with_min_lr(  # pylint: disable=attribute-defined-outside-init
+                    optimizer,
+                    num_warmup_steps=self.args.get_warmup_steps(num_training_steps),
+                    num_training_steps=num_training_steps,
+                    min_lr_ratio=self.args.cosine_min_lr_ratio,
+                )
            else:
                return super().create_scheduler(num_training_steps, optimizer)
        return self.lr_scheduler

    def _get_train_sampler(self) -> Optional[torch.utils.data.Sampler]:
-        if self.args.sample_packing:
+        if self.args.sample_packing and not self.args.pretraining:
            return MultipackBatchSampler(
                RandomSampler(self.train_dataset),
                self.args.train_batch_size,
                drop_last=True,
                batch_max_len=self._train_batch_size * self.args.max_seq_length,
-                lengths=(
-                    self.train_dataset.data.column("position_ids")
-                    .to_pandas()
-                    .apply(lambda x: x[-1] + 1)
-                    .values
-                ),
+                lengths=get_dataset_lengths(self.train_dataset),
                packing_efficiency_estimate=self.args.sample_packing_efficiency,
            )
        return super()._get_train_sampler()
@@ -181,18 +212,13 @@ class AxolotlTrainer(Trainer):
                self.args.per_device_eval_batch_size,
                drop_last=True,
                batch_max_len=self.args.eval_batch_size * self.args.max_seq_length,
-                lengths=(
-                    eval_dataset.data.column("position_ids")
-                    .to_pandas()
-                    .apply(lambda x: x[-1] + 1)
-                    .values
-                ),
+                lengths=get_dataset_lengths(eval_dataset),
                packing_efficiency_estimate=self.args.sample_packing_efficiency,
            )
        return super()._get_eval_sampler(eval_dataset)

    def get_train_dataloader(self) -> DataLoader:
-        if self.args.sample_packing:
+        if self.args.sample_packing and not self.args.pretraining:
            train_dataset = self.train_dataset
            train_dataset = train_dataset.remove_columns(["length"])
            data_collator = self.data_collator
@@ -223,6 +249,16 @@ class AxolotlTrainer(Trainer):
        return super().get_train_dataloader()

    def get_eval_dataloader(self, eval_dataset: Optional[Dataset] = None) -> DataLoader:
+        if self.args.sample_packing and self.args.eval_sample_packing is False:
+            self.data_collator = (  # pylint: disable=attribute-defined-outside-init
+                self.eval_data_collator
+            )
+            dataloader = super().get_eval_dataloader(eval_dataset)
+            self.data_collator = (  # pylint: disable=attribute-defined-outside-init
+                self.train_data_collator
+            )
+            return dataloader
+
        if self.args.sample_packing and self.args.eval_sample_packing is not False:
            eval_dataset = (
                eval_dataset if eval_dataset is not None else self.eval_dataset
@@ -253,6 +289,7 @@ class AxolotlTrainer(Trainer):
            return self.accelerator.prepare_data_loader(
                DataLoader(eval_dataset, **dataloader_params)
            )
+
        return super().get_eval_dataloader(eval_dataset)

    def _get_bench_sampler(
@@ -420,12 +457,21 @@ class TrainerBuilderBase(abc.ABC):

    _train_dataset = None
    _eval_dataset = None
+    _model_ref = None

    def __init__(self, cfg, model, tokenizer):
        self.cfg = cfg
        self.model = model
        self.tokenizer = tokenizer

+    @property
+    def model_ref(self):
+        return self._model_ref
+
+    @model_ref.setter
+    def model_ref(self, model):
+        self._model_ref = model
+
    @property
    def train_dataset(self):
        return self._train_dataset
@@ -566,6 +612,14 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            training_arguments_kwargs[
                "gradient_checkpointing"
            ] = self.cfg.gradient_checkpointing
+            if self.cfg.gradient_checkpointing_kwargs:
+                training_arguments_kwargs[
+                    "gradient_checkpointing_kwargs"
+                ] = self.cfg.gradient_checkpointing_kwargs
+            else:
+                training_arguments_kwargs["gradient_checkpointing_kwargs"] = {
+                    "use_reentrant": False
+                }
        if self.cfg.fsdp:
            training_arguments_kwargs["fsdp"] = self.cfg.fsdp
            if self.cfg.fsdp_config:
@@ -593,6 +647,7 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            training_arguments_kwargs["hub_model_id"] = self.cfg.hub_model_id
            training_arguments_kwargs["push_to_hub"] = True
            training_arguments_kwargs["hub_private_repo"] = True
+            training_arguments_kwargs["hub_always_push"] = True

            if self.cfg.hub_strategy:
                training_arguments_kwargs["hub_strategy"] = self.cfg.hub_strategy
@@ -617,6 +672,12 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            training_arguments_kwargs[
                "dataloader_prefetch_factor"
            ] = self.cfg.dataloader_prefetch_factor
+        if self.cfg.dataloader_drop_last is not None:
+            training_arguments_kwargs[
+                "dataloader_drop_last"
+            ] = self.cfg.dataloader_drop_last
+        elif self.cfg.sample_packing and self.cfg.eval_sample_packing is False:
+            training_arguments_kwargs["dataloader_drop_last"] = True

        if self.cfg.val_set_size == 0:
            # no eval set, so don't eval
@@ -713,7 +774,12 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            False if self.cfg.ddp else None
        )
        training_arguments_kwargs["group_by_length"] = self.cfg.group_by_length
-        training_arguments_kwargs["report_to"] = "wandb" if self.cfg.use_wandb else None
+        report_to = None
+        if self.cfg.use_wandb:
+            report_to = "wandb"
+        if self.cfg.use_mlflow:
+            report_to = "mlflow"
+        training_arguments_kwargs["report_to"] = report_to
        training_arguments_kwargs["run_name"] = (
            self.cfg.wandb_name if self.cfg.use_wandb else None
        )
@@ -729,6 +795,7 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
        training_arguments_kwargs["lr_scheduler_kwargs"] = (
            self.cfg.lr_scheduler_kwargs if self.cfg.lr_scheduler_kwargs else {}
        )
+        training_arguments_kwargs["cosine_min_lr_ratio"] = self.cfg.cosine_min_lr_ratio
        training_arguments_kwargs["weight_decay"] = (
            self.cfg.weight_decay if self.cfg.weight_decay is not None else 0.0
        )
@@ -749,6 +816,7 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            training_arguments_kwargs
        )
        training_arguments_kwargs["model_type"] = self.cfg.model_config_type
+        training_arguments_kwargs["pretraining"] = bool(self.cfg.pretraining_dataset)

        if self.cfg.neftune_noise_alpha is not None:
            training_arguments_kwargs[
@@ -789,7 +857,10 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):
            train_dataset=self.train_dataset,
            eval_dataset=self.eval_dataset,
            args=training_args,
-            data_collator=self.build_collator(**data_collator_kwargs),
+            data_collator=self.build_collator(training_args, **data_collator_kwargs),
+            eval_data_collator=self.build_collator(
+                training_args, is_eval=True, **data_collator_kwargs
+            ),
            bench_data_collator=transformers.DataCollatorForSeq2Seq(
                self.tokenizer,
                return_tensors="pt",
@@ -810,12 +881,125 @@ class HFCausalTrainerBuilder(TrainerBuilderBase):

        return trainer

-    def build_collator(self, **kwargs):
+    def build_collator(
+        self, training_args: AxolotlTrainingArguments, is_eval=False, **kwargs
+    ):
+        if training_args.pretraining:
+            return None
+
        if self.cfg.model_config_type == "mamba":
            return MambaDataCollator(tokenizer=self.tokenizer)

-        return BatchSamplerDataCollatorForSeq2Seq(
+        use_batch_sampler_collator = False
+        if is_eval is False and training_args.sample_packing:
+            use_batch_sampler_collator = True
+        if is_eval and training_args.eval_sample_packing:
+            use_batch_sampler_collator = True
+
+        if use_batch_sampler_collator:
+            return BatchSamplerDataCollatorForSeq2Seq(
+                self.tokenizer,
+                return_tensors="pt",
+                **kwargs,
+            )
+
+        return DataCollatorForSeq2Seq(
            self.tokenizer,
            return_tensors="pt",
            **kwargs,
        )
+
+
+class HFDPOTrainerBuilder(TrainerBuilderBase):
+    """
+    Trainer factory class for DPO Trainer
+    """
+
+    def get_callbacks(self):
+        callbacks = []
+        return callbacks
+
+    def get_post_trainer_create_callbacks(self, trainer):
+        callbacks = []
+        return callbacks
+
+    def build_training_arguments(self, total_num_steps):
+        training_args_kwargs = {}
+        for arg in [
+            "adam_beta1",
+            "adam_beta2",
+            "adam_epsilon",
+            "dataloader_num_workers",
+            "dataloader_pin_memory",
+        ]:
+            if hasattr(self.cfg, arg) and getattr(self.cfg, arg) is not None:
+                training_args_kwargs[arg] = getattr(self.cfg, arg)
+        training_args = TrainingArguments(
+            per_device_train_batch_size=self.cfg.micro_batch_size,
+            max_steps=total_num_steps,
+            remove_unused_columns=False,
+            gradient_accumulation_steps=self.cfg.gradient_accumulation_steps,
+            learning_rate=self.cfg.learning_rate,
+            evaluation_strategy="no",
+            # eval_steps=self.cfg.eval_steps,
+            save_strategy="steps",
+            save_steps=self.cfg.save_steps,
+            output_dir=self.cfg.output_dir,
+            warmup_steps=self.cfg.warmup_steps,
+            bf16=True,
+            gradient_checkpointing=self.cfg.gradient_checkpointing,
+            gradient_checkpointing_kwargs={"use_reentrant": False},
+            logging_first_step=True,
+            logging_steps=1,
+            optim=self.cfg.optimizer,
+            save_total_limit=self.cfg.save_total_limit or 5,
+            **training_args_kwargs,
+        )
+
+        return training_args
+
+    def build(self, total_num_steps):
+        training_args = self.build_training_arguments(total_num_steps)
+        dpo_trainer_kwargs = {}
+        if self.cfg.rl == "ipo":
+            dpo_trainer_kwargs["loss_type"] = "ipo"
+            if self.cfg.dpo_label_smoothing:
+                dpo_trainer_kwargs["label_smoothing"] = self.cfg.dpo_label_smoothing
+        elif self.cfg.rl == "kto_pair":
+            dpo_trainer_kwargs["loss_type"] = "kto_pair"
+
+        dpo_trainer = DPOTrainer(
+            self.model,
+            self.model_ref,
+            args=training_args,
+            beta=self.cfg.dpo_beta or 0.1,
+            train_dataset=self.train_dataset,
+            # eval_dataset=self.eval_dataset,
+            eval_dataset=None,
+            tokenizer=self.tokenizer,
+            max_length=self.cfg.sequence_len,
+            max_target_length=None,
+            max_prompt_length=self.cfg.sequence_len,
+            generate_during_eval=True,
+            **dpo_trainer_kwargs,
+        )
+
+        return dpo_trainer
+
+
+class HFPPOTrainerBuilder(TrainerBuilderBase):
+    """
+    HF Factory class for PPO Trainer
+    """
+
+    def get_callbacks(self):
+        callbacks = []
+        return callbacks
+
+    def get_post_trainer_create_callbacks(self, trainer):
+        callbacks = []
+        return callbacks
+
+    def build(self, total_num_steps):
+        # build PPOConfig
+        pass
--- a/src/axolotl/core/trainers/init.py
+++ b/src/axolotl/core/trainers/init.py
--- a/src/axolotl/core/trainers/trl.py
+++ b/src/axolotl/core/trainers/trl.py
@@ -0,0 +1,66 @@
+"""
+module for TRL PPO training
+"""
+import torch
+from tqdm import tqdm
+from trl import PPOTrainer
+
+
+class TRLPPOTrainer(PPOTrainer):
+    """
+    wrapper for ppo trainer to handle customizations
+    """
+
+    def train(
+        self,
+        reward_pipe,
+        resume_from_checkpoint=None,  # pylint: disable=unused-argument
+    ):
+        generation_kwargs = {
+            "min_length": -1,
+            "top_k": 0.0,
+            "top_p": 1.0,
+            "do_sample": True,
+            "pad_token_id": self.tokenizer.eos_token_id,
+            "max_new_tokens": 32,
+        }
+        sent_kwargs = {
+            "return_all_scores": True,
+            "function_to_apply": "none",
+            "batch_size": 16,
+        }
+
+        for epoch, batch in tqdm(  # pylint: disable=unused-variable
+            enumerate(self.dataloader)
+        ):
+            query_tensors = batch["input_ids"]
+
+            # generate model response
+            response_tensors, ref_response_tensors = self.generate(
+                query_tensors,
+                return_prompt=False,
+                generate_ref_response=True,
+                **generation_kwargs
+            )
+            batch["response"] = self.tokenizer.batch_decode(response_tensors)
+            batch["ref_response"] = self.tokenizer.batch_decode(ref_response_tensors)
+
+            # Compute sentiment score
+            texts = [q + r for q, r in zip(batch["query"], batch["response"])]
+            pipe_outputs = reward_pipe(texts, **sent_kwargs)
+            rewards = [torch.tensor(output[1]["score"]) for output in pipe_outputs]
+            ref_texts = [q + r for q, r in zip(batch["query"], batch["ref_response"])]
+            ref_pipe_outputs = reward_pipe(ref_texts, **sent_kwargs)
+            ref_rewards = [
+                torch.tensor(output[1]["score"]) for output in ref_pipe_outputs
+            ]
+            batch["ref_rewards"] = ref_rewards
+
+            # Run PPO step
+            stats = self.step(query_tensors, response_tensors, rewards)
+            self.log_stats(
+                stats,
+                batch,
+                rewards,
+                columns_to_log=["query", "response", "ref_response", "ref_rewards"],
+            )
--- a/src/axolotl/models/mamba/init.py
+++ b/src/axolotl/models/mamba/init.py
@@ -2,8 +2,20 @@
 Modeling module for Mamba models
 """

+import importlib
+
+
+def check_mamba_ssm_installed():
+    mamba_ssm_spec = importlib.util.find_spec("mamba_ssm")
+    if mamba_ssm_spec is None:
+        raise ImportError(
+            "MambaLMHeadModel requires mamba_ssm. Please install it with `pip install -e .[mamba-ssm]`"
+        )
+

 def fix_mamba_attn_for_loss():
+    check_mamba_ssm_installed()
+
    from mamba_ssm.models import mixer_seq_simple

    from .modeling_mamba import MambaLMHeadModel as MambaLMHeadModelFixed
--- a/src/axolotl/models/phi/modeling_phi.py
+++ b/src/axolotl/models/phi/modeling_phi.py
@@ -9,27 +9,32 @@ from __future__ import annotations

 import math
 from dataclasses import dataclass, field
-from typing import Any, Dict, Optional, Tuple, Union
+from typing import Any, Callable, Dict, Optional, Tuple, Union

 import torch
 import torch.nn as nn
 from einops import rearrange, repeat
+from torch.utils.checkpoint import checkpoint
 from transformers import PretrainedConfig, PreTrainedModel
 from transformers.activations import ACT2FN
 from transformers.modeling_outputs import CausalLMOutputWithPast

-from ...monkeypatch.utils import get_cu_seqlens_from_pos_ids
 from .configuration_phi import PhiConfig

 try:
    from flash_attn.bert_padding import pad_input, unpad_input
    from flash_attn.layers.rotary import RotaryEmbedding as FlashRotaryEmbedding
    from flash_attn.modules.mha import FlashCrossAttention, FlashSelfAttention
-    from flash_attn.ops.fused_dense import FusedDense
-except:  # noqa: E722
+except ImportError:
    pad_input, unpad_input = None, None
    FlashRotaryEmbedding = None
    FlashSelfAttention, FlashCrossAttention = None, None
+
+# this is in a seperate try/except block since sometimes fused_dense isn't available
+# and it shouldn't completely disable flash attn when it isn't
+try:
+    from flash_attn.ops.fused_dense import FusedDense
+except ImportError:
    FusedDense = None


@@ -224,7 +229,9 @@ class RotaryEmbedding(nn.Module):

        # Initialize cached attributes since ONNX can't rely on dynamic initialization
        self._update_cos_sin_cache(
-            max_position_embeddings, device=device, dtype=torch.float32
+            max_position_embeddings,
+            device=device,
+            dtype=torch.float32,
        )

    def _compute_inv_freq(self, device: Optional[str] = None) -> torch.FloatTensor:
@@ -281,34 +288,32 @@ class RotaryEmbedding(nn.Module):
        seqlen_offset: int = 0,
        **kwargs,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
-        seq_start = seqlen_offset
-        seq_end = seq_start + qkv.shape[1]
-
        if (
-            self._cos_cached.device != qkv.device
+            self._seq_len_cached < qkv.shape[1] + seqlen_offset
+            or self._cos_cached.device != qkv.device
            or self._cos_cached.dtype != qkv.dtype
            or (self.training and self._cos_cached.is_inference())
        ):
            self._update_cos_sin_cache(
-                self.max_position_embeddings, device=qkv.device, dtype=qkv.dtype
+                qkv.shape[1] + seqlen_offset, device=qkv.device, dtype=qkv.dtype
            )

        if kv is None:
            return _apply_rotary_emb_qkv(
                qkv,
-                self._cos_cached[seq_start:seq_end],
-                self._sin_cached[seq_start:seq_end],
+                self._cos_cached[seqlen_offset:],
+                self._sin_cached[seqlen_offset:],
            )
        else:
            q = _apply_rotary_emb(
                qkv,
-                self._cos_cached[seq_start:seq_end],
-                self._sin_cached[seq_start:seq_end],
+                self._cos_cached[seqlen_offset:],
+                self._sin_cached[seqlen_offset:],
            )
            kv = _apply_rotary_emb_kv(
                kv,
-                self._cos_cached[seq_start:seq_end],
-                self._sin_cached[seq_start:seq_end],
+                self._cos_cached[seqlen_offset:],
+                self._sin_cached[seqlen_offset:],
            )

            return q, kv
@@ -511,7 +516,7 @@ def _update_kv_cache(
    num_heads, head_dim = kv.shape[-2:]

    if layer_idx not in inference_params.key_value_memory_dict:
-        kv_cache = torch.empty(
+        inference_params.key_value_memory_dict[layer_idx] = torch.empty(
            inference_params.max_batch_size,
            inference_params.max_seqlen,
            2,
@@ -520,9 +525,6 @@ def _update_kv_cache(
            dtype=kv.dtype,
            device=kv.device,
        )
-        inference_params.key_value_memory_dict[layer_idx] = kv_cache
-    else:
-        kv_cache = inference_params.key_value_memory_dict[layer_idx]

    batch_start = inference_params.batch_size_offset
    batch_end = batch_start + kv.shape[0]
@@ -530,8 +532,19 @@ def _update_kv_cache(
    sequence_start = inference_params.seqlen_offset
    sequence_end = sequence_start + kv.shape[1]

-    kv_cache[batch_start:batch_end, sequence_start:sequence_end, ...] = kv
-    kv = kv_cache[batch_start:batch_end, :sequence_end, ...]
+    # When the current sequence length is equal to or larger than the maximum sequence length,
+    # we need to concatenate the current `kv` with the cached `kv` to expand its length
+    if sequence_end >= inference_params.max_seqlen:
+        inference_params.key_value_memory_dict[layer_idx] = torch.concatenate(
+            (inference_params.key_value_memory_dict[layer_idx], kv), dim=1
+        )
+
+    inference_params.key_value_memory_dict[layer_idx][
+        batch_start:batch_end, sequence_start:sequence_end, ...
+    ] = kv
+    kv = inference_params.key_value_memory_dict[layer_idx][
+        batch_start:batch_end, :sequence_end, ...
+    ]

    return kv

@@ -624,13 +637,10 @@ class MHA(nn.Module):
        self.layer_idx = layer_idx
        self.return_residual = return_residual
        self.checkpointing = checkpointing
+        self._gradient_checkpointing_func = None

    def _forward_self_attn(
-        self,
-        x: torch.FloatTensor,
-        key_padding_mask: Optional[torch.BoolTensor],
-        cu_seqlens: Optional[torch.LongTensor] = None,
-        max_seqlen: Optional[int] = None,
+        self, x: torch.FloatTensor, key_padding_mask: Optional[torch.BoolTensor]
    ) -> torch.FloatTensor:
        qkv = self.Wqkv(x)
        qkv = rearrange(
@@ -643,20 +653,21 @@ class MHA(nn.Module):
        if self.flash_attn:
            batch_size, seqlen = qkv.shape[0], qkv.shape[1]

-            if (
-                key_padding_mask is not None
-                and cu_seqlens is None
-                and max_seqlen is None
-            ):
+            cu_seqlens, max_seqlen = None, None
+            if key_padding_mask is not None:
                # If `key_padding_mask` is supplied, we need to unpad the input and retrieve
                # the `cu_seqlens` and `max_seqlen` to be used by `flash-attn`
                qkv, indices, cu_seqlens, max_seqlen = unpad_input(
                    qkv, key_padding_mask
                )

-            if self.checkpointing:
-                attn_output = torch.utils.checkpoint.checkpoint(
-                    self.inner_attn, qkv, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen
+            if self.checkpointing and self.training:
+                attn_output = self._gradient_checkpointing_func(
+                    self.inner_attn,
+                    qkv,
+                    cu_seqlens=cu_seqlens,
+                    max_seqlen=max_seqlen,
+                    use_reentrant=False,
                )
            else:
                attn_output = self.inner_attn(
@@ -670,9 +681,12 @@ class MHA(nn.Module):
                else attn_output
            )

-        if self.checkpointing:
-            return torch.utils.checkpoint.checkpoint(
-                self.inner_attn, qkv, key_padding_mask=key_padding_mask
+        if self.checkpointing and self.training:
+            return self._gradient_checkpointing_func(
+                self.inner_attn,
+                qkv,
+                key_padding_mask=key_padding_mask,
+                use_reentrant=False,
            )

        return self.inner_attn(qkv, key_padding_mask=key_padding_mask)
@@ -725,8 +739,8 @@ class MHA(nn.Module):
                    q, key_padding_mask
                )

-            if self.checkpointing:
-                attn_output = torch.utils.checkpoint.checkpoint(
+            if self.checkpointing and self.training:
+                attn_output = self._gradient_checkpointing_func(
                    self.inner_cross_attn,
                    q,
                    kv,
@@ -735,6 +749,7 @@ class MHA(nn.Module):
                    max_seqlen=max_seqlen_q,
                    cu_seqlens_k=cu_seqlens_k,
                    max_seqlen_k=max_seqlen_k,
+                    use_reentrant=False,
                )
            else:
                attn_output = self.inner_cross_attn(
@@ -753,13 +768,14 @@ class MHA(nn.Module):
                else attn_output
            )

-        if self.checkpointing:
-            return torch.utils.checkpoint.checkpoint(
+        if self.checkpointing and self.training:
+            return self._gradient_checkpointing_func(
                self.inner_cross_attn,
                q,
                kv,
                key_padding_mask=key_padding_mask,
                causal=causal,
+                use_reentrant=False,
            )

        return self.inner_cross_attn(
@@ -771,11 +787,8 @@ class MHA(nn.Module):
        x: torch.FloatTensor,
        past_key_values: Optional[InferenceParams] = None,
        attention_mask: Optional[Union[torch.LongTensor, torch.BoolTensor]] = None,
-        cu_seqlens: Optional[torch.LongTensor] = None,
-        max_seqlen: Optional[int] = None,
        **kwargs,
    ) -> Tuple[torch.FloatTensor, torch.FloatTensor]:
-        # TODO: Need an alternative way for dynamic control flow: torch.any(~attention_mask.bool())
        if attention_mask is not None:
            attention_mask = attention_mask.bool()
        else:
@@ -785,18 +798,12 @@ class MHA(nn.Module):
        if self.n_head == self.n_head_kv:
            if past_key_values is None:
                # If `past_key_values` are not supplied, we run self-attention
-                attn_output = self._forward_self_attn(
-                    x, attention_mask, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen
-                )
+                attn_output = self._forward_self_attn(x, attention_mask)
            else:
                # If `past_key_values` are supplied, it means that we might have cached values and
                # could take advantage of cross-attention
                attn_output = self._forward_cross_attn(
-                    x,
-                    past_key_values,
-                    attention_mask,
-                    cu_seqlens=cu_seqlens,
-                    max_seqlen=max_seqlen,
+                    x, past_key_values, attention_mask
                )
        # MQA / GQA
        else:
@@ -830,6 +837,8 @@ class ParallelBlock(nn.Module):

        self.mixer = MHA(config, layer_idx=block_idx)
        self.mlp = MLP(config)
+        self.checkpointing = False
+        self._gradient_checkpointing_func = None

    def forward(
        self,
@@ -838,23 +847,52 @@ class ParallelBlock(nn.Module):
        attention_mask: Optional[torch.BoolTensor] = None,
        **kwargs,
    ) -> torch.FloatTensor:
-        residual = hidden_states
-        hidden_states = self.ln(hidden_states)
-
-        attn_outputs = self.mixer(
+        def _forward(
+            mixer,
+            resid_dropout,
+            mlp,
+            ln,
            hidden_states,
-            past_key_values=past_key_values,
-            attention_mask=attention_mask,
+            past_key_values,
+            attention_mask,
+        ):
+            residual = hidden_states
+            hidden_states = ln(hidden_states)
+
+            attn_outputs = mixer(
+                hidden_states,
+                past_key_values=past_key_values,
+                attention_mask=attention_mask,
+            )
+            if isinstance(attn_outputs, tuple):
+                attn_outputs = attn_outputs[0]
+
+            attn_outputs = resid_dropout(attn_outputs)
+            feed_forward_hidden_states = resid_dropout(mlp(hidden_states))
+
+            return attn_outputs + feed_forward_hidden_states + residual
+
+        if self.training and self.checkpointing:
+            return self._gradient_checkpointing_func(
+                _forward,
+                self.mixer,
+                self.resid_dropout,
+                self.mlp,
+                self.ln,
+                hidden_states,
+                past_key_values,
+                attention_mask,
+            )
+
+        return _forward(
+            self.mixer,
+            self.resid_dropout,
+            self.mlp,
+            self.ln,
+            hidden_states,
+            past_key_values,
+            attention_mask,
        )
-        if isinstance(attn_outputs, tuple):
-            attn_outputs = attn_outputs[0]
-
-        attn_outputs = self.resid_dropout(attn_outputs)
-        feed_forward_hidden_states = self.resid_dropout(self.mlp(hidden_states))
-
-        hidden_states = attn_outputs + feed_forward_hidden_states + residual
-
-        return hidden_states


 class CausalLMHead(nn.Module):
@@ -911,7 +949,7 @@ class PhiPreTrainedModel(PreTrainedModel):

    config_class = PhiConfig
    base_model_prefix = "transformer"
-    supports_gradient_checkpointing = False
+    supports_gradient_checkpointing = True
    _no_split_modules = ["ParallelBlock"]

    def __init__(self, *inputs, **kwargs) -> None:
@@ -931,6 +969,14 @@ class PhiPreTrainedModel(PreTrainedModel):
                module.bias.data.zero_()
            module.weight.data.fill_(1.0)

+    def _set_gradient_checkpointing(
+        self, enable: bool = True, gradient_checkpointing_func: Callable = checkpoint
+    ):
+        for module in self.modules():
+            if hasattr(module, "checkpointing"):
+                module._gradient_checkpointing_func = gradient_checkpointing_func
+                module.checkpointing = enable
+
    def prepare_inputs_for_generation(
        self,
        input_ids: torch.LongTensor,
@@ -951,7 +997,7 @@ class PhiPreTrainedModel(PreTrainedModel):
            )
        else:
            # Assume that `past_key_values` has cached all tokens up to the last token in `input_ids`
-            past_key_values.seqlen_offset = len(input_ids[0]) - 1
+            past_key_values.seqlen_offset = input_ids.shape[1] - 1
            input_ids = input_ids[:, -1].unsqueeze(-1)

        return {
@@ -988,8 +1034,6 @@ class PhiModel(PhiPreTrainedModel):
        input_ids: torch.LongTensor,
        past_key_values: Optional[Union[torch.FloatTensor, InferenceParams]] = None,
        attention_mask: Optional[torch.BoolTensor] = None,
-        cu_seqlens: Optional[torch.LongTensor] = None,
-        max_seqlen: Optional[int] = None,
    ) -> torch.FloatTensor:
        hidden_states = self.embd(input_ids)

@@ -998,8 +1042,6 @@ class PhiModel(PhiPreTrainedModel):
                hidden_states,
                past_key_values=past_key_values,
                attention_mask=attention_mask,
-                cu_seqlens=cu_seqlens,
-                max_seqlen=max_seqlen,
            )

        return hidden_states
@@ -1034,23 +1076,10 @@ class PhiForCausalLM(PhiPreTrainedModel):
        past_key_values: Optional[Union[torch.FloatTensor, InferenceParams]] = None,
        attention_mask: Optional[torch.BoolTensor] = None,
        labels: Optional[torch.LongTensor] = None,
-        position_ids: Optional[torch.LongTensor] = None,
        **kwargs,
    ) -> CausalLMOutputWithPast:
-        cu_seqlens: Optional[torch.LongTensor] = None
-        max_seqlen: Optional[int] = None
-        if position_ids is not None:
-            batch_size, seq_length = input_ids.shape
-            position_ids = position_ids.view(-1, seq_length).long()
-            cu_seqlens, max_seqlen = get_cu_seqlens_from_pos_ids(position_ids)
-            cu_seqlens = cu_seqlens.squeeze()
-
        hidden_states = self.transformer(
-            input_ids,
-            past_key_values=past_key_values,
-            attention_mask=attention_mask,
-            cu_seqlens=cu_seqlens,
-            max_seqlen=max_seqlen,
+            input_ids, past_key_values=past_key_values, attention_mask=attention_mask
        )
        lm_logits = self.lm_head(hidden_states)

--- a/src/axolotl/monkeypatch/fastchat_conversation_turns.py
+++ b/src/axolotl/monkeypatch/fastchat_conversation_turns.py
@@ -147,6 +147,15 @@ def get_turns(  # pylint: disable=too-many-return-statements
            else:
                yield role + "\n", ""
        return
+    if self.sep_style == SeparatorStyle.CHATGLM3:
+        if self.system_message:
+            yield "", system_prompt
+        for role, message in self.messages:
+            if message:
+                yield role + "\n", " " + message
+            else:
+                yield role
+        return
    if self.sep_style == SeparatorStyle.CHATINTERN:
        # source: https://huggingface.co/internlm/internlm-chat-7b-8k/blob/bd546fa984b4b0b86958f56bf37f94aa75ab8831/modeling_internlm.py#L771
        seps = [self.sep, self.sep2]
--- a/src/axolotl/monkeypatch/mixtral/init.py
+++ b/src/axolotl/monkeypatch/mixtral/init.py
@@ -17,6 +17,6 @@ def replace_mixtral_attn_with_multipack_flash_attn():
    transformers.models.mixtral.modeling_mixtral.MixtralModel.forward = (
        mixtral_model_forward
    )
-    transformers.models.mixtral.modeling_mixtral.MISTRAL_ATTENTION_CLASSES[
+    transformers.models.mixtral.modeling_mixtral.MIXTRAL_ATTENTION_CLASSES[
        "flash_attention_2"
    ] = MixtralMultipackFlashAttention2
--- a/src/axolotl/monkeypatch/mixtral/modeling_mixtral.py
+++ b/src/axolotl/monkeypatch/mixtral/modeling_mixtral.py
@@ -261,7 +261,11 @@ def mixtral_model_forward(
    if inputs_embeds is None:
        inputs_embeds = self.embed_tokens(input_ids)

-    if attention_mask is not None and self._use_flash_attention_2 and use_cache:
+    if (
+        attention_mask is not None
+        and self._attn_implementation == "flash_attention_2"
+        and use_cache
+    ):
        is_padding_right = attention_mask[:, -1].sum().item() != batch_size
        if is_padding_right:
            raise ValueError(
@@ -270,7 +274,7 @@ def mixtral_model_forward(
                " call `tokenizer.padding_side  = 'left'` before tokenizing the input. "
            )

-    if self._use_flash_attention_2:
+    if self._attn_implementation == "flash_attention_2":
        # 2d mask is passed through the layers
        attention_mask = (
            attention_mask
--- a/src/axolotl/monkeypatch/utils.py
+++ b/src/axolotl/monkeypatch/utils.py
@@ -55,6 +55,7 @@ def get_cu_seqlens(attn_mask):
    return torch.stack(results).to(dtype=torch.int32), torch.stack(max_seq_lens)


+@torch.jit.script
 def get_cu_seqlens_from_pos_ids(position_ids):
    """generate a cumulative sequence length mask for flash attention using pos ids"""
    if len(position_ids.shape) == 1:
@@ -81,7 +82,7 @@ def get_cu_seqlens_from_pos_ids(position_ids):
        # Get the indices where the sequence starts
        start_indices = torch.cat(
            [
-                (seq_starts).nonzero(as_tuple=True)[0],
+                torch.nonzero(seq_starts).unbind(dim=1)[0],
                torch.tensor([len(adjusted_row)], dtype=torch.int32, device=device),
            ]
        )
--- a/src/axolotl/prompt_tokenizers.py
+++ b/src/axolotl/prompt_tokenizers.py
@@ -379,10 +379,12 @@ class ShareGPTPromptTokenizingStrategy(PromptTokenizingStrategy):
                        add_eos_token=False,
                        strip_bos_token=True,
                    )
-                    # everything from this is masked out from the labels
-                    labels = [IGNORE_TOKEN_ID] * len(res["input_ids"])
+                    if self.train_on_inputs:
+                        labels = copy.deepcopy(res["input_ids"])
+                    else:
+                        # everything from this is masked out from the labels
+                        labels = [IGNORE_TOKEN_ID] * len(res["input_ids"])
                elif assistant in role:
-                    # TODO label assistant token/tokens w/ IGNORE_TOKEN_ID
                    role = (
                        role.replace(role_remap[1]["from"], role_remap[1]["to"])
                        if role_remap
@@ -392,9 +394,13 @@ class ShareGPTPromptTokenizingStrategy(PromptTokenizingStrategy):
                    # this should be the assistant response, should end with an eos token
                    if not content.strip():
                        LOG.warning(f"assistant turn has empty text: {prompt}")
+                    add_eos_token = not (
+                        conversation.name == "chatml"
+                        and conversation.sep == self.tokenizer.eos_token
+                    )
                    res = self._tokenize(
                        turn,
-                        add_eos_token=True,
+                        add_eos_token=add_eos_token,
                        strip_bos_token=True,
                    )
                    role_res = self._tokenize(
@@ -402,18 +408,24 @@ class ShareGPTPromptTokenizingStrategy(PromptTokenizingStrategy):
                        add_eos_token=False,
                        strip_bos_token=True,
                    )
-                    # not masked out from labels
                    labels = copy.deepcopy(res["input_ids"])
-                    len_role = len(role_res["input_ids"])
-                    labels[:len_role] = [IGNORE_TOKEN_ID] * min(len_role, len(labels))
+                    if not self.train_on_inputs:
+                        # mask out role tokens from the labels
+                        len_role = len(role_res["input_ids"])
+                        labels[:len_role] = [IGNORE_TOKEN_ID] * min(
+                            len_role, len(labels)
+                        )
                elif role == "":
                    turn = content
                    # this is only ever the first part, should include the bos token and the user query
                    res = self._tokenize(
                        turn, add_eos_token=False, strip_bos_token=False
                    )
-                    # everything from this is masked out from the labels
-                    labels = [IGNORE_TOKEN_ID] * len(res["input_ids"])
+                    if self.train_on_inputs:
+                        labels = copy.deepcopy(res["input_ids"])
+                    else:
+                        # everything from this is masked out from the labels
+                        labels = [IGNORE_TOKEN_ID] * len(res["input_ids"])
                else:
                    LOG.warning(f"unhandled role: {role}")
                    continue
--- a/src/axolotl/train.py
+++ b/src/axolotl/train.py
@@ -5,14 +5,16 @@ import signal
 import sys
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Optional
+from typing import Optional, Tuple, Union

 import torch
 import transformers.modelcard
 from accelerate.logging import get_logger
 from datasets import Dataset
 from optimum.bettertransformer import BetterTransformer
+from peft import PeftModel
 from pkg_resources import get_distribution  # type: ignore
+from transformers import PreTrainedModel, PreTrainedTokenizer
 from transformers.deepspeed import is_deepspeed_zero3_enabled

 from axolotl.common.cli import TrainerCliArgs
@@ -43,7 +45,7 @@ class TrainDatasetMeta:

 def train(
    *, cfg: DictDefault, cli_args: TrainerCliArgs, dataset_meta: TrainDatasetMeta
-):
+) -> Tuple[Union[PeftModel, PreTrainedModel], PreTrainedTokenizer]:
    # load the tokenizer first
    LOG.debug(
        f"loading tokenizer... {cfg.tokenizer_config or cfg.base_model_config}",
@@ -61,6 +63,17 @@ def train(
        msg += " and peft_config..."
    LOG.debug(msg)
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
+    model_ref = None
+    if cfg.rl:
+        if cfg.adapter and not cfg.rl_adapter_ref_model:
+            # use built-in trl autounwrap
+            LOG.debug("Passing model_ref: None to RL trainer")
+            model_ref = None  # explicit setting to None
+        else:
+            # load the model again for model_ref/baseline
+            model_ref, _ = load_model(
+                cfg, tokenizer, inference=cli_args.inference, reference_model=True
+            )

    safe_serialization = cfg.save_safetensors is True

@@ -83,7 +96,7 @@ def train(
        freeze_parameters_except(model, cfg.unfrozen_parameters)

    trainer = setup_trainer(
-        cfg, train_dataset, eval_dataset, model, tokenizer, total_num_steps
+        cfg, train_dataset, eval_dataset, (model, model_ref), tokenizer, total_num_steps
    )

    if hasattr(model, "config"):
@@ -182,6 +195,9 @@ def train(

    if not cfg.hub_model_id:
        trainer.create_model_card(model_name=cfg.output_dir.lstrip("./"))
+    elif cfg.hub_model_id:
+        # defensively push to the hub to ensure the model card is updated
+        trainer.push_to_hub()

    return model, tokenizer

--- a/src/axolotl/utils/collators.py
+++ b/src/axolotl/utils/collators.py
@@ -178,3 +178,24 @@ class MambaDataCollator:
            "input_ids": input_ids,
            "labels": labels,
        }
+
+
+@dataclass
+class PretrainingBatchSamplerDataCollatorForSeq2Seq(DataCollatorForSeq2Seq):
+    """
+    Collator for multipack specific to the using the BatchSampler
+    """
+
+    def __call__(self, features, return_tensors=None):
+        chunked_data = {}
+        for feature in features.keys():
+            if feature == "length":
+                continue
+            if feature == "attention_mask":
+                arrays = [(1) * np.array(item) for item in features[feature]]
+                chunked_data[feature] = np.concatenate(arrays)
+            else:
+                arrays = [np.array(item) for item in features[feature]]
+                chunked_data[feature] = np.concatenate(arrays)
+        features = [chunked_data]
+        return super().__call__(features, return_tensors=return_tensors)
--- a/src/axolotl/utils/config.py
+++ b/src/axolotl/utils/config.py
@@ -61,6 +61,14 @@ def normalize_config(cfg):
        cfg.device_map = {"": int(os.environ.get("LOCAL_RANK", 0))}
        cfg.batch_size = cfg.batch_size * cfg.world_size

+    if cfg.bf16 == "auto":
+        if is_torch_bf16_gpu_available():
+            LOG.debug("bf16 support detected, enabling for this configuration.")
+            cfg.bf16 = True
+        else:
+            LOG.debug("bf16 support not detected, disabling for this configuration.")
+            cfg.bf16 = False
+
    if cfg.device == "mps":
        cfg.load_in_8bit = False
        cfg.tf32 = False
@@ -150,7 +158,26 @@ def normalize_config(cfg):
    log_gpu_memory_usage(LOG, "baseline", cfg.device)


+def normalize_cfg_datasets(cfg):
+    """
+    helpers for mapping chat_template to various dataset configurations as necessary
+    """
+
+    if cfg.chat_template and cfg.chat_template == "chatml":
+        if cfg.datasets:
+            for idx, ds_cfg in enumerate(cfg.datasets):
+                if ds_cfg.type == "sharegpt" and not ds_cfg.conversation:
+                    LOG.info(
+                        f"updating dataset {ds_cfg.path} with `conversation: chatml` to match your chat_template"
+                    )
+                    cfg.datasets[idx].conversation = "chatml"
+
+
 def validate_config(cfg):
+    """
+    This is a "pre-validation" step that handles the yaml configuration before we have any
+    information about the model architecture
+    """
    if is_torch_bf16_gpu_available():
        if not cfg.bf16 and not cfg.bfloat16:
            LOG.info("bf16 support detected, but not enabled for this configuration.")
@@ -422,11 +449,6 @@ def validate_config(cfg):
    if cfg.warmup_steps and cfg.warmup_ratio:
        raise ValueError("warmup_steps and warmup_ratio are mutually exclusive")

-    if cfg.is_qwen_derived_model and cfg.gradient_checkpointing:
-        LOG.warning(
-            "Gradient checkpointing is broken for Qwen models for transformers>=4.35.0, except main branch."
-        )
-
    if cfg.wandb_run_id and not cfg.wandb_name:
        cfg.wandb_name = cfg.wandb_run_id

@@ -448,18 +470,9 @@ def validate_config(cfg):
    if cfg.neftune_noise_alpha is not None and cfg.neftune_noise_alpha <= 0.0:
        raise ValueError("neftune_noise_alpha must be > 0.0")

-    if (
-        cfg.adapter
-        and cfg.tokens
-        and (
-            not cfg.lora_modules_to_save
-            or not all(
-                x in cfg.lora_modules_to_save for x in ["embed_tokens", "lm_head"]
-            )
-        )
-    ):
+    if cfg.max_memory is not None and cfg.gpu_memory_limit is not None:
        raise ValueError(
-            "lora_modules_to_save not properly set yet adding new tokens. Please add `embed_tokens` and `lm_head` to `lora_modules_to_save`."
+            "max_memory and gpu_memory_limit are mutually exclusive and cannot be used together."
        )

    # TODO
--- a/src/axolotl/utils/data.py
+++ b/src/axolotl/utils/data.py
@@ -2,6 +2,7 @@
 import functools
 import hashlib
 import logging
+from collections import defaultdict
 from pathlib import Path
 from typing import Dict, List, Tuple, Union

@@ -14,6 +15,7 @@ from datasets import (
    load_from_disk,
 )
 from huggingface_hub import hf_hub_download
+from torch.utils.data import RandomSampler
 from transformers import PreTrainedTokenizerBase

 from axolotl.common.const import DEFAULT_DATASET_PREPARED_PATH
@@ -39,11 +41,14 @@ from axolotl.prompters import (
    SummarizeTLDRPrompter,
    UnsupportedPrompter,
 )
+from axolotl.utils.collators import PretrainingBatchSamplerDataCollatorForSeq2Seq
 from axolotl.utils.dict import DictDefault
 from axolotl.utils.distributed import is_main_process, zero_first
+from axolotl.utils.samplers import MultipackBatchSampler, get_dataset_lengths
 from axolotl.utils.trainer import (
    calculate_total_num_steps,
    process_datasets_for_packing,
+    process_pretraining_datasets_for_packing,
 )

 LOG = logging.getLogger("axolotl")
@@ -64,9 +69,17 @@ def prepare_dataset(cfg, tokenizer):
                tokenizer, cfg, DEFAULT_DATASET_PREPARED_PATH
            )
    else:
+        path = cfg.pretraining_dataset
+        name = None
+        if isinstance(cfg.pretraining_dataset, dict):
+            path = cfg.pretraining_dataset["path"]
+            name = cfg.pretraining_dataset["name"]
+
        train_dataset = load_pretraining_dataset(
-            cfg.pretraining_dataset,
+            path,
            tokenizer,
+            cfg,
+            name=name,
            max_tokens=cfg.sequence_len,
            seed=cfg.seed or 42,
        )
@@ -806,9 +819,27 @@ def encode_pretraining(
    return ret


-def load_pretraining_dataset(path, tokenizer, max_tokens=2048, seed=42):
-    encode = functools.partial(encode_pretraining, tokenizer, max_tokens)
-    dataset = load_dataset(path, streaming=True, split="train")
+def load_pretraining_dataset(path, tokenizer, cfg, name=None, max_tokens=2048, seed=42):
+    if cfg.sample_packing:
+        collate_fn = PretrainingBatchSamplerDataCollatorForSeq2Seq(
+            tokenizer,
+            return_tensors="pt",
+            padding=True,
+            pad_to_multiple_of=max_tokens * cfg.micro_batch_size,
+        )
+        encode = functools.partial(
+            encode_packed_pretraining,
+            tokenizer,
+            collate_fn,
+            max_seq_length=max_tokens,
+            batch_size=cfg.micro_batch_size,
+        )
+        # set this to 1 so downstream data_loader doesn't try to increase the batch again
+        cfg.micro_batch_size = 1
+    else:
+        encode = functools.partial(encode_pretraining, tokenizer, max_tokens)
+
+    dataset = load_dataset(path, streaming=True, split="train", name=name)
    dataset = dataset.shuffle(seed=seed, buffer_size=10_000)
    dataset = dataset.map(
        encode,
@@ -819,3 +850,58 @@ def load_pretraining_dataset(path, tokenizer, max_tokens=2048, seed=42):
        remove_columns=dataset.features.keys(),
    )
    return dataset
+
+
+def encode_packed_pretraining(
+    tokenizer: PreTrainedTokenizerBase,
+    collate_fn,
+    examples: List[str],
+    max_seq_length: int = 2048,
+    batch_size: int = 4,
+) -> Dict[str, List]:
+    # pylint: disable=duplicate-code
+    # tokenize all the examples
+    # rows get split with stride (overlap)
+    res = tokenizer(
+        examples,
+        truncation=True,
+        max_length=max_seq_length - 1,
+        add_special_tokens=True,
+        return_overflowing_tokens=True,
+        stride=256,
+    )
+
+    input_ids = [seq + [tokenizer.eos_token_id] for seq in res["input_ids"]]
+    attention_mask = [seq + [1] for seq in res["attention_mask"]]
+
+    tokenized_examples = {
+        "input_ids": input_ids,
+        "attention_mask": attention_mask,
+    }
+
+    train_dataset = Dataset.from_dict(tokenized_examples)
+    train_dataset = process_pretraining_datasets_for_packing(
+        train_dataset, max_seq_length
+    )
+
+    sampler = MultipackBatchSampler(
+        RandomSampler(train_dataset),
+        batch_size=batch_size,
+        drop_last=True,
+        batch_max_len=batch_size * max_seq_length,
+        lengths=get_dataset_lengths(train_dataset),
+    )
+
+    chunked_data = defaultdict(list)
+
+    for data in sampler:
+        features = train_dataset[data]
+        features["labels"] = features["input_ids"].copy()
+        collated_features = collate_fn(features)
+
+        for feature in features.keys():
+            if feature == "length":
+                continue
+            chunked_data[feature].append(collated_features[feature].squeeze(0))
+
+    return chunked_data
--- a/src/axolotl/utils/lora_embeddings.py
+++ b/src/axolotl/utils/lora_embeddings.py
@@ -0,0 +1,14 @@
+"""
+helpers for lora embeddings
+"""
+
+
+def get_linear_embedding_layers(model_type):
+    """
+    returns the linear embedding layers needed for loras, dependent on the model arch
+    """
+    if model_type == "phi-msft":
+        return ["embd.wte", "lm_head.linear"]
+    if model_type == "gpt_neox":
+        return ["embed_in", "embed_out"]
+    return ["embed_tokens", "lm_head"]
--- a/src/axolotl/utils/mlflow_.py
+++ b/src/axolotl/utils/mlflow_.py
@@ -0,0 +1,18 @@
+"""Module for mlflow utilities"""
+
+import os
+
+from axolotl.utils.dict import DictDefault
+
+
+def setup_mlflow_env_vars(cfg: DictDefault):
+    for key in cfg.keys():
+        if key.startswith("mlflow_"):
+            value = cfg.get(key, "")
+
+            if value and isinstance(value, str) and len(value) > 0:
+                os.environ[key.upper()] = value
+
+    # Enable mlflow if experiment name is present
+    if cfg.mlflow_experiment_name and len(cfg.mlflow_experiment_name) > 0:
+        cfg.use_mlflow = True
--- a/src/axolotl/utils/models.py
+++ b/src/axolotl/utils/models.py
@@ -2,7 +2,7 @@
 import logging
 import math
 import os
-from typing import Optional, Tuple  # noqa: F401
+from typing import Any, Optional, Tuple, Union  # noqa: F401

 import addict
 import bitsandbytes as bnb
@@ -28,12 +28,16 @@ from axolotl.prompt_tokenizers import LLAMA_DEFAULT_EOS_TOKEN
 from axolotl.utils.bench import log_gpu_memory_usage
 from axolotl.utils.chat_templates import chat_templates
 from axolotl.utils.dict import DictDefault
+from axolotl.utils.lora_embeddings import get_linear_embedding_layers

 LOG = logging.getLogger("axolotl")


-def check_model_config(cfg: DictDefault, model_config: AutoConfig):
-    quant_config_exists = hasattr(model_config, "quantization_config")
+def check_model_config(cfg: DictDefault, model_config: Union[AutoConfig, DictDefault]):
+    quant_config_exists = (
+        hasattr(model_config, "quantization_config")
+        and model_config.quantization_config
+    )
    quant_config_method_is_gptq = (
        quant_config_exists
        and "quant_method" in model_config.quantization_config
@@ -52,9 +56,25 @@ def check_model_config(cfg: DictDefault, model_config: AutoConfig):
            "Please use the `gptq` flag to train quantized model or point to a non-quantized model."
        )

+    lora_modules_to_save = get_linear_embedding_layers(model_config.model_type)
+    if (
+        cfg.adapter
+        and cfg.tokens
+        and (
+            not cfg.lora_modules_to_save
+            or not all(x in cfg.lora_modules_to_save for x in lora_modules_to_save)
+        )
+    ):
+        lora_modules_to_save = ", ".join(map(lambda x: f"`{x}`", lora_modules_to_save))
+        raise ValueError(
+            f"`lora_modules_to_save` not properly set when adding new tokens. Please include {lora_modules_to_save} in `lora_modules_to_save`."
+        )
+

 def load_model_config(cfg):
    model_config_name = cfg.base_model_config or cfg.base_model
+    if not model_config_name and cfg.tokenizer_config:
+        model_config_name = cfg.tokenizer_config
    trust_remote_code = cfg.trust_remote_code is True

    try:
@@ -80,6 +100,7 @@ def load_model_config(cfg):


 def load_tokenizer(cfg):
+    model_config = load_model_config(cfg)
    tokenizer_kwargs = {}
    use_fast = True  # this is the default

@@ -136,22 +157,26 @@ def load_tokenizer(cfg):
                setattr(tokenizer, attr_name, "<|endoftext|>")

    if cfg.special_tokens:
+        lora_modules_to_save = get_linear_embedding_layers(model_config.model_type)
        for k, val in cfg.special_tokens.items():
            # check if new special token is not already in tokenizer and
            # is adapter training to make sure lora_modules_to_save is set
+            # pylint: disable=too-many-boolean-expressions
            if (
                (getattr(tokenizer, k) is None or getattr(tokenizer, k) != val)
                and cfg.adapter
                and (
                    not cfg.lora_modules_to_save
                    or not all(
-                        x in cfg.lora_modules_to_save
-                        for x in ["embed_tokens", "lm_head"]
+                        x in cfg.lora_modules_to_save for x in lora_modules_to_save
                    )
                )
            ):
+                lora_modules_to_save = ", ".join(
+                    [f"`{x}`" for x in lora_modules_to_save]
+                )
                raise ValueError(
-                    "Please set lora_modules_to_save to ['embed_tokens', 'lm_head'] when using an adapter and changing the special tokens."
+                    f"Please set lora_modules_to_save to {lora_modules_to_save} when using an adapter and changing the special tokens."
                )

            tokenizer.add_special_tokens(
@@ -200,6 +225,7 @@ def load_model(
    cfg: DictDefault,
    tokenizer: PreTrainedTokenizerBase,
    inference: bool = False,
+    reference_model: bool = False,
 ) -> Tuple[PreTrainedModel, Optional[PeftConfig]]:
    """
    Load a model for a given configuration and tokenizer.
@@ -287,9 +313,47 @@ def load_model(

    model_kwargs = {}

-    model_kwargs["device_map"] = cfg.device_map
-    model_kwargs["max_memory"] = cfg.max_memory
+    max_memory = cfg.max_memory
+    device_map = cfg.device_map
+
+    if cfg.gpu_memory_limit:
+        gpu_memory_limit = (
+            str(cfg.gpu_memory_limit) + "GiB"
+            if isinstance(cfg.gpu_memory_limit, int)
+            else cfg.gpu_memory_limit
+        )
+
+        max_memory = {}
+        for i in range(torch.cuda.device_count()):
+            max_memory[i] = gpu_memory_limit
+        max_memory["cpu"] = "256GiB"  # something sufficiently large to fit anything
+
+    if max_memory is not None:
+        # Based on https://github.com/togethercomputer/OpenChatKit/blob/main/inference/bot.py
+        from accelerate import infer_auto_device_map, init_empty_weights
+
+        with init_empty_weights():
+            model_canvas = AutoModelForCausalLM.from_config(model_config)
+        model_canvas.tie_weights()
+        device_map = infer_auto_device_map(
+            model_canvas,
+            max_memory=max_memory,
+            dtype=cfg.torch_dtype,
+        )
+        # We can discard max_memory now as we have a device map set up for us
+        max_memory = None
+
+    model_kwargs["device_map"] = device_map
    model_kwargs["torch_dtype"] = cfg.torch_dtype
+    # TODO can we put the reference model on it's own gpu? I think we have to move logits around to calculate loss
+    # if cfg.rl:
+    #     if torch.cuda.device_count() > 1:
+    #         if reference_model:
+    #             model_kwargs["device_map"] = "cuda:" + str(
+    #                 torch.cuda.current_device() + 1
+    #             )
+    #         else:
+    #             model_kwargs["device_map"] = "cuda:" + str(torch.cuda.current_device())

    if is_deepspeed_zero3_enabled():
        del model_kwargs["device_map"]
@@ -332,18 +396,25 @@ def load_model(
                or cfg.is_mistral_derived_model
                or model_config.model_type == "mixtral"
            ):
+                model_kwargs["attn_implementation"] = "flash_attention_2"
                model_config._attn_implementation = (  # pylint: disable=protected-access
                    "flash_attention_2"
                )
        else:
            if model_config.model_type == "mixtral":
+                model_kwargs["attn_implementation"] = "flash_attention_2"
                model_config._attn_implementation = (  # pylint: disable=protected-access
                    "flash_attention_2"
                )
            else:
+                model_kwargs["attn_implementation"] = "eager"
                model_config._attn_implementation = (  # pylint: disable=protected-access
                    "eager"
                )
+        if model_config.model_type == "phi-msft":
+            model_config.flash_attn = True
+            model_config.flash_rotary = True
+            model_config.fused_dense = True

    try:
        if cfg.is_llama_derived_model and not cfg.trust_remote_code and not cfg.gptq:
@@ -396,11 +467,12 @@ def load_model(
        #         device=cfg.device,
        #     )
        #     model.train() # sets to train instead of eval mode
-        elif model_type == "PhiForCausalLM":
+        elif model_type == "PhiForCausalLM" or model_config.model_type == "phi-msft":
            from axolotl.models.phi import PhiForCausalLM

            model = PhiForCausalLM.from_pretrained(
                base_model,
+                config=model_config,
                load_in_8bit=cfg.load_in_8bit and cfg.adapter is not None,
                load_in_4bit=cfg.load_in_4bit and cfg.adapter is not None,
                **model_kwargs,
@@ -413,7 +485,6 @@ def load_model(
            model_kwargs["device"] = torch.cuda.current_device()
            del model_kwargs["torch_dtype"]
            del model_kwargs["device_map"]
-            del model_kwargs["max_memory"]

            model = MambaLMHeadModel.from_pretrained(
                base_model,
@@ -517,13 +588,14 @@ def load_model(
        log_gpu_memory_usage(LOG, "after model load", model.device)

    # make sure these are fp32 per Ramesh et al. (2021)
+    embedding_modules = get_linear_embedding_layers(cfg.model_config_type)
    for name, module in model.named_modules():
-        if "norm" in name:
+        if any(m in name for m in ["norm", "gate"]):
            module.to(torch.float32)
        if model_config.model_type == "btlm":
            # don't upcast lm_head for btlm
            continue
-        if "lm_head" in name or "embed_tokens" in name:
+        if any(m in name for m in embedding_modules):
            if hasattr(module, "weight"):
                module.to(torch.float32)

@@ -548,18 +620,20 @@ def load_model(

    # LlamaRMSNorm layers are in fp32 after kbit_training or full finetune, so we need to
    # convert them back to fp16/bf16 for flash-attn compatibility.
-    if needs_fa2_dtype or (cfg.flash_attention and cfg.is_llama_derived_model):
+    if needs_fa2_dtype or cfg.flash_attention:
        LOG.info("converting modules to %s for flash attention", cfg.torch_dtype)
        for name, module in model.named_modules():
            if "norm" in name:
                module.to(cfg.torch_dtype)
-            if "lm_head" in name or "embed_tokens" in name:
+            if any(m in name for m in embedding_modules):
                if hasattr(module, "weight"):
                    module.to(cfg.torch_dtype)

-    model, lora_config = load_adapter(model, cfg, cfg.adapter)
+    lora_config = None
+    if not reference_model or cfg.lora_model_dir:
+        model, lora_config = load_adapter(model, cfg, cfg.adapter)

-    if cfg.ddp and not load_in_8bit:
+    if cfg.ddp and not load_in_8bit and not (cfg.rl and cfg.load_in_4bit):
        model.to(f"cuda:{cfg.local_rank}")

    if torch.cuda.device_count() > 1 and int(os.getenv("WORLD_SIZE", "1")) == 1:
@@ -668,10 +742,15 @@ def load_lora(model, cfg, inference=False):

    if cfg.lora_model_dir:
        LOG.debug("Loading pretained PEFT - LoRA")
+        model_kwargs: Any = {}
+        if cfg.lora_on_cpu:
+            model_kwargs["max_memory"] = {"cpu": "256GiB"}
+            model_kwargs["device_map"] = {"": "cpu"}
        model = PeftModel.from_pretrained(
            model,
            cfg.lora_model_dir,
            is_trainable=(not inference),
+            **model_kwargs,
        )
    else:
        model = get_peft_model(model, lora_config)
--- a/src/axolotl/utils/samplers/init.py
+++ b/src/axolotl/utils/samplers/init.py
@@ -2,3 +2,4 @@
 axolotl samplers module
 """
 from .multipack import MultipackBatchSampler  # noqa: F401
+from .utils import get_dataset_lengths  # noqa: F401
--- a/src/axolotl/utils/samplers/utils.py
+++ b/src/axolotl/utils/samplers/utils.py
@@ -0,0 +1,17 @@
+"""
+helper util to calculate dataset lengths
+"""
+import numpy as np
+
+
+def get_dataset_lengths(dataset):
+    if "length" in dataset.data.column_names:
+        lengths = np.array(dataset.data.column("length"))
+    else:
+        lengths = (
+            dataset.data.column("position_ids")
+            .to_pandas()
+            .apply(lambda x: x[-1] + 1)
+            .values
+        )
+    return lengths
--- a/src/axolotl/utils/schedulers.py
+++ b/src/axolotl/utils/schedulers.py
@@ -100,3 +100,43 @@ def get_cosine_schedule_with_quadratic_warmup(
        num_cycles=num_cycles,
    )
    return LambdaLR(optimizer, lr_lambda, last_epoch)
+
+
+def _get_cosine_schedule_with_min_lr_lambda(
+    current_step: int,
+    *,
+    num_warmup_steps: int,
+    num_training_steps: int,
+    min_lr_ratio: float
+):
+    # Warm up
+    if current_step < num_warmup_steps:
+        return float(current_step) / float(max(1, num_warmup_steps))
+
+    # Cosine learning rate decay
+    progress = float(current_step - num_warmup_steps) / float(
+        max(1, num_training_steps - num_warmup_steps)
+    )
+    scaling = 0.5 * (1.0 + math.cos(math.pi * progress))
+    return (1 - min_lr_ratio) * scaling + min_lr_ratio
+
+
+def get_cosine_schedule_with_min_lr(
+    optimizer: Optimizer,
+    num_warmup_steps: int,
+    num_training_steps: int,
+    min_lr_ratio: float = 0.0,
+):
+    """
+    Create a learning rate schedule which has:
+        - linear warmup from 0 -> `max_lr` over `num_warmup_steps`
+        - cosine learning rate annealing from `max_lr` -> `min_lr` over `num_training_steps`
+    """
+
+    lr_lambda = partial(
+        _get_cosine_schedule_with_min_lr_lambda,
+        num_warmup_steps=num_warmup_steps,
+        num_training_steps=num_training_steps,
+        min_lr_ratio=min_lr_ratio,
+    )
+    return LambdaLR(optimizer, lr_lambda)
--- a/src/axolotl/utils/trainer.py
+++ b/src/axolotl/utils/trainer.py
@@ -12,9 +12,9 @@ from accelerate.logging import get_logger
 from datasets import set_caching_enabled
 from torch.utils.data import DataLoader, RandomSampler

-from axolotl.core.trainer_builder import HFCausalTrainerBuilder
+from axolotl.core.trainer_builder import HFCausalTrainerBuilder, HFDPOTrainerBuilder
 from axolotl.utils.distributed import is_main_process, reduce_and_broadcast, zero_first
-from axolotl.utils.samplers import MultipackBatchSampler
+from axolotl.utils.samplers import MultipackBatchSampler, get_dataset_lengths

 LOG = get_logger("axolotl")

@@ -108,13 +108,7 @@ def disable_datasets_caching():

 def process_datasets_for_packing(cfg, train_dataset, eval_dataset, tokenizer):
    drop_long = partial(drop_long_seq, sequence_len=cfg.sequence_len)
-    with zero_first(is_main_process()):
-        train_dataset = train_dataset.filter(drop_long, num_proc=cfg.dataset_processes)
-        if eval_dataset:
-            eval_dataset = eval_dataset.filter(
-                drop_long, num_proc=cfg.dataset_processes
-            )
-
+    with zero_first(is_main_process()), disable_datasets_caching():
        if cfg.group_by_length:
            train_dataset = train_dataset.map(
                add_length, num_proc=cfg.dataset_processes
@@ -130,6 +124,16 @@ def process_datasets_for_packing(cfg, train_dataset, eval_dataset, tokenizer):
                        add_position_ids, num_proc=cfg.dataset_processes
                    )

+        if cfg.group_by_length or cfg.sample_packing:
+            max_input_len = np.max(get_dataset_lengths(train_dataset))
+            LOG.debug(f"max_input_len: {max_input_len}", main_process_only=True)
+
+        train_dataset = train_dataset.filter(drop_long, num_proc=cfg.dataset_processes)
+        if eval_dataset:
+            eval_dataset = eval_dataset.filter(
+                drop_long, num_proc=cfg.dataset_processes
+            )
+
        # Phi doesn't want the attention_mask feature when training
        if (
            "CodeGenTokenizer" in tokenizer.__class__.__name__
@@ -143,6 +147,16 @@ def process_datasets_for_packing(cfg, train_dataset, eval_dataset, tokenizer):
    return train_dataset, eval_dataset


+def process_pretraining_datasets_for_packing(train_dataset, sequence_len):
+    drop_long = partial(drop_long_seq, sequence_len=sequence_len)
+
+    train_dataset = train_dataset.filter(drop_long)
+    train_dataset = train_dataset.map(
+        add_position_ids,
+    )
+    return train_dataset
+
+
 def calculate_total_num_steps(cfg, train_dataset, update=True):
    if not cfg.total_num_tokens:
        total_num_tokens = np.sum(
@@ -202,12 +216,7 @@ def calculate_total_num_steps(cfg, train_dataset, update=True):
                drop_last=True,
                batch_max_len=cfg.micro_batch_size
                * (cfg.max_packed_sequence_len or cfg.sequence_len),
-                lengths=(
-                    train_dataset.data.column("position_ids")
-                    .to_pandas()
-                    .apply(lambda x: x[-1] + 1)
-                    .values
-                ),
+                lengths=get_dataset_lengths(train_dataset),
            )

            data_loader = DataLoader(
@@ -280,7 +289,12 @@ def prepare_optim_env(cfg):


 def setup_trainer(cfg, train_dataset, eval_dataset, model, tokenizer, total_num_steps):
-    trainer_builder = HFCausalTrainerBuilder(cfg, model, tokenizer)
+    if cfg.rl:
+        trainer_builder = HFDPOTrainerBuilder(cfg, model[0], tokenizer)
+        trainer_builder.model_ref = model[1]
+    else:
+        trainer_builder = HFCausalTrainerBuilder(cfg, model[0], tokenizer)
+
    trainer_builder.train_dataset = train_dataset
    trainer_builder.eval_dataset = eval_dataset

--- a/tests/core/test_trainer_builder.py
+++ b/tests/core/test_trainer_builder.py
@@ -0,0 +1,60 @@
+"""
+unit tests for axolotl.core.trainer_builder
+"""
+import pytest
+
+from axolotl.core.trainer_builder import HFDPOTrainerBuilder
+from axolotl.utils.dict import DictDefault
+from axolotl.utils.models import load_model, load_tokenizer
+
+
+@pytest.fixture(name="cfg")
+def fixture_cfg():
+    return DictDefault(
+        {
+            "base_model": "TinyLlama/TinyLlama-1.1B-Chat-v0.6",
+            "model_type": "AutoModelForCausalLM",
+            "tokenizer_type": "LlamaTokenizer",
+            "micro_batch_size": 1,
+            "gradient_accumulation_steps": 1,
+            "learning_rate": 0.00005,
+            "save_steps": 100,
+            "output_dir": "./model-out",
+            "warmup_steps": 10,
+            "gradient_checkpointing": False,
+            "optimizer": "adamw_torch",
+            "sequence_len": 2048,
+            "rl": True,
+            "adam_beta1": 0.998,
+            "adam_beta2": 0.9,
+            "adam_epsilon": 0.00001,
+            "dataloader_num_workers": 1,
+            "dataloader_pin_memory": True,
+            "model_config_type": "llama",
+        }
+    )
+
+
+@pytest.fixture(name="tokenizer")
+def fixture_tokenizer(cfg):
+    return load_tokenizer(cfg)
+
+
+@pytest.fixture(name="model")
+def fixture_model(cfg, tokenizer):
+    return load_model(cfg, tokenizer)
+
+
+class TestHFDPOTrainerBuilder:
+    """
+    TestCase class for DPO trainer builder
+    """
+
+    def test_build_training_arguments(self, cfg, model, tokenizer):
+        builder = HFDPOTrainerBuilder(cfg, model, tokenizer)
+        training_arguments = builder.build_training_arguments(100)
+        assert training_arguments.adam_beta1 == 0.998
+        assert training_arguments.adam_beta2 == 0.9
+        assert training_arguments.adam_epsilon == 0.00001
+        assert training_arguments.dataloader_num_workers == 1
+        assert training_arguments.dataloader_pin_memory is True
--- a/tests/e2e/patched/init.py
+++ b/tests/e2e/patched/init.py
--- a/tests/e2e/patched/test_fused_llama.py
+++ b/tests/e2e/patched/test_fused_llama.py
@@ -15,7 +15,7 @@ from axolotl.train import train
 from axolotl.utils.config import normalize_config
 from axolotl.utils.dict import DictDefault

-from .utils import with_temp_dir
+from ..utils import with_temp_dir

 LOG = logging.getLogger("axolotl.tests.e2e")
 os.environ["WANDB_DISABLED"] = "true"
--- a/tests/e2e/patched/test_lora_llama_multipack.py
+++ b/tests/e2e/patched/test_lora_llama_multipack.py
@@ -0,0 +1,126 @@
+"""
+E2E tests for lora llama
+"""
+
+import logging
+import os
+import unittest
+from pathlib import Path
+
+import pytest
+from transformers.utils import is_auto_gptq_available, is_torch_bf16_gpu_available
+
+from axolotl.cli import load_datasets
+from axolotl.common.cli import TrainerCliArgs
+from axolotl.train import train
+from axolotl.utils.config import normalize_config
+from axolotl.utils.dict import DictDefault
+
+from ..utils import with_temp_dir
+
+LOG = logging.getLogger("axolotl.tests.e2e")
+os.environ["WANDB_DISABLED"] = "true"
+
+
+class TestLoraLlama(unittest.TestCase):
+    """
+    Test case for Llama models using LoRA w multipack
+    """
+
+    @with_temp_dir
+    def test_lora_packing(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "JackFram/llama-68m",
+                "tokenizer_type": "LlamaTokenizer",
+                "sequence_len": 1024,
+                "sample_packing": True,
+                "flash_attention": True,
+                "load_in_8bit": True,
+                "adapter": "lora",
+                "lora_r": 32,
+                "lora_alpha": 64,
+                "lora_dropout": 0.05,
+                "lora_target_linear": True,
+                "val_set_size": 0.1,
+                "special_tokens": {
+                    "unk_token": "<unk>",
+                    "bos_token": "<s>",
+                    "eos_token": "</s>",
+                },
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 8,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_torch",
+                "lr_scheduler": "cosine",
+            }
+        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @pytest.mark.skipif(not is_auto_gptq_available(), reason="auto-gptq not available")
+    @with_temp_dir
+    def test_lora_gptq_packed(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "TheBlokeAI/jackfram_llama-68m-GPTQ",
+                "model_type": "AutoModelForCausalLM",
+                "tokenizer_type": "LlamaTokenizer",
+                "sequence_len": 1024,
+                "sample_packing": True,
+                "flash_attention": True,
+                "load_in_8bit": True,
+                "adapter": "lora",
+                "gptq": True,
+                "gptq_disable_exllama": True,
+                "lora_r": 32,
+                "lora_alpha": 64,
+                "lora_dropout": 0.05,
+                "lora_target_linear": True,
+                "val_set_size": 0.1,
+                "special_tokens": {
+                    "unk_token": "<unk>",
+                    "bos_token": "<s>",
+                    "eos_token": "</s>",
+                },
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "save_steps": 0.5,
+                "micro_batch_size": 8,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_torch",
+                "lr_scheduler": "cosine",
+            }
+        )
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
--- a/tests/e2e/patched/test_mistral_samplepack.py
+++ b/tests/e2e/patched/test_mistral_samplepack.py
@@ -15,7 +15,7 @@ from axolotl.train import train
 from axolotl.utils.config import normalize_config
 from axolotl.utils.dict import DictDefault

-from .utils import with_temp_dir
+from ..utils import with_temp_dir

 LOG = logging.getLogger("axolotl.tests.e2e")
 os.environ["WANDB_DISABLED"] = "true"
--- a/tests/e2e/patched/test_mixtral_samplepack.py
+++ b/tests/e2e/patched/test_mixtral_samplepack.py
@@ -0,0 +1,123 @@
+"""
+E2E tests for mixtral
+"""
+
+import logging
+import os
+import unittest
+from pathlib import Path
+
+from transformers.utils import is_torch_bf16_gpu_available
+
+from axolotl.cli import load_datasets
+from axolotl.common.cli import TrainerCliArgs
+from axolotl.train import train
+from axolotl.utils.config import normalize_config
+from axolotl.utils.dict import DictDefault
+
+from ..utils import with_temp_dir
+
+LOG = logging.getLogger("axolotl.tests.e2e")
+os.environ["WANDB_DISABLED"] = "true"
+
+
+class TestMixtral(unittest.TestCase):
+    """
+    Test case for Llama models using LoRA
+    """
+
+    @with_temp_dir
+    def test_qlora(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sequence_len": 2048,
+                "load_in_4bit": True,
+                "adapter": "qlora",
+                "lora_r": 16,
+                "lora_alpha": 32,
+                "lora_dropout": 0.1,
+                "lora_target_linear": True,
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+                "sample_packing": True,
+            }
+        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @with_temp_dir
+    def test_ft(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sequence_len": 2048,
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+                "sample_packing": True,
+            }
+        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        model, _ = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (
+            "axolotl.monkeypatch.mixtral.modeling_mixtral"
+            in model.model.layers[0].self_attn.__class__.__module__
+        )
+        assert (
+            "MixtralMultipackFlashAttention2"
+            in model.model.layers[0].self_attn.__class__.__name__
+        )
+        assert (Path(temp_dir) / "pytorch_model.bin").exists()
--- a/tests/e2e/patched/test_model_patches.py
+++ b/tests/e2e/patched/test_model_patches.py
@@ -0,0 +1,99 @@
+"""
+E2E smoke tests to check that the monkeypatches are in place for certain configurations
+"""
+
+import unittest
+
+from axolotl.common.cli import TrainerCliArgs
+from axolotl.utils.config import normalize_config
+from axolotl.utils.dict import DictDefault
+from axolotl.utils.models import load_model, load_tokenizer
+
+from ..utils import with_temp_dir
+
+
+class TestModelPatches(unittest.TestCase):
+    """
+    TestCases for the multipack monkey patches
+    """
+
+    @with_temp_dir
+    def test_mixtral_multipack(self, temp_dir):
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sample_packing": True,
+                "sequence_len": 2048,
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        tokenizer = load_tokenizer(cfg)
+        model, _ = load_model(cfg, tokenizer, inference=cli_args.inference)
+
+        assert (
+            "axolotl.monkeypatch.mixtral.modeling_mixtral"
+            in model.model.layers[0].self_attn.__class__.__module__
+        )
+        assert (
+            "MixtralMultipackFlashAttention2"
+            in model.model.layers[0].self_attn.__class__.__name__
+        )
+
+    @with_temp_dir
+    def test_mistral_multipack(self, temp_dir):
+        cfg = DictDefault(
+            {
+                "base_model": "openaccess-ai-collective/tiny-mistral",
+                "flash_attention": True,
+                "sample_packing": True,
+                "sequence_len": 2048,
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        tokenizer = load_tokenizer(cfg)
+        model, _ = load_model(cfg, tokenizer, inference=cli_args.inference)
+
+        assert (
+            "axolotl.monkeypatch.mistral_attn_hijack_flash"
+            in model.model.layers[0].self_attn.forward.__module__
+        )
--- a/tests/e2e/patched/test_resume.py
+++ b/tests/e2e/patched/test_resume.py
@@ -17,7 +17,7 @@ from axolotl.train import train
 from axolotl.utils.config import normalize_config
 from axolotl.utils.dict import DictDefault

-from .utils import most_recent_subdir, with_temp_dir
+from ..utils import most_recent_subdir, with_temp_dir

 LOG = logging.getLogger("axolotl.tests.e2e")
 os.environ["WANDB_DISABLED"] = "true"
@@ -29,7 +29,7 @@ class TestResumeLlama(unittest.TestCase):
    """

    @with_temp_dir
-    def test_resume_qlora(self, temp_dir):
+    def test_resume_qlora_packed(self, temp_dir):
        # pylint: disable=duplicate-code
        cfg = DictDefault(
            {
--- a/tests/e2e/test_lora_llama.py
+++ b/tests/e2e/test_lora_llama.py
@@ -65,96 +65,3 @@ class TestLoraLlama(unittest.TestCase):

        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
        assert (Path(temp_dir) / "adapter_model.bin").exists()
-
-    @with_temp_dir
-    def test_lora_packing(self, temp_dir):
-        # pylint: disable=duplicate-code
-        cfg = DictDefault(
-            {
-                "base_model": "JackFram/llama-68m",
-                "tokenizer_type": "LlamaTokenizer",
-                "sequence_len": 1024,
-                "sample_packing": True,
-                "flash_attention": True,
-                "load_in_8bit": True,
-                "adapter": "lora",
-                "lora_r": 32,
-                "lora_alpha": 64,
-                "lora_dropout": 0.05,
-                "lora_target_linear": True,
-                "val_set_size": 0.1,
-                "special_tokens": {
-                    "unk_token": "<unk>",
-                    "bos_token": "<s>",
-                    "eos_token": "</s>",
-                },
-                "datasets": [
-                    {
-                        "path": "mhenrichsen/alpaca_2k_test",
-                        "type": "alpaca",
-                    },
-                ],
-                "num_epochs": 2,
-                "micro_batch_size": 8,
-                "gradient_accumulation_steps": 1,
-                "output_dir": temp_dir,
-                "learning_rate": 0.00001,
-                "optimizer": "adamw_torch",
-                "lr_scheduler": "cosine",
-                "bf16": True,
-            }
-        )
-        normalize_config(cfg)
-        cli_args = TrainerCliArgs()
-        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
-
-        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
-        assert (Path(temp_dir) / "adapter_model.bin").exists()
-
-    @with_temp_dir
-    def test_lora_gptq(self, temp_dir):
-        # pylint: disable=duplicate-code
-        cfg = DictDefault(
-            {
-                "base_model": "TheBlokeAI/jackfram_llama-68m-GPTQ",
-                "model_type": "AutoModelForCausalLM",
-                "tokenizer_type": "LlamaTokenizer",
-                "sequence_len": 1024,
-                "sample_packing": True,
-                "flash_attention": True,
-                "load_in_8bit": True,
-                "adapter": "lora",
-                "gptq": True,
-                "gptq_disable_exllama": True,
-                "lora_r": 32,
-                "lora_alpha": 64,
-                "lora_dropout": 0.05,
-                "lora_target_linear": True,
-                "val_set_size": 0.1,
-                "special_tokens": {
-                    "unk_token": "<unk>",
-                    "bos_token": "<s>",
-                    "eos_token": "</s>",
-                },
-                "datasets": [
-                    {
-                        "path": "mhenrichsen/alpaca_2k_test",
-                        "type": "alpaca",
-                    },
-                ],
-                "num_epochs": 2,
-                "save_steps": 0.5,
-                "micro_batch_size": 8,
-                "gradient_accumulation_steps": 1,
-                "output_dir": temp_dir,
-                "learning_rate": 0.00001,
-                "optimizer": "adamw_torch",
-                "lr_scheduler": "cosine",
-            }
-        )
-        normalize_config(cfg)
-        cli_args = TrainerCliArgs()
-        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
-
-        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
-        assert (Path(temp_dir) / "adapter_model.bin").exists()
--- a/tests/e2e/test_mamba.py
+++ b/tests/e2e/test_mamba.py
@@ -7,6 +7,8 @@ import os
 import unittest
 from pathlib import Path

+import pytest
+
 from axolotl.cli import load_datasets
 from axolotl.common.cli import TrainerCliArgs
 from axolotl.train import train
@@ -19,9 +21,10 @@ LOG = logging.getLogger("axolotl.tests.e2e")
 os.environ["WANDB_DISABLED"] = "true"


-class TestMistral(unittest.TestCase):
+@pytest.mark.skip(reason="skipping until upstreamed into transformers")
+class TestMamba(unittest.TestCase):
    """
-    Test case for Llama models using LoRA
+    Test case for Mamba models
    """

    @with_temp_dir
--- a/tests/e2e/test_mixtral.py
+++ b/tests/e2e/test_mixtral.py
@@ -0,0 +1,290 @@
+"""
+E2E tests for mixtral
+"""
+
+import logging
+import os
+import unittest
+from pathlib import Path
+
+import torch
+from transformers.utils import is_torch_bf16_gpu_available
+
+from axolotl.cli import load_datasets
+from axolotl.common.cli import TrainerCliArgs
+from axolotl.train import train
+from axolotl.utils.config import normalize_config
+from axolotl.utils.dict import DictDefault
+
+from .utils import with_temp_dir
+
+LOG = logging.getLogger("axolotl.tests.e2e")
+os.environ["WANDB_DISABLED"] = "true"
+
+
+class TestMixtral(unittest.TestCase):
+    """
+    Test case for Llama models using LoRA
+    """
+
+    @with_temp_dir
+    def test_qlora_w_fa2(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sequence_len": 1024,
+                "load_in_4bit": True,
+                "adapter": "qlora",
+                "lora_r": 4,
+                "lora_alpha": 8,
+                "lora_dropout": 0.1,
+                "lora_target_modules": [
+                    "o_proj",
+                    "w3",
+                    "k_proj",
+                    "v_proj",
+                    "w1",
+                    "q_proj",
+                    "w2",
+                ],
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        model, _ = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (
+            model.base_model.model.model.layers[0].block_sparse_moe.gate.weight.dtype
+            == torch.uint8
+        )
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @with_temp_dir
+    def test_qlora_wo_fa2(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": False,
+                "sequence_len": 1024,
+                "load_in_4bit": True,
+                "adapter": "qlora",
+                "lora_r": 4,
+                "lora_alpha": 8,
+                "lora_dropout": 0.1,
+                "lora_target_modules": [
+                    "o_proj",
+                    "w3",
+                    "k_proj",
+                    "v_proj",
+                    "w1",
+                    "q_proj",
+                    "w2",
+                ],
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        model, _ = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (
+            model.base_model.model.model.layers[0].block_sparse_moe.gate.weight.dtype
+            == torch.uint8
+        )
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @with_temp_dir
+    def test_16bit_lora_w_fa2(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sequence_len": 1024,
+                "adapter": "lora",
+                "lora_r": 4,
+                "lora_alpha": 8,
+                "lora_dropout": 0.1,
+                "lora_target_modules": [
+                    "o_proj",
+                    "w3",
+                    "k_proj",
+                    "v_proj",
+                    "w1",
+                    "q_proj",
+                    "w2",
+                ],
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        model, _ = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (
+            model.base_model.model.model.layers[0].block_sparse_moe.gate.weight.dtype
+            == torch.float32
+        )
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @with_temp_dir
+    def test_16bit_lora_wo_fa2(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": False,
+                "sequence_len": 1024,
+                "adapter": "lora",
+                "lora_r": 4,
+                "lora_alpha": 8,
+                "lora_dropout": 0.1,
+                "lora_target_modules": [
+                    "o_proj",
+                    "w3",
+                    "k_proj",
+                    "v_proj",
+                    "w1",
+                    "q_proj",
+                    "w2",
+                ],
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        normalize_config(cfg)
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        model, _ = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (
+            model.base_model.model.model.layers[0].block_sparse_moe.gate.weight.dtype
+            == torch.float32
+        )
+        assert (Path(temp_dir) / "adapter_model.bin").exists()
+
+    @with_temp_dir
+    def test_ft(self, temp_dir):
+        # pylint: disable=duplicate-code
+        cfg = DictDefault(
+            {
+                "base_model": "hf-internal-testing/Mixtral-tiny",
+                "tokenizer_config": "mistralai/Mixtral-8x7B-v0.1",
+                "flash_attention": True,
+                "sequence_len": 1024,
+                "val_set_size": 0.1,
+                "special_tokens": {},
+                "datasets": [
+                    {
+                        "path": "mhenrichsen/alpaca_2k_test",
+                        "type": "alpaca",
+                    },
+                ],
+                "num_epochs": 2,
+                "micro_batch_size": 2,
+                "gradient_accumulation_steps": 1,
+                "output_dir": temp_dir,
+                "learning_rate": 0.00001,
+                "optimizer": "adamw_bnb_8bit",
+                "lr_scheduler": "cosine",
+                "max_steps": 20,
+                "save_steps": 10,
+                "eval_steps": 10,
+            }
+        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+        normalize_config(cfg)
+        cli_args = TrainerCliArgs()
+        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
+
+        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
+        assert (Path(temp_dir) / "pytorch_model.bin").exists()
--- a/tests/e2e/test_phi.py
+++ b/tests/e2e/test_phi.py
@@ -7,6 +7,9 @@ import os
 import unittest
 from pathlib import Path

+import pytest
+from transformers.utils import is_torch_bf16_gpu_available
+
 from axolotl.cli import load_datasets
 from axolotl.common.cli import TrainerCliArgs
 from axolotl.train import train
@@ -21,17 +24,18 @@ os.environ["WANDB_DISABLED"] = "true"

 class TestPhi(unittest.TestCase):
    """
-    Test case for Llama models using LoRA
+    Test case for Phi2 models
    """

+    @pytest.mark.skip(reason="fixme later")
    @with_temp_dir
-    def test_ft(self, temp_dir):
+    def test_phi2_ft(self, temp_dir):
        # pylint: disable=duplicate-code
        cfg = DictDefault(
            {
-                "base_model": "microsoft/phi-1_5",
+                "base_model": "microsoft/phi-2",
                "trust_remote_code": True,
-                "model_type": "PhiForCausalLM",
+                "model_type": "AutoModelForCausalLM",
                "tokenizer_type": "AutoTokenizer",
                "sequence_len": 512,
                "sample_packing": False,
@@ -39,9 +43,6 @@ class TestPhi(unittest.TestCase):
                "adapter": None,
                "val_set_size": 0.1,
                "special_tokens": {
-                    "unk_token": "<|endoftext|>",
-                    "bos_token": "<|endoftext|>",
-                    "eos_token": "<|endoftext|>",
                    "pad_token": "<|endoftext|>",
                },
                "datasets": [
@@ -57,11 +58,19 @@ class TestPhi(unittest.TestCase):
                "gradient_accumulation_steps": 1,
                "output_dir": temp_dir,
                "learning_rate": 0.00001,
-                "optimizer": "adamw_bnb_8bit",
+                "optimizer": "paged_adamw_8bit",
                "lr_scheduler": "cosine",
-                "bf16": True,
+                "flash_attention": True,
+                "max_steps": 10,
+                "save_steps": 10,
+                "eval_steps": 10,
+                "save_safetensors": True,
            }
        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
        normalize_config(cfg)
        cli_args = TrainerCliArgs()
        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
@@ -69,12 +78,13 @@ class TestPhi(unittest.TestCase):
        train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
        assert (Path(temp_dir) / "pytorch_model.bin").exists()

+    @pytest.mark.skip(reason="multipack no longer supported atm")
    @with_temp_dir
    def test_ft_packed(self, temp_dir):
        # pylint: disable=duplicate-code
        cfg = DictDefault(
            {
-                "base_model": "microsoft/phi-1_5",
+                "base_model": "microsoft/phi-2",
                "trust_remote_code": True,
                "model_type": "PhiForCausalLM",
                "tokenizer_type": "AutoTokenizer",
@@ -104,9 +114,13 @@ class TestPhi(unittest.TestCase):
                "learning_rate": 0.00001,
                "optimizer": "adamw_bnb_8bit",
                "lr_scheduler": "cosine",
-                "bf16": True,
            }
        )
+        if is_torch_bf16_gpu_available():
+            cfg.bf16 = True
+        else:
+            cfg.fp16 = True
+
        normalize_config(cfg)
        cli_args = TrainerCliArgs()
        dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
--- a/tests/prompt_strategies/test_sharegpt.py
+++ b/tests/prompt_strategies/test_sharegpt.py
@@ -0,0 +1,153 @@
+"""
+Test module for sharegpt integration w chatml
+"""
+import pytest
+from datasets import Dataset
+from tokenizers import AddedToken
+from transformers import AutoTokenizer
+
+from axolotl.datasets import TokenizedPromptDataset
+from axolotl.prompt_strategies.sharegpt import SimpleShareGPTPromptTokenizingStrategy
+from axolotl.prompters import ShareGPTPrompterV2
+
+
+@pytest.fixture(name="sharegpt_dataset")
+def fixture_sharegpt_dataset():
+    return Dataset.from_list(
+        [
+            {
+                "conversations": [
+                    {
+                        "from": "system",
+                        "value": "repeat",
+                    },
+                    {
+                        "from": "human",
+                        "value": "hello",
+                    },
+                    {
+                        "from": "gpt",
+                        "value": "hello",
+                    },
+                    {
+                        "from": "human",
+                        "value": "goodbye",
+                    },
+                    {
+                        "from": "gpt",
+                        "value": "goodbye",
+                    },
+                ]
+            }
+        ]
+    )
+
+
+@pytest.fixture(name="tokenizer")
+def fixture_tokenizer():
+    tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
+    tokenizer.add_special_tokens(
+        {
+            "eos_token": AddedToken(
+                "<|im_end|>", rstrip=False, lstrip=False, normalized=False
+            )
+        }
+    )
+    tokenizer.add_tokens(
+        [
+            AddedToken("<|im_start|>", rstrip=False, lstrip=False, normalized=False),
+        ]
+    )
+
+    return tokenizer
+
+
+class TestSharegpt:
+    """
+    Test class for sharegpt prompter
+    """
+
+    def test_no_double_im_end(self, sharegpt_dataset, tokenizer):
+        strategy = SimpleShareGPTPromptTokenizingStrategy(
+            ShareGPTPrompterV2(
+                conversation="chatml",
+                role_key_model=None,
+                role_key_human=None,
+            ),
+            tokenizer,
+            False,  # train_on_inputs
+            2048,  # sequence_len
+        )
+
+        dataset_wrapper = TokenizedPromptDataset(
+            strategy, sharegpt_dataset, process_count=1
+        )
+
+        input_ids = dataset_wrapper[0]["input_ids"]
+        # fmt: off
+        assert input_ids == [
+            #  28705, 13, is " \n"
+            1,   # bos
+            32001, 1587, 13, 25997, 32000, 28705, 13,  # system
+            32001, 2188, 13, 21558, 32000, 28705, 13,  # human
+            32001, 13892, 13, 21558, 32000, 28705, 13,  # gpt
+            32001, 2188, 13, 12684, 17664, 32000, 28705, 13,   # human
+            32001, 13892, 13, 12684, 17664, 32000, 28705, 13,  # gpt
+        ]
+        # fmt: on
+
+    def test_w_train_on_input(self, sharegpt_dataset, tokenizer):
+        strategy = SimpleShareGPTPromptTokenizingStrategy(
+            ShareGPTPrompterV2(
+                conversation="chatml",
+                role_key_model=None,
+                role_key_human=None,
+            ),
+            tokenizer,
+            False,  # train_on_inputs
+            2048,  # sequence_len
+        )
+
+        dataset_wrapper = TokenizedPromptDataset(
+            strategy, sharegpt_dataset, process_count=1
+        )
+
+        labels = dataset_wrapper[0]["labels"]
+        # fmt: off
+        assert labels == [
+            -100,   # bos
+            -100, -100, -100, -100, -100, -100, -100,  # system
+            -100, -100, -100, -100, -100, -100, -100,  # human
+            -100, -100, 13, 21558, 32000, 28705, 13,  # gpt
+            -100, -100, -100, -100, -100, -100, -100, -100,   # human
+            -100, -100, 13, 12684, 17664, 32000, 28705, 13,  # gpt
+        ]
+        # fmt: on
+
+    def test_no_train_on_input(self, sharegpt_dataset, tokenizer):
+        strategy = SimpleShareGPTPromptTokenizingStrategy(
+            ShareGPTPrompterV2(
+                conversation="chatml",
+                role_key_model=None,
+                role_key_human=None,
+            ),
+            tokenizer,
+            True,  # train_on_inputs
+            2048,  # sequence_len
+        )
+
+        dataset_wrapper = TokenizedPromptDataset(
+            strategy, sharegpt_dataset, process_count=1
+        )
+
+        labels = dataset_wrapper[0]["labels"]
+        # fmt: off
+        assert labels == [
+            1,   # bos
+            32001, 1587, 13, 25997, 32000, 28705, 13,  # system
+            32001, 2188, 13, 21558, 32000, 28705, 13,  # human
+            32001, 13892, 13, 21558, 32000, 28705, 13,  # gpt
+            32001, 2188, 13, 12684, 17664, 32000, 28705, 13,   # human
+            32001, 13892, 13, 12684, 17664, 32000, 28705, 13,  # gpt
+        ]
+        # fmt: on
--- a/tests/test_normalize_config.py
+++ b/tests/test_normalize_config.py
@@ -2,8 +2,9 @@
 Test classes for checking functionality of the cfg normalization
 """
 import unittest
+from unittest.mock import patch

-from axolotl.utils.config import normalize_config
+from axolotl.utils.config import normalize_cfg_datasets, normalize_config
 from axolotl.utils.dict import DictDefault


@@ -44,3 +45,46 @@ class NormalizeConfigTestCase(unittest.TestCase):
        normalize_config(cfg)

        assert cfg.base_model_config == cfg.base_model
+
+    def test_chat_template_chatml(self):
+        cfg = DictDefault(
+            {
+                "chat_template": "chatml",
+                "datasets": [
+                    {
+                        "path": "lorem/ipsum",
+                        "type": "sharegpt",
+                        "conversation": "vicuna_v1.1",
+                    },
+                    {
+                        "path": "sit/amet",
+                        "type": "sharegpt",
+                    },
+                ],
+            }
+        )
+
+        normalize_cfg_datasets(cfg)
+
+        assert cfg.datasets[0].conversation == "vicuna_v1.1"
+        assert cfg.datasets[1].conversation == "chatml"
+
+    @patch("axolotl.utils.config.is_torch_bf16_gpu_available")
+    def test_bf16_auto_setter_available(self, mock_bf16_avail):
+        cfg = self._get_base_cfg()
+        cfg.bf16 = "auto"
+        mock_bf16_avail.return_value = True
+
+        normalize_config(cfg)
+
+        self.assertTrue(cfg.bf16)
+
+    @patch("axolotl.utils.config.is_torch_bf16_gpu_available")
+    def test_bf16_auto_setter_not_available(self, mock_bf16_avail):
+        cfg = self._get_base_cfg()
+        cfg.bf16 = "auto"
+        mock_bf16_avail.return_value = False
+
+        normalize_config(cfg)
+
+        self.assertFalse(cfg.bf16)
--- a/tests/test_packed_pretraining.py
+++ b/tests/test_packed_pretraining.py
@@ -0,0 +1,82 @@
+"""Module for testing streaming dataset sequence packing"""
+import unittest
+from functools import partial
+
+import torch
+from datasets import load_dataset
+from torch.utils.data import DataLoader
+from transformers import AutoTokenizer
+
+from axolotl.utils.collators import PretrainingBatchSamplerDataCollatorForSeq2Seq
+from axolotl.utils.data import encode_packed_pretraining
+
+
+class TestPacking(unittest.TestCase):
+    """
+    Test class for packing streaming dataset sequences
+    """
+
+    def setUp(self) -> None:
+        # pylint: disable=duplicate-code
+        self.tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
+        self.tokenizer.pad_token = "</s>"
+        self.max_seq_length = 2048
+        self.batch_size = 2
+
+    def test_packing_stream_dataset(self):
+        # pylint: disable=duplicate-code
+        dataset = load_dataset(
+            "c4",
+            "en",
+            streaming=True,
+        )["train"]
+
+        collate_fn = PretrainingBatchSamplerDataCollatorForSeq2Seq(
+            self.tokenizer,
+            return_tensors="pt",
+            padding=True,
+            pad_to_multiple_of=self.max_seq_length,
+        )
+
+        encode = partial(
+            encode_packed_pretraining,
+            self.tokenizer,
+            collate_fn,
+            max_seq_length=self.max_seq_length,
+            batch_size=self.batch_size,
+        )
+
+        dataset = dataset.map(
+            encode,
+            batched=True,
+            input_columns="text",
+            remove_columns=dataset.features.keys(),
+        )
+
+        trainer_loader = DataLoader(
+            dataset,
+            batch_size=1,
+            collate_fn=None,
+            drop_last=True,
+        )
+        idx = 0
+        for data in trainer_loader:
+            if idx > 10:
+                break
+            assert data["input_ids"].shape == torch.Size(
+                [1, self.batch_size * self.max_seq_length]
+            )
+            assert data["position_ids"].shape == torch.Size(
+                [1, self.batch_size * self.max_seq_length]
+            )
+            assert data["labels"].shape == torch.Size(
+                [1, self.batch_size * self.max_seq_length]
+            )
+            assert data["attention_mask"].shape == torch.Size(
+                [1, self.batch_size * self.max_seq_length]
+            )
+            idx += 1
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/tests/test_validation.py
+++ b/tests/test_validation.py
@@ -6,15 +6,17 @@ import unittest
 from typing import Optional

 import pytest
+from transformers.utils import is_torch_bf16_gpu_available

 from axolotl.utils.config import validate_config
 from axolotl.utils.dict import DictDefault
+from axolotl.utils.models import check_model_config
 from axolotl.utils.wandb_ import setup_wandb_env_vars


-class ValidationTest(unittest.TestCase):
+class BaseValidation(unittest.TestCase):
    """
-    Test the validation module
+    Base validation module to setup the log capture
    """

    _caplog: Optional[pytest.LogCaptureFixture] = None
@@ -23,6 +25,12 @@ class ValidationTest(unittest.TestCase):
    def inject_fixtures(self, caplog):
        self._caplog = caplog

+
+class ValidationTest(BaseValidation):
+    """
+    Test the validation module
+    """
+
    def test_load_4bit_deprecate(self):
        cfg = DictDefault(
            {
@@ -354,6 +362,10 @@ class ValidationTest(unittest.TestCase):
        with pytest.raises(ValueError, match=regex_exp):
            validate_config(cfg)

+    @pytest.mark.skipif(
+        is_torch_bf16_gpu_available(),
+        reason="test should only run on gpus w/o bf16 support",
+    )
    def test_merge_lora_no_bf16_fail(self):
        """
        This is assumed to be run on a CPU machine, so bf16 is not supported.
@@ -682,16 +694,23 @@ class ValidationTest(unittest.TestCase):

        validate_config(cfg)

-    def test_add_tokens_adapter(self):
+
+class ValidationCheckModelConfig(BaseValidation):
+    """
+    Test the validation for the config when the model config is available
+    """
+
+    def test_llama_add_tokens_adapter(self):
        cfg = DictDefault(
            {"adapter": "qlora", "load_in_4bit": True, "tokens": ["<|imstart|>"]}
        )
+        model_config = DictDefault({"model_type": "llama"})

        with pytest.raises(
            ValueError,
-            match=r".*lora_modules_to_save not properly set yet adding new tokens*",
+            match=r".*`lora_modules_to_save` not properly set when adding new tokens*",
        ):
-            validate_config(cfg)
+            check_model_config(cfg, model_config)

        cfg = DictDefault(
            {
@@ -704,9 +723,9 @@ class ValidationTest(unittest.TestCase):

        with pytest.raises(
            ValueError,
-            match=r".*lora_modules_to_save not properly set yet adding new tokens*",
+            match=r".*`lora_modules_to_save` not properly set when adding new tokens*",
        ):
-            validate_config(cfg)
+            check_model_config(cfg, model_config)

        cfg = DictDefault(
            {
@@ -717,10 +736,48 @@ class ValidationTest(unittest.TestCase):
            }
        )

-        validate_config(cfg)
+        check_model_config(cfg, model_config)
+
+    def test_phi2_add_tokens_adapter(self):
+        cfg = DictDefault(
+            {"adapter": "qlora", "load_in_4bit": True, "tokens": ["<|imstart|>"]}
+        )
+        model_config = DictDefault({"model_type": "phi-msft"})
+
+        with pytest.raises(
+            ValueError,
+            match=r".*`lora_modules_to_save` not properly set when adding new tokens*",
+        ):
+            check_model_config(cfg, model_config)
+
+        cfg = DictDefault(
+            {
+                "adapter": "qlora",
+                "load_in_4bit": True,
+                "tokens": ["<|imstart|>"],
+                "lora_modules_to_save": ["embed_tokens", "lm_head"],
+            }
+        )
+
+        with pytest.raises(
+            ValueError,
+            match=r".*`lora_modules_to_save` not properly set when adding new tokens*",
+        ):
+            check_model_config(cfg, model_config)
+
+        cfg = DictDefault(
+            {
+                "adapter": "qlora",
+                "load_in_4bit": True,
+                "tokens": ["<|imstart|>"],
+                "lora_modules_to_save": ["embd.wte", "lm_head.linear"],
+            }
+        )
+
+        check_model_config(cfg, model_config)


-class ValidationWandbTest(ValidationTest):
+class ValidationWandbTest(BaseValidation):
    """
    Validation test for wandb
    """
@@ -778,6 +835,15 @@ class ValidationWandbTest(ValidationTest):
        assert os.environ.get("WANDB_LOG_MODEL", "") == "checkpoint"
        assert os.environ.get("WANDB_DISABLED", "") != "true"

+        os.environ.pop("WANDB_PROJECT", None)
+        os.environ.pop("WANDB_NAME", None)
+        os.environ.pop("WANDB_RUN_ID", None)
+        os.environ.pop("WANDB_ENTITY", None)
+        os.environ.pop("WANDB_MODE", None)
+        os.environ.pop("WANDB_WATCH", None)
+        os.environ.pop("WANDB_LOG_MODEL", None)
+        os.environ.pop("WANDB_DISABLED", None)
+
    def test_wandb_set_disabled(self):
        cfg = DictDefault({})

@@ -798,3 +864,6 @@ class ValidationWandbTest(ValidationTest):
        setup_wandb_env_vars(cfg)

        assert os.environ.get("WANDB_DISABLED", "") != "true"
+
+        os.environ.pop("WANDB_PROJECT", None)
+        os.environ.pop("WANDB_DISABLED", None)
Author	SHA1	Message	Date
Casper	eea6e8303a	Disable datasets caching when preparing dataset for packing	2024-01-15 23:48:24 +01:00
Wing Lian	c1b741d9fb	pin model_revision for phi2 (#1123 )	2024-01-14 17:31:51 -05:00
Wing Lian	0abf4d6504	update PR template so we can capture twitter or discord handles (#1121 ) [skip ci] * update PR template so we can capture twitter or discord handles [skip ci] * ensure that the PR template is in the correct place	2024-01-14 16:19:01 -05:00
Simon Hällqvist	086561326f	Enable or disable bf16 support based on availability (#1116 )	2024-01-14 12:06:56 -05:00
Casper	2202a20f60	Reverse caching PR (#1115 )	2024-01-13 10:17:40 -05:00
Casper	d66b10141e	Disable caching on `--disable_caching` in CLI (#1110 ) * Disable caching on `--disable_caching` in CLI * chore: lint --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-13 10:13:35 +01:00
Hamel Husain	304ea1b814	Update debugging.md (#1111 )	2024-01-12 21:07:31 -08:00
Wing Lian	da97285e63	keep gate in fp32 for 16 bit loras (#1105 ) * keep gate in fp32 for loras * add e2e check for lora w/o flash attention for mixtral to check gate * add checks for gate in fp32 for mixtral, add typehints to train outputs * mixtral doesn't support basic lora 🤦 add lora tests @ 16bit and fix gate layer check fix the parameter name, was using the old disco name don't lora over the gate so we can check that is in fp32 fix dtype check * ensure we're using fp16/bf16 for 16bit and qlora is always going to be in uint8	2024-01-12 14:58:21 -05:00
Hamel Husain	2dc431078c	Add link on README to Docker Debugging (#1107 ) * add docker debug * Update docs/debugging.md Co-authored-by: Wing Lian <wing.lian@gmail.com> * explain editable install * explain editable install * upload new video * add link to README * Update README.md * Update README.md * chore: lint * make sure to lint markdown too --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-12 08:51:35 -05:00
Hamel Husain	6d342b52a4	Add section for debugging with Docker (#1104 ) * add docker debug * Update docs/debugging.md Co-authored-by: Wing Lian <wing.lian@gmail.com> * explain editable install * explain editable install * upload new video --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-11 18:43:33 -08:00
Hamel Husain	b502392e82	Update README.md (#1103 ) * Update README.md * Update README.md	2024-01-11 16:41:58 -08:00
Mark Saroufim	44ba616da2	Fix broken pypi.yml (#1099 ) [skip ci]	2024-01-11 12:35:31 -05:00
NanoCode012	b432889256	feat: enable trl's autounwrap (#1060 ) * feat: test trl's autounwrap * fix: add check for adapter * feat: add config to disable autounwrap * chore: fix lint	2024-01-11 08:43:41 -05:00
Hamel Husain	54fe07a905	Fix debugging.md (#1091 )	2024-01-10 21:44:40 -08:00
Hamel Husain	7512c3ad20	Add Debugging Guide (#1089 ) * add debug guide * add background * add .gitignore * Update devtools/dev_sharegpt.yml Co-authored-by: Wing Lian <wing.lian@gmail.com> * Update docs/debugging.md Co-authored-by: Wing Lian <wing.lian@gmail.com> * simplify example axolotl config * add additional comments * add video and TOC * try jsonc for better md rendering * style video thumbnail better * fix footnote --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-10 20:49:24 -08:00
Wing Lian	78c5b1979e	add gptneox embeddings, fix phi2 inputs, also fix the casting (#1083 )	2024-01-10 22:32:43 -05:00
Wing Lian	23495a80af	misc fixes from #943 (#1086 ) [skip ci]	2024-01-10 22:31:36 -05:00
Casper	91502b98d4	Remove fused-dense-lib from requirements.txt (#1087 )	2024-01-10 21:26:41 +01:00
Wing Lian	6c19e9302a	add python 3.11 to the matrix for unit tests (#1085 ) [skip ci]	2024-01-10 13:02:01 -05:00
Wing Lian	90036ebbc6	optimize calculation of cu_seqlens from position_ids (#1084 ) [skip ci]	2024-01-10 11:54:50 -05:00
Wing Lian	9032e610b1	use tags again for test image, only run docker e2e after pre-commit checks (#1081 )	2024-01-10 09:04:56 -05:00
NanoCode012	d69ba2b0b7	fix: warn user to install mamba_ssm package (#1019 )	2024-01-10 02:50:56 -05:00
Wing Lian	9e3f0cb5a7	pin accelerate for deepspeed fix (#1080 )	2024-01-10 00:50:04 -05:00
Wing Lian	2f2582e6ed	additional logging to get maximum token length of a sequence in the dataset (#1066 ) [skip ci] * additional logging to get maximum token length of a sequence in the dataset * fix ordering to properly determine the max_len of tokens before dropping anything longer	2024-01-10 00:49:31 -05:00
Wing Lian	0ce1a6594e	update sharegpt conversations when chatml chat template is set (#1075 ) [skip ci] * update sharegpt conversations when chatml chat template is set * add info log when updating sharegpt/chatml conversation	2024-01-10 00:49:07 -05:00
NanoCode012	043c3860cd	fix: `train_on_inputs: true` ignored for sharegpt (#1045 ) [skip ci] * fix: `train_on_inputs: true` ignored for sharegpt * enable unit test for train_on_inputs for sharegpt --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 23:00:09 -05:00
Wing Lian	0f100800e3	be more robust about checking embedding modules for lora finetunes (#1074 ) [skip ci] * be more robust about checking embedding modules for lora finetunes * update dynamic error message	2024-01-09 22:58:54 -05:00
Wing Lian	ead34c516a	swap the data collator for evals if not using sample packing (#1076 ) * swap the data collator for evals if not using sample packing * drop last from dataloader to help with issues with evals	2024-01-09 22:16:24 -05:00
Wing Lian	ec02b7cc4e	Update FUNDING.yml [skip ci]	2024-01-09 22:15:27 -05:00
Wing Lian	3b4c646f87	Update FUNDING.yml with bitcoin (#1079 ) [skip ci]	2024-01-09 21:56:52 -05:00
Wing Lian	788649fe95	attempt to also run e2e tests that needs gpus (#1070 ) * attempt to also run e2e tests that needs gpus * fix stray quote * checkout specific github ref * dockerfile for tests with proper checkout ensure wandb is dissabled for docker pytests clear wandb env after testing clear wandb env after testing make sure to provide a default val for pop tryin skipping wandb validation tests explicitly disable wandb in the e2e tests explicitly report_to None to see if that fixes the docker e2e tests split gpu from non-gpu unit tests skip bf16 check in test for now build docker w/o cache since it uses branch name ref revert some changes now that caching is fixed skip bf16 check if on gpu w support * pytest skip for auto-gptq requirements * skip mamba tests for now, split multipack and non packed lora llama tests * split tests that use monkeypatches * fix relative import for prev commit * move other tests using monkeypatches to the correct run	2024-01-09 21:23:23 -05:00
Casper	9be92d1448	Separate AutoGPTQ dep to `pip install -e .[auto-gptq]` (#1077 ) * Separate AutoGPTQ dep to `pip install -e .[auto-gptq]` * Fix code review	2024-01-09 23:39:25 +01:00
Wing Lian	d7057ccd36	paired kto support (#1069 )	2024-01-09 13:30:45 -05:00
mtenenholtz	768d348f42	update peft to 0.7.0 (#1073 )	2024-01-09 12:22:14 -05:00
Johan Hansson	090c24dcb0	Add: mlflow for experiment tracking (#1059 ) [skip ci] * Update requirements.txt adding mlflow * Update __init__.py Imports for mlflow * Update README.md * Create mlflow_.py (#1) * Update README.md * fix precommits * Update README.md Update mlflow_tracking_uri * Update trainer_builder.py update trainer building * chore: lint * make ternary a bit more readable --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 09:34:09 -05:00
Wing Lian	651b7a31fc	fix double eos token for chatml (#1054 ) [skip ci] * fix double eos token for chatml * isolate fix to chatml conversation * fix add special tokens to include rstrip * add test for train_on_inputs for sharegpt * don't use rstrip for chatml	2024-01-09 09:33:38 -05:00
Ricardo Dominguez-Olmedo	04b978b428	Cosine learning rate schedule - minimum learning rate (#1062 ) * Cosine min lr * Cosine min lr - warn if using deepspeed * cosine_min_lr_ratio readme * chore: lint --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-09 09:29:56 -05:00
NanoCode012	c3e8165f26	fix: torch_dtype mistral default to fp32 (#1050 )	2024-01-09 07:48:15 -05:00
Wing Lian	7f381750d9	Update FUNDING.yml for Kofi link (#1067 )	2024-01-08 19:26:51 -05:00
Wing Lian	14964417ee	Sponsors (#1065 ) * wip sponsors section in readme * add ko-fi and contributors list	2024-01-08 18:52:00 -05:00
Ricardo Dominguez-Olmedo	81d384598e	Efficiently get the length of the tokenized docs (#1063 ) * Efficiently get the length of the tokenized docs * chore: lint --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-08 15:48:30 -05:00
Wing Lian	732851f105	Phi2 rewrite (#1058 ) * restore to current phi modeling code from phi-2 * enable gradient checkpointing * don't cast everything to float32 all the time * gradient checkpointing for phi2 ParallelBlock module too * fix enabling flash attn for phi2 * add comment about import * fix phi2 example * fix model type check for tokenizer * revert float32 -> bf16 casting changes * support fused dense flash attn * fix the repo for flash-attn * add package name for subdir pkg * fix the data collator when not using sample packing * install packaging for pytests in ci * also fix setup to not install flash attn fused dense subdir if not extras * split out the fused-dense-lib in extra requires * don't train w group_by_length for phi * update integration test to use phi2 * set max steps and save steps for phi e2e tests * try to workaround ssave issue in ci * skip phi2 e2e test for now	2024-01-08 14:04:22 -05:00
Hamel Husain	9ca358b671	Simplify Docker Unit Test CI (#1055 ) [skip ci] * Update tests-docker.yml * Update tests-docker.yml * run ci tests on ci yaml updates --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-06 08:20:33 -05:00
JinK	553c80f79a	streaming multipack for pretraining dataset (#959 ) * [Feat] streaming multipack * WIP make continued pretraining work w multipack * fix up hadrcoding, lint * fix dict check * update test for updated pretraining multipack code * fix hardcoded data collator fix for multipack pretraining * fix the collator to be the max length for multipack pretraining * don't bother with latest tag for test * cleanup docker build/test --------- Co-authored-by: jinwonkim93@github.com <jinwonkim> Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-05 22:13:21 -05:00
Hamel Husain	eb4c99431b	Update tests-docker.yml (#1052 ) [skip ci]	2024-01-05 14:26:18 -05:00
NanoCode012	cbdbf9e6e5	feat: always push checkpoint to hub if set (#1049 ) [skip ci]	2024-01-05 13:09:42 -05:00
kallewoof	bdfefaf054	feature: better device mapping for large models (#918 ) * fix: improved memory handling when model is bigger than existing VRAM * feature: add lora_on_cpu flag to do LoRA loading on CPU (RAM) For big models where the models are taking up the entire GPU VRAM, the LoRA part will fail unless it is loaded on CPU only. * doc: add README * fix: enable progress bars in do_merge_lora() * doc: mention gpu_memory_limit and lora_on_cpu in merge part of README * Update src/axolotl/utils/models.py Co-authored-by: Wing Lian <wing.lian@gmail.com> * fix: remove deletion of removed model_kwargs key * fix: validate that gpu_memory_limit and max_memory are not both set --------- Co-authored-by: Karl-Johan Alm <kalle@gmail.com> Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-01-05 22:22:21 +09:00
Hamel Husain	63fb3eb426	set default for merge (#1044 )	2024-01-04 18:14:20 -08:00
Hamel Husain	31d23504a5	fix model card upload for PEFT models (#1043 )	2024-01-04 18:13:54 -08:00
Wing Lian	f243c2186d	RL/DPO (#935 ) * ipo-dpo trainer * fix missing abstract method * chatml template, grad checkpointing kwargs support * fix steps calc for RL and add dataloader kwargs * wip to fix dpo and start ppo * more fixes * refactor to generalize map fn * fix dataset loop and handle argilla pref dataset * set training args * load reference model on seperate gpu if more than one device * no auto upload to hub for dpo, don't add lora adapters to ref model for dpo * fixes for rl training * support for ipo from yaml * set dpo training args from the config, add tests * chore: lint * set sequence_len for model in test * add RLHF docs	2024-01-04 18:22:55 -05:00
xaviviro	59b2d302c8	Added chatglm3 conversation type for training models like TinyLLama (#1036 ) * Added chatgml3 conversation type for training models like TinyLLama * Added chatgml3 conversation type for training models like TinyLLama with lint * Added chatgml3 conversation type for training models like TinyLLama with lint	2024-01-04 21:03:04 +09:00
Wing Lian	bcc78d8fa3	bump transformers and update attention class map name (#1023 ) * bump transformers and update attention class map name * also run the tests in docker * add mixtral e2e smoke test * fix base name for docker image in test * mixtral lora doesn't seem to work, at least check qlora * add testcase for mixtral w sample packing * check monkeypatch for flash attn multipack * also run the e2e tests in docker * use all gpus to run tests in docker ci * use privileged mode too for docker w gpus * rename the docker e2e actions for gh ci * set privileged mode for docker and update mixtral model self attn check * use fp16/bf16 for mixtral w fa2 * skip e2e tests on docker w gpus for now * tests to validate mistral and mixtral patches * fix rel import	2024-01-03 12:11:04 -08:00
NanoCode012	74532ddc45	chore(config): clean up old log for Qwen (#1034 )	2024-01-04 01:19:52 +09:00
NanoCode012	8ba27f3bde	fix: lint (#1037 )	2024-01-03 10:23:44 -05:00
Hamel Husain	a3e8783328	[Docs] delete unused cfg value `lora_out_dir` (#1029 ) * Update README.md * Update README.md * Update README.md Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> --------- Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>	2024-01-02 21:35:20 -08:00
NanoCode012	b31038aae9	chore(readme): update instruction to set config to load from cache (#1030 )	2024-01-03 11:56:19 +09:00
Tim Dolan	c75f916745	added tiny llama examples for lora and qlora (#1027 ) * added tiny llama examples for lora and qlora * corrected yml files and removed tiny-llama.yml from llama-2 example	2024-01-02 20:00:37 -05:00
Wing Lian	4d2e842e46	use recommended setting for use_reentrant w gradient checkpointing (#1021 ) * use recommended setting for use_reentrant w gradient checkpointing * add doc for gradient_checkpointing_kwargs	2024-01-01 22:17:27 -05:00
				`@@ -0,0 +1 @@`
				`See [docs/debugging.md](../docs/debugging.md) for guidance on how to modify these files to debug axolotl with VSCode.`
				`@@ -0,0 +1 @@`
				`This directory contains example config files that might be useful for debugging. Please see [docs/debugging.md](../docs/debugging.md) for more information.`