rename jamba example (#1846) [skip ci]

* rename jamba example

* feat: change readme

---------

Co-authored-by: Gal Cohen <galc@ai21.com>

README.md

@@ -22,39 +22,49 @@ Features:
 <td>
 
 ## Table of Contents
-- [Introduction](#axolotl)
+- [Axolotl](#axolotl)
-- [Supported Features](#axolotl-supports)
+  - [Table of Contents](#table-of-contents)
-- [Quickstart](#quickstart-)
+  - [Axolotl supports](#axolotl-supports)
-- [Environment](#environment)
+  - [Quickstart ⚡](#quickstart-)
-  - [Docker](#docker)
+    - [Usage](#usage)
-  - [Conda/Pip venv](#condapip-venv)
+  - [Advanced Setup](#advanced-setup)
-  - [Cloud GPU](#cloud-gpu) - Latitude.sh, JarvisLabs, RunPod
+    - [Environment](#environment)
-  - [Bare Metal Cloud GPU](#bare-metal-cloud-gpu)
+      - [Docker](#docker)
-  - [Windows](#windows)
+      - [Conda/Pip venv](#condapip-venv)
-  - [Mac](#mac)
+      - [Cloud GPU](#cloud-gpu)
-  - [Google Colab](#google-colab)
+      - [Bare Metal Cloud GPU](#bare-metal-cloud-gpu)
-  - [Launching on public clouds via SkyPilot](#launching-on-public-clouds-via-skypilot)
+        - [LambdaLabs](#lambdalabs)
-  - [Launching on public clouds via dstack](#launching-on-public-clouds-via-dstack)
+        - [GCP](#gcp)
-- [Dataset](#dataset)
+      - [Windows](#windows)
-- [Config](#config)
+      - [Mac](#mac)
-  - [Train](#train)
+      - [Google Colab](#google-colab)
-  - [Inference](#inference-playground)
+      - [Launching on public clouds via SkyPilot](#launching-on-public-clouds-via-skypilot)
-  - [Merge LORA to Base](#merge-lora-to-base)
+      - [Launching on public clouds via dstack](#launching-on-public-clouds-via-dstack)
-  - [Special Tokens](#special-tokens)
+    - [Dataset](#dataset)
-  - [All Config Options](#all-config-options)
+    - [Config](#config)
-- Advanced Topics
+      - [All Config Options](#all-config-options)
-  - [Multipack](./docs/multipack.qmd)<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M17 13.5v6H5v-12h6m3-3h6v6m0-6-9 9" class="icon_svg-stroke" stroke="#666" stroke-width="1.5" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg>
+    - [Train](#train)
-  - [RLHF & DPO](./docs/rlhf.qmd)<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M17 13.5v6H5v-12h6m3-3h6v6m0-6-9 9" class="icon_svg-stroke" stroke="#666" stroke-width="1.5" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg>
+      - [Preprocess dataset](#preprocess-dataset)
-  - [Dataset Pre-Processing](./docs/dataset_preprocessing.qmd)<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M17 13.5v6H5v-12h6m3-3h6v6m0-6-9 9" class="icon_svg-stroke" stroke="#666" stroke-width="1.5" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg>
+      - [Multi-GPU](#multi-gpu)
-  - [Unsloth](./docs/unsloth.qmd)<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M17 13.5v6H5v-12h6m3-3h6v6m0-6-9 9" class="icon_svg-stroke" stroke="#666" stroke-width="1.5" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg>
+        - [DeepSpeed](#deepspeed)
-- [Common Errors](#common-errors-)
+        - [FSDP](#fsdp)
-  - [Tokenization Mismatch b/w Training & Inference](#tokenization-mismatch-bw-inference--training)
+        - [FSDP + QLoRA](#fsdp--qlora)
-- [Debugging Axolotl](#debugging-axolotl)
+        - [Weights \& Biases Logging](#weights--biases-logging)
-- [Need Help?](#need-help-)
+        - [Special Tokens](#special-tokens)
-- [Badge](#badge-)
+    - [Inference Playground](#inference-playground)
-- [Community Showcase](#community-showcase)
+    - [Merge LORA to base](#merge-lora-to-base)
-- [Contributing](#contributing-)
+  - [Common Errors 🧰](#common-errors-)
-- [Sponsors](#sponsors-)
+    - [Tokenization Mismatch b/w Inference \& Training](#tokenization-mismatch-bw-inference--training)
+  - [Debugging Axolotl](#debugging-axolotl)
+  - [Need help? 🙋](#need-help-)
+  - [Badge ❤🏷️](#badge-️)
+  - [Community Showcase](#community-showcase)
+  - [Contributing 🤝](#contributing-)
+  - [Sponsors 🤝❤](#sponsors-)
+      - [💎 Diamond Sponsors - Contact directly](#-diamond-sponsors---contact-directly)
+      - [🥇 Gold Sponsors - $5000/mo](#-gold-sponsors---5000mo)
+      - [🥈 Silver Sponsors - $1000/mo](#-silver-sponsors---1000mo)
+      - [🥉 Bronze Sponsors - $500/mo](#-bronze-sponsors---500mo)
 
 </td>
 <td>
@@ -96,6 +106,7 @@ Features:
 | RWKV        | ✅         | ❓    | ❓     | ❓             | ❓                 | ❓          | ❓            |
 | Qwen        | ✅         | ✅    | ✅     | ❓             | ❓                 | ❓          | ❓            |
 | Gemma       | ✅         | ✅    | ✅     | ❓             | ❓                 | ✅          | ❓            |
+| Jamba       | ✅         | ✅    | ✅     | ❓             | ❓                 | ✅          | ❓            |
 
 ✅: supported
 ❌: not supported

examples/jamba/README.md

@@ -6,5 +6,5 @@
 - ✅ qlora w/ deepspeed Zero-3 needs at least 2x GPUs and 67GiB VRAM (wtf?)
 - ✅ qlora single-gpu, ~51GiB VRAM
 - ✅ multipack
-- ❓ FSDP
+- ✅ FSDP
 - ❓ 8-bit LoRA

examples/jamba/qlora_fsdp_large.yaml

@@ -1,4 +1,4 @@
-base_model: ai21labs/Jamba-v0.1
+base_model: ai21labs/AI21-Jamba-1.5-Large
 tokenizer_type: AutoTokenizer
 
 load_in_4bit: true
@@ -11,7 +11,7 @@ datasets:
     drop_system_message: true
 dataset_prepared_path: last_run_prepared
 val_set_size: 0.0
-output_dir: jamba-fsdp-qlora-ft
+output_dir: jamba-large-fsdp-qlora-ft
 save_safetensors: true
 adapter: qlora
 sequence_len: 2048