diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
new file mode 100644
index 000000000..a2b797fa2
--- /dev/null
+++ b/.github/workflows/docs.yml
@@ -0,0 +1,28 @@
+name: Publish Docs
+on:
+ push:
+ branches:
+ - main
+
+permissions:
+ contents: write
+ pages: write
+
+jobs:
+ build-deploy:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v4
+ - name: Set up Quarto
+ uses: quarto-dev/quarto-actions/setup@v2
+ - name: Setup Python
+ uses: actions/setup-python@v3
+ with:
+ python-version: '3.10'
+ - name: Publish to GitHub Pages (and render)
+ uses: quarto-dev/quarto-actions/publish@v2
+ with:
+ target: gh-pages
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.gitignore b/.gitignore
index 9d6a103da..589440abf 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,7 @@
configs
last_run_prepared/
.vscode
+_site/
# Byte-compiled / optimized / DLL files
__pycache__/
@@ -172,3 +173,5 @@ wandb
lora-out/*
qlora-out/*
mlruns/*
+
+/.quarto/
diff --git a/README.md b/README.md
index eb84a0246..9b5d4cc3f 100644
--- a/README.md
+++ b/README.md
@@ -149,7 +149,7 @@ accelerate launch -m axolotl.cli.train https://raw.githubusercontent.com/OpenAcc
```
>[!Tip]
-> If you want to debug axolotl or prefer to use Docker as your development environment, see the [debugging guide's section on Docker](docs/debugging.md#debugging-with-docker).
+> If you want to debug axolotl or prefer to use Docker as your development environment, see the [debugging guide's section on Docker](docs/debugging.qmd#debugging-with-docker).
@@ -267,7 +267,7 @@ Use the below instead of the install method in QuickStart.
```
pip3 install -e '.'
```
-More info: [mac.md](/docs/mac.md)
+More info: [mac.md](/docs/mac.qmd)
#### Launching on public clouds via SkyPilot
To launch on GPU instances (both on-demand and spot instances) on 7+ clouds (GCP, AWS, Azure, OCI, and more), you can use [SkyPilot](https://skypilot.readthedocs.io/en/latest/index.html):
@@ -409,7 +409,7 @@ pretraining_dataset: # hf path only
{"segments": [{"label": true|false, "text": "..."}]}
```
-This is a special format that allows you to construct prompts without using templates. This is for advanced users who want more freedom with prompt construction. See [these docs](docs/input_output.md) for more details.
+This is a special format that allows you to construct prompts without using templates. This is for advanced users who want more freedom with prompt construction. See [these docs](docs/input_output.qmd) for more details.
##### Conversation
@@ -1125,7 +1125,7 @@ fsdp_config:
##### FSDP + QLoRA
-Axolotl supports training with FSDP and QLoRA, see [these docs](docs/fsdp_qlora.md) for more information.
+Axolotl supports training with FSDP and QLoRA, see [these docs](docs/fsdp_qlora.qmd) for more information.
##### Weights & Biases Logging
@@ -1204,7 +1204,7 @@ although this will be very slow, and using the config options above are recommen
## Common Errors 🧰
-See also the [FAQ's](./docs/faq.md) and [debugging guide](docs/debugging.md).
+See also the [FAQ's](./docs/faq.qmd) and [debugging guide](docs/debugging.qmd).
> If you encounter a 'Cuda out of memory' error, it means your GPU ran out of memory during the training process. Here's how to resolve it:
@@ -1238,7 +1238,7 @@ It's safe to ignore it.
> NCCL Timeouts during training
-See the [NCCL](docs/nccl.md) guide.
+See the [NCCL](docs/nccl.qmd) guide.
### Tokenization Mismatch b/w Inference & Training
@@ -1256,7 +1256,7 @@ Having misalignment between your prompts during training and inference can cause
## Debugging Axolotl
-See [this debugging guide](docs/debugging.md) for tips on debugging Axolotl, along with an example configuration for debugging with VSCode.
+See [this debugging guide](docs/debugging.qmd) for tips on debugging Axolotl, along with an example configuration for debugging with VSCode.
## Need help? 🙋
diff --git a/_quarto.yml b/_quarto.yml
new file mode 100644
index 000000000..31aa90398
--- /dev/null
+++ b/_quarto.yml
@@ -0,0 +1,51 @@
+project:
+ type: website
+
+website:
+ title: "Axolotl"
+ description: "Fine-tuning"
+ favicon: favicon.jpg
+ navbar:
+ title: Axolotl
+ background: dark
+ pinned: false
+ collapse: false
+ tools:
+ - icon: twitter
+ href: https://twitter.com/axolotl_ai
+ - icon: github
+ href: https://github.com/OpenAccess-AI-Collective/axolotl/
+ - icon: discord
+ href: https://discord.gg/7m9sfhzaf3
+
+ sidebar:
+ pinned: true
+ collapse-level: 2
+ style: docked
+ contents:
+ - text: Home
+ href: index.qmd
+ - section: "How-To Guides"
+ contents:
+ # TODO Edit folder structure after we have more docs.
+ - docs/debugging.qmd
+ - docs/multipack.qmd
+ - docs/fdsp_qlora.qmd
+ - docs/input_output.qmd
+ - docs/rlhf.qmd
+ - docs/nccl.qmd
+ - docs/mac.qmd
+ - docs/multi-node.qmd
+ - section: "Reference"
+ contents:
+ - docs/config.qmd
+ - docs/faq.qmd
+
+
+
+
+format:
+ html:
+ theme: materia
+ css: styles.css
+ toc: true
diff --git a/devtools/README.md b/devtools/README.md
index 1d727ed8b..0114ee3a8 100644
--- a/devtools/README.md
+++ b/devtools/README.md
@@ -1 +1 @@
-This directory contains example config files that might be useful for debugging. Please see [docs/debugging.md](../docs/debugging.md) for more information.
+This directory contains example config files that might be useful for debugging. Please see [docs/debugging.qmd](../docs/debugging.qmd) for more information.
diff --git a/docs/.gitignore b/docs/.gitignore
new file mode 100644
index 000000000..4c23a061f
--- /dev/null
+++ b/docs/.gitignore
@@ -0,0 +1,2 @@
+/.quarto/
+_site/
diff --git a/docs/config.qmd b/docs/config.qmd
new file mode 100644
index 000000000..d93b170e7
--- /dev/null
+++ b/docs/config.qmd
@@ -0,0 +1,17 @@
+---
+title: Config options
+description: A complete list of all configuration options.
+---
+
+```{python}
+#|echo: false
+#|output: asis
+import re
+# Regex pattern to match the YAML block including its code fence
+pattern = r']*id="all-yaml-options"[^>]*>.*?All yaml options.*?```yaml(.*?)```.*?'
+
+with open('../README.md', 'r') as f:
+ doc = f.read()
+match = re.search(pattern, doc, re.DOTALL)
+print("```yaml", match.group(1).strip(), "```", sep="\n")
+```
diff --git a/docs/debugging.md b/docs/debugging.qmd
similarity index 99%
rename from docs/debugging.md
rename to docs/debugging.qmd
index 59df0b785..7237fbd6f 100644
--- a/docs/debugging.md
+++ b/docs/debugging.qmd
@@ -1,4 +1,8 @@
-# Debugging Axolotl
+---
+title: Debugging
+description: How to debug Axolotl
+---
+
This document provides some tips and tricks for debugging Axolotl. It also provides an example configuration for debugging with VSCode. A good debugging setup is essential to understanding how Axolotl code works behind the scenes.
diff --git a/docs/faq.md b/docs/faq.md
deleted file mode 100644
index 654230653..000000000
--- a/docs/faq.md
+++ /dev/null
@@ -1,18 +0,0 @@
-# Axolotl FAQ's
-
-
-> The trainer stopped and hasn't progressed in several minutes.
-
-Usually an issue with the GPU's communicating with each other. See the [NCCL doc](../docs/nccl.md)
-
-> Exitcode -9
-
-This usually happens when you run out of system RAM.
-
-> Exitcode -7 while using deepspeed
-
-Try upgrading deepspeed w: `pip install -U deepspeed`
-
-> AttributeError: 'DummyOptim' object has no attribute 'step'
-
-You may be using deepspeed with single gpu. Please don't set `deepspeed:` in yaml or cli.
diff --git a/docs/faq.qmd b/docs/faq.qmd
new file mode 100644
index 000000000..91413d24e
--- /dev/null
+++ b/docs/faq.qmd
@@ -0,0 +1,21 @@
+---
+title: FAQ
+description: Frequently asked questions
+---
+
+
+**Q: The trainer stopped and hasn't progressed in several minutes.**
+
+> A: Usually an issue with the GPUs communicating with each other. See the [NCCL doc](nccl.qmd)
+
+**Q: Exitcode -9**
+
+> A: This usually happens when you run out of system RAM.
+
+**Q: Exitcode -7 while using deepspeed**
+
+> A: Try upgrading deepspeed w: `pip install -U deepspeed`
+
+**Q: AttributeError: 'DummyOptim' object has no attribute 'step'**
+
+> A: You may be using deepspeed with single gpu. Please don't set `deepspeed:` in yaml or cli.
diff --git a/docs/fsdp_qlora.md b/docs/fsdp_qlora.qmd
similarity index 92%
rename from docs/fsdp_qlora.md
rename to docs/fsdp_qlora.qmd
index 14b2c1a57..69b4ad445 100644
--- a/docs/fsdp_qlora.md
+++ b/docs/fsdp_qlora.qmd
@@ -1,4 +1,10 @@
-# FDSP + QLoRA
+---
+title: FDSP + QLoRA
+description: Use FSDP with QLoRA to fine-tune large LLMs on consumer GPUs.
+format:
+ html:
+ toc: true
+---
## Background
diff --git a/docs/input_output.md b/docs/input_output.qmd
similarity index 98%
rename from docs/input_output.md
rename to docs/input_output.qmd
index dbc6979c6..4e2ea1345 100644
--- a/docs/input_output.md
+++ b/docs/input_output.qmd
@@ -1,4 +1,7 @@
-# Template-free prompt construction with the `input_output` format
+---
+title: Template-free prompt construction
+description: "Template-free prompt construction with the `input_output` format"
+---
diff --git a/docs/mac.md b/docs/mac.qmd
similarity index 89%
rename from docs/mac.md
rename to docs/mac.qmd
index 59eacce6d..2a8303538 100644
--- a/docs/mac.md
+++ b/docs/mac.qmd
@@ -1,8 +1,12 @@
-# Mac M series support
+---
+title: Mac M-series
+description: Mac M-series support
+---
Currently Axolotl on Mac is partially usable, many of the dependencies of Axolotl including Pytorch do not support MPS or have incomplete support.
Current support:
+
- [x] Support for all models
- [x] Full training of models
- [x] LoRA training
diff --git a/docs/multi-node.md b/docs/multi-node.qmd
similarity index 95%
rename from docs/multi-node.md
rename to docs/multi-node.qmd
index 680615969..5c6fa976b 100644
--- a/docs/multi-node.md
+++ b/docs/multi-node.qmd
@@ -1,4 +1,7 @@
-# Multi Node
+---
+title: Multi Node
+description: How to use Axolotl on multiple machines
+---
You will need to create a configuration for accelerate, either by using `accelerate config` and follow the instructions or you can use one of the preset below:
diff --git a/docs/multipack.md b/docs/multipack.qmd
similarity index 92%
rename from docs/multipack.md
rename to docs/multipack.qmd
index bee13b62c..097bcd2e5 100644
--- a/docs/multipack.md
+++ b/docs/multipack.qmd
@@ -1,4 +1,7 @@
-# Multipack (Sample Packing)
+---
+title: Multipack (Sample Packing)
+description: Multipack is a technique to pack multiple sequences into a single batch to increase training throughput.
+---
## Visualization of Multipack with Flash Attention
diff --git a/docs/nccl.md b/docs/nccl.qmd
similarity index 98%
rename from docs/nccl.md
rename to docs/nccl.qmd
index 4a7ff5d5d..3b616aa66 100644
--- a/docs/nccl.md
+++ b/docs/nccl.qmd
@@ -1,4 +1,7 @@
-# NCCL
+---
+title: NCCL
+description: Troubleshooting NCCL issues
+---
NVIDIA NCCL is a library to facilitate and optimize multi-GPU communication operations, such as broadcast, all-gather, reduce, all-reduce, etc. Broadly, NCCL configuration is highly environment-specific and is configured via several [environment variables](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html). A common NCCL-related problem occurs when a long-running operation times out causing the training process to abort:
diff --git a/docs/rlhf.md b/docs/rlhf.qmd
similarity index 90%
rename from docs/rlhf.md
rename to docs/rlhf.qmd
index 4f71184fc..7db68915a 100644
--- a/docs/rlhf.md
+++ b/docs/rlhf.qmd
@@ -1,4 +1,7 @@
-# RLHF (Beta)
+---
+title: "RLHF (Beta)"
+description: "Reinforcement Learning from Human Feedback is a method whereby a language model is optimized from data using human feedback."
+---
### Overview
diff --git a/favicon.jpg b/favicon.jpg
new file mode 100644
index 000000000..43c690244
Binary files /dev/null and b/favicon.jpg differ
diff --git a/index.qmd b/index.qmd
new file mode 100644
index 000000000..87d685880
--- /dev/null
+++ b/index.qmd
@@ -0,0 +1,19 @@
+
+
+```{python}
+#|output: asis
+#|echo: false
+
+# This cell steals the README as the home page for now, but excludes the table of contents (quarto adds its own)
+import re
+pattern = re.compile(
+ r"