tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Built site for gh-pages

Browse Source
This commit is contained in:
Quarto GHA Workflow Runner
2025-02-19 14:24:39 +00:00
parent d8295344ec
commit 109cae8122
4 changed files with 49 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.nojekyll
View File

@@ -1 +1 @@
6e8747fa
2adccae7

9
docs/lora_optims.html
View File

@@ -369,7 +369,14 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
<p>Inspired by <a href="https://github.com/unslothai/unsloth">Unsloth</a>, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (in the DDP and DeepSpeed settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.</p>
<p>We currently support several common model architectures, including (but not limited to): - <code>llama</code> - <code>mistral</code> - <code>qwen2</code> - <code>gemma</code> - <code>gemma2</code></p>
<p>We currently support several common model architectures, including (but not limited to):</p>
<ul>
<li><code>llama</code></li>
<li><code>mistral</code></li>
<li><code>qwen2</code></li>
<li><code>gemma</code></li>
<li><code>gemma2</code></li>
</ul>
<details>
<p>The set of models we support is currently limited by our attention patching strategy, which assumes (and replaces) specific code blocks for query / key / value and output projections:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>ORIGINAL_QKV_CODE <span class="op">=</span> <span class="st">"""</span></span>

2
search.json
View File

@@ -1321,7 +1321,7 @@
"href": "docs/lora_optims.html",
"title": "LoRA Optimizations",
"section": "",
"text": "Inspired by Unsloth, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (in the DDP and DeepSpeed settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.\nWe currently support several common model architectures, including (but not limited to): - llama - mistral - qwen2 - gemma - gemma2"
"text": "Inspired by Unsloth, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (in the DDP and DeepSpeed settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.\nWe currently support several common model architectures, including (but not limited to):"
},
{
"objectID": "docs/lora_optims.html#usage",

78
sitemap.xml
View File

@@ -2,158 +2,158 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/index.html</loc>
<lastmod>2025-02-19T05:02:45.640Z</lastmod>
<lastmod>2025-02-19T14:23:44.385Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/rlhf.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/amd_hpc.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/faq.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/inference.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/unsloth.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/multimodal.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/input_output.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/index.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/template_free.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/inst_tune.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/stepwise_supervised.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/reward_modelling.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/fsdp_qlora.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/lr_groups.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/mac.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset_preprocessing.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/src/axolotl/integrations/cut_cross_entropy/ACKNOWLEDGEMENTS.html</loc>
<lastmod>2025-02-19T05:02:45.644Z</lastmod>
<lastmod>2025-02-19T14:23:44.388Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/examples/colab-notebooks/colab-axolotl-example.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/FAQS.html</loc>
<lastmod>2025-02-19T05:02:45.622Z</lastmod>
<lastmod>2025-02-19T14:23:44.369Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/TODO.html</loc>
<lastmod>2025-02-19T05:02:45.623Z</lastmod>
<lastmod>2025-02-19T14:23:44.369Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/src/axolotl/integrations/kd/topk_logprob/LICENSE.html</loc>
<lastmod>2025-02-19T05:02:45.644Z</lastmod>
<lastmod>2025-02-19T14:23:44.389Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/src/axolotl/integrations/LICENSE.html</loc>
<lastmod>2025-02-19T05:02:45.643Z</lastmod>
<lastmod>2025-02-19T14:23:44.388Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/installation.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/torchao.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/ray-integration.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/cli.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/debugging.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/tokenized.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/pretraining.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/dataset-formats/conversation.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/multi-gpu.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/batch_vs_grad.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/getting-started.html</loc>
<lastmod>2025-02-19T05:02:45.625Z</lastmod>
<lastmod>2025-02-19T14:23:44.371Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/config.html</loc>
<lastmod>2025-02-19T05:02:45.624Z</lastmod>
<lastmod>2025-02-19T14:23:44.370Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/multipack.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/multi-node.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/nccl.html</loc>
<lastmod>2025-02-19T05:02:45.629Z</lastmod>
<lastmod>2025-02-19T14:23:44.374Z</lastmod>
</url>
<url>
<loc>https://axolotl-ai-cloud.github.io/axolotl/docs/lora_optims.html</loc>
<lastmod>2025-02-19T05:02:45.628Z</lastmod>
<lastmod>2025-02-19T14:23:44.373Z</lastmod>
</url>
</urlset>