tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Built site for gh-pages

Browse Source
This commit is contained in:
Quarto GHA Workflow Runner
2025-02-19 14:24:39 +00:00
parent d8295344ec
commit 109cae8122
4 changed files with 49 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
search.json
View File

@@ -1321,7 +1321,7 @@
"href": "docs/lora_optims.html",
"title": "LoRA Optimizations",
"section": "",
"text": "Inspired by Unsloth, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (in the DDP and DeepSpeed settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.\nWe currently support several common model architectures, including (but not limited to): - llama - mistral - qwen2 - gemma - gemma2"
"text": "Inspired by Unsloth, we’ve implemented two optimizations for LoRA and QLoRA fine-tuning, supporting both single GPU and multi-GPU (in the DDP and DeepSpeed settings) training. These include (1) SwiGLU and GEGLU activation function Triton kernels, and (2) LoRA MLP and attention custom autograd functions. Our goal was to leverage operator fusion and tensor re-use in order to improve speed and reduce memory usage during the forward and backward passes of these calculations.\nWe currently support several common model architectures, including (but not limited to):"
},
{
"objectID": "docs/lora_optims.html#usage",