Built site for gh-pages
This commit is contained in:
Quarto GHA Workflow Runner
@@ -472,6 +472,7 @@ gtag('config', 'G-9KYCVJBNMQ', { 'anonymize_ip': true});
|
||||
<li><a href="#classes" id="toc-classes" class="nav-link" data-scroll-target="#classes">Classes</a>
|
||||
<ul class="collapse">
|
||||
<li><a href="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer" id="toc-axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer" class="nav-link" data-scroll-target="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer">CPU_Offloaded_Gradient_Checkpointer</a></li>
|
||||
<li><a href="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload" id="toc-axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload" class="nav-link" data-scroll-target="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload">CheckpointFunctionWithCPUOffload</a></li>
|
||||
</ul></li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
@@ -501,6 +502,10 @@ gtag('config', 'G-9KYCVJBNMQ', { 'anonymize_ip': true});
|
||||
<td><a href="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer">CPU_Offloaded_Gradient_Checkpointer</a></td>
|
||||
<td>Saves VRAM by smartly offloading to RAM.</td>
|
||||
</tr>
|
||||
<tr class="even">
|
||||
<td><a href="#axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload">CheckpointFunctionWithCPUOffload</a></td>
|
||||
<td>This is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: <code>((100_000*4096)*2*32/2**30)</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<section id="axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer" class="level3">
|
||||
@@ -509,6 +514,13 @@ gtag('config', 'G-9KYCVJBNMQ', { 'anonymize_ip': true});
|
||||
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
|
||||
<p>Saves VRAM by smartly offloading to RAM.
|
||||
Tiny hit to performance, since we mask the movement via non blocking calls.</p>
|
||||
</section>
|
||||
<section id="axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload" class="level3">
|
||||
<h3 class="anchored" data-anchor-id="axolotl.monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload">CheckpointFunctionWithCPUOffload</h3>
|
||||
<div class="sourceCode" id="cb2"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>monkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload(</span>
|
||||
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
|
||||
<p>This is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: <code>((100_000*4096)*2*32/2**30)</code>
|
||||
In the case of a very long seqlen 100k+ the copying to/from cpu overhead is not big, because dense quadratic attention compute will dominate.</p>
|
||||
|
||||
|
||||
</section>
|
||||
|
||||
@@ -572,6 +572,21 @@ Tip
|
||||
<p>Start from Stage 1 -> Stage 2 -> Stage 3.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="callout callout-style-default callout-tip callout-titled">
|
||||
<div class="callout-header d-flex align-content-center">
|
||||
<div class="callout-icon-container">
|
||||
<i class="callout-icon"></i>
|
||||
</div>
|
||||
<div class="callout-title-container flex-fill">
|
||||
Tip
|
||||
</div>
|
||||
</div>
|
||||
<div class="callout-body-container callout-body">
|
||||
<p>Using ZeRO Stage 3 with Single-GPU training</p>
|
||||
<p>ZeRO Stage 3 can be used for training on a single GPU by manually setting the environment variables:
|
||||
<code>WORLD_SIZE=1 LOCAL_RANK=0 MASTER_ADDR=0.0.0.0 MASTER_PORT=29500</code></p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</section>
|
||||
<section id="sec-fsdp" class="level2" data-number="3">
|
||||
@@ -1161,65 +1176,74 @@ single sequence causes OOM errors during model training.</p>
|
||||
<span id="cb4-66"><a href="#cb4-66" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-67"><a href="#cb4-67" aria-hidden="true" tabindex="-1"></a>:::</span>
|
||||
<span id="cb4-68"><a href="#cb4-68" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-69"><a href="#cb4-69" aria-hidden="true" tabindex="-1"></a><span class="fu">## FSDP {#sec-fsdp}</span></span>
|
||||
<span id="cb4-69"><a href="#cb4-69" aria-hidden="true" tabindex="-1"></a>::: {.callout-tip}</span>
|
||||
<span id="cb4-70"><a href="#cb4-70" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-71"><a href="#cb4-71" aria-hidden="true" tabindex="-1"></a><span class="fu">### Basic FSDP Configuration {#sec-fsdp-config}</span></span>
|
||||
<span id="cb4-71"><a href="#cb4-71" aria-hidden="true" tabindex="-1"></a>Using ZeRO Stage 3 with Single-GPU training</span>
|
||||
<span id="cb4-72"><a href="#cb4-72" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-73"><a href="#cb4-73" aria-hidden="true" tabindex="-1"></a><span class="in">```{.yaml}</span></span>
|
||||
<span id="cb4-74"><a href="#cb4-74" aria-hidden="true" tabindex="-1"></a><span class="fu">fsdp</span><span class="kw">:</span></span>
|
||||
<span id="cb4-75"><a href="#cb4-75" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span>full_shard</span>
|
||||
<span id="cb4-76"><a href="#cb4-76" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span>auto_wrap</span>
|
||||
<span id="cb4-77"><a href="#cb4-77" aria-hidden="true" tabindex="-1"></a><span class="fu">fsdp_config</span><span class="kw">:</span></span>
|
||||
<span id="cb4-78"><a href="#cb4-78" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_offload_params</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
|
||||
<span id="cb4-79"><a href="#cb4-79" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_state_dict_type</span><span class="kw">:</span><span class="at"> FULL_STATE_DICT</span></span>
|
||||
<span id="cb4-80"><a href="#cb4-80" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_transformer_layer_cls_to_wrap</span><span class="kw">:</span><span class="at"> LlamaDecoderLayer</span></span>
|
||||
<span id="cb4-81"><a href="#cb4-81" aria-hidden="true" tabindex="-1"></a><span class="in">```</span></span>
|
||||
<span id="cb4-82"><a href="#cb4-82" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-83"><a href="#cb4-83" aria-hidden="true" tabindex="-1"></a><span class="fu">## Sequence parallelism {#sec-sequence-parallelism}</span></span>
|
||||
<span id="cb4-84"><a href="#cb4-84" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-85"><a href="#cb4-85" aria-hidden="true" tabindex="-1"></a>We support sequence parallelism (SP) via the</span>
|
||||
<span id="cb4-86"><a href="#cb4-86" aria-hidden="true" tabindex="-1"></a><span class="co">[</span><span class="ot">ring-flash-attention</span><span class="co">](https://github.com/zhuzilin/ring-flash-attention)</span> project. This</span>
|
||||
<span id="cb4-87"><a href="#cb4-87" aria-hidden="true" tabindex="-1"></a>allows one to split up sequences across GPUs, which is useful in the event that a</span>
|
||||
<span id="cb4-88"><a href="#cb4-88" aria-hidden="true" tabindex="-1"></a>single sequence causes OOM errors during model training.</span>
|
||||
<span id="cb4-89"><a href="#cb4-89" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-90"><a href="#cb4-90" aria-hidden="true" tabindex="-1"></a>See our <span class="co">[</span><span class="ot">dedicated guide</span><span class="co">](sequence_parallelism.qmd)</span> for more information.</span>
|
||||
<span id="cb4-73"><a href="#cb4-73" aria-hidden="true" tabindex="-1"></a>ZeRO Stage 3 can be used for training on a single GPU by manually setting the environment variables:</span>
|
||||
<span id="cb4-74"><a href="#cb4-74" aria-hidden="true" tabindex="-1"></a><span class="in">`WORLD_SIZE=1 LOCAL_RANK=0 MASTER_ADDR=0.0.0.0 MASTER_PORT=29500`</span></span>
|
||||
<span id="cb4-75"><a href="#cb4-75" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-76"><a href="#cb4-76" aria-hidden="true" tabindex="-1"></a>:::</span>
|
||||
<span id="cb4-77"><a href="#cb4-77" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-78"><a href="#cb4-78" aria-hidden="true" tabindex="-1"></a><span class="fu">## FSDP {#sec-fsdp}</span></span>
|
||||
<span id="cb4-79"><a href="#cb4-79" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-80"><a href="#cb4-80" aria-hidden="true" tabindex="-1"></a><span class="fu">### Basic FSDP Configuration {#sec-fsdp-config}</span></span>
|
||||
<span id="cb4-81"><a href="#cb4-81" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-82"><a href="#cb4-82" aria-hidden="true" tabindex="-1"></a><span class="in">```{.yaml}</span></span>
|
||||
<span id="cb4-83"><a href="#cb4-83" aria-hidden="true" tabindex="-1"></a><span class="fu">fsdp</span><span class="kw">:</span></span>
|
||||
<span id="cb4-84"><a href="#cb4-84" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span>full_shard</span>
|
||||
<span id="cb4-85"><a href="#cb4-85" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span>auto_wrap</span>
|
||||
<span id="cb4-86"><a href="#cb4-86" aria-hidden="true" tabindex="-1"></a><span class="fu">fsdp_config</span><span class="kw">:</span></span>
|
||||
<span id="cb4-87"><a href="#cb4-87" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_offload_params</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
|
||||
<span id="cb4-88"><a href="#cb4-88" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_state_dict_type</span><span class="kw">:</span><span class="at"> FULL_STATE_DICT</span></span>
|
||||
<span id="cb4-89"><a href="#cb4-89" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">fsdp_transformer_layer_cls_to_wrap</span><span class="kw">:</span><span class="at"> LlamaDecoderLayer</span></span>
|
||||
<span id="cb4-90"><a href="#cb4-90" aria-hidden="true" tabindex="-1"></a><span class="in">```</span></span>
|
||||
<span id="cb4-91"><a href="#cb4-91" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-92"><a href="#cb4-92" aria-hidden="true" tabindex="-1"></a><span class="fu">### FSDP + QLoRA {#sec-fsdp-qlora}</span></span>
|
||||
<span id="cb4-92"><a href="#cb4-92" aria-hidden="true" tabindex="-1"></a><span class="fu">## Sequence parallelism {#sec-sequence-parallelism}</span></span>
|
||||
<span id="cb4-93"><a href="#cb4-93" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-94"><a href="#cb4-94" aria-hidden="true" tabindex="-1"></a>For combining FSDP with QLoRA, see our <span class="co">[</span><span class="ot">dedicated guide</span><span class="co">](fsdp_qlora.qmd)</span>.</span>
|
||||
<span id="cb4-95"><a href="#cb4-95" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-96"><a href="#cb4-96" aria-hidden="true" tabindex="-1"></a><span class="fu">## Performance Optimization {#sec-performance}</span></span>
|
||||
<span id="cb4-97"><a href="#cb4-97" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-98"><a href="#cb4-98" aria-hidden="true" tabindex="-1"></a><span class="fu">### Liger Kernel Integration {#sec-liger}</span></span>
|
||||
<span id="cb4-99"><a href="#cb4-99" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-100"><a href="#cb4-100" aria-hidden="true" tabindex="-1"></a>Please see <span class="co">[</span><span class="ot">docs</span><span class="co">](custom_integrations.qmd#liger)</span> for more info.</span>
|
||||
<span id="cb4-101"><a href="#cb4-101" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-102"><a href="#cb4-102" aria-hidden="true" tabindex="-1"></a><span class="fu">## Troubleshooting {#sec-troubleshooting}</span></span>
|
||||
<span id="cb4-103"><a href="#cb4-103" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-104"><a href="#cb4-104" aria-hidden="true" tabindex="-1"></a><span class="fu">### NCCL Issues {#sec-nccl}</span></span>
|
||||
<span id="cb4-105"><a href="#cb4-105" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-106"><a href="#cb4-106" aria-hidden="true" tabindex="-1"></a>For NCCL-related problems, see our <span class="co">[</span><span class="ot">NCCL troubleshooting guide</span><span class="co">](nccl.qmd)</span>.</span>
|
||||
<span id="cb4-107"><a href="#cb4-107" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-108"><a href="#cb4-108" aria-hidden="true" tabindex="-1"></a><span class="fu">### Common Problems {#sec-common-problems}</span></span>
|
||||
<span id="cb4-109"><a href="#cb4-109" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-110"><a href="#cb4-110" aria-hidden="true" tabindex="-1"></a>::: {.panel-tabset}</span>
|
||||
<span id="cb4-111"><a href="#cb4-111" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-112"><a href="#cb4-112" aria-hidden="true" tabindex="-1"></a><span class="fu">## Memory Issues</span></span>
|
||||
<span id="cb4-113"><a href="#cb4-113" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-114"><a href="#cb4-114" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Reduce <span class="in">`micro_batch_size`</span></span>
|
||||
<span id="cb4-115"><a href="#cb4-115" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Reduce <span class="in">`eval_batch_size`</span></span>
|
||||
<span id="cb4-116"><a href="#cb4-116" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Adjust <span class="in">`gradient_accumulation_steps`</span></span>
|
||||
<span id="cb4-117"><a href="#cb4-117" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Consider using a higher ZeRO stage</span>
|
||||
<span id="cb4-94"><a href="#cb4-94" aria-hidden="true" tabindex="-1"></a>We support sequence parallelism (SP) via the</span>
|
||||
<span id="cb4-95"><a href="#cb4-95" aria-hidden="true" tabindex="-1"></a><span class="co">[</span><span class="ot">ring-flash-attention</span><span class="co">](https://github.com/zhuzilin/ring-flash-attention)</span> project. This</span>
|
||||
<span id="cb4-96"><a href="#cb4-96" aria-hidden="true" tabindex="-1"></a>allows one to split up sequences across GPUs, which is useful in the event that a</span>
|
||||
<span id="cb4-97"><a href="#cb4-97" aria-hidden="true" tabindex="-1"></a>single sequence causes OOM errors during model training.</span>
|
||||
<span id="cb4-98"><a href="#cb4-98" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-99"><a href="#cb4-99" aria-hidden="true" tabindex="-1"></a>See our <span class="co">[</span><span class="ot">dedicated guide</span><span class="co">](sequence_parallelism.qmd)</span> for more information.</span>
|
||||
<span id="cb4-100"><a href="#cb4-100" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-101"><a href="#cb4-101" aria-hidden="true" tabindex="-1"></a><span class="fu">### FSDP + QLoRA {#sec-fsdp-qlora}</span></span>
|
||||
<span id="cb4-102"><a href="#cb4-102" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-103"><a href="#cb4-103" aria-hidden="true" tabindex="-1"></a>For combining FSDP with QLoRA, see our <span class="co">[</span><span class="ot">dedicated guide</span><span class="co">](fsdp_qlora.qmd)</span>.</span>
|
||||
<span id="cb4-104"><a href="#cb4-104" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-105"><a href="#cb4-105" aria-hidden="true" tabindex="-1"></a><span class="fu">## Performance Optimization {#sec-performance}</span></span>
|
||||
<span id="cb4-106"><a href="#cb4-106" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-107"><a href="#cb4-107" aria-hidden="true" tabindex="-1"></a><span class="fu">### Liger Kernel Integration {#sec-liger}</span></span>
|
||||
<span id="cb4-108"><a href="#cb4-108" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-109"><a href="#cb4-109" aria-hidden="true" tabindex="-1"></a>Please see <span class="co">[</span><span class="ot">docs</span><span class="co">](custom_integrations.qmd#liger)</span> for more info.</span>
|
||||
<span id="cb4-110"><a href="#cb4-110" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-111"><a href="#cb4-111" aria-hidden="true" tabindex="-1"></a><span class="fu">## Troubleshooting {#sec-troubleshooting}</span></span>
|
||||
<span id="cb4-112"><a href="#cb4-112" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-113"><a href="#cb4-113" aria-hidden="true" tabindex="-1"></a><span class="fu">### NCCL Issues {#sec-nccl}</span></span>
|
||||
<span id="cb4-114"><a href="#cb4-114" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-115"><a href="#cb4-115" aria-hidden="true" tabindex="-1"></a>For NCCL-related problems, see our <span class="co">[</span><span class="ot">NCCL troubleshooting guide</span><span class="co">](nccl.qmd)</span>.</span>
|
||||
<span id="cb4-116"><a href="#cb4-116" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-117"><a href="#cb4-117" aria-hidden="true" tabindex="-1"></a><span class="fu">### Common Problems {#sec-common-problems}</span></span>
|
||||
<span id="cb4-118"><a href="#cb4-118" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-119"><a href="#cb4-119" aria-hidden="true" tabindex="-1"></a><span class="fu">## Training Instability</span></span>
|
||||
<span id="cb4-119"><a href="#cb4-119" aria-hidden="true" tabindex="-1"></a>::: {.panel-tabset}</span>
|
||||
<span id="cb4-120"><a href="#cb4-120" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-121"><a href="#cb4-121" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Start with DeepSpeed ZeRO-2</span>
|
||||
<span id="cb4-122"><a href="#cb4-122" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Monitor loss values</span>
|
||||
<span id="cb4-123"><a href="#cb4-123" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Check learning rates</span>
|
||||
<span id="cb4-124"><a href="#cb4-124" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-125"><a href="#cb4-125" aria-hidden="true" tabindex="-1"></a>:::</span>
|
||||
<span id="cb4-126"><a href="#cb4-126" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-127"><a href="#cb4-127" aria-hidden="true" tabindex="-1"></a>For more detailed troubleshooting, see our <span class="co">[</span><span class="ot">debugging guide</span><span class="co">](debugging.qmd)</span>.</span></code><button title="Copy to Clipboard" class="code-copy-button" data-in-quarto-modal=""><i class="bi"></i></button></pre></div>
|
||||
<span id="cb4-121"><a href="#cb4-121" aria-hidden="true" tabindex="-1"></a><span class="fu">## Memory Issues</span></span>
|
||||
<span id="cb4-122"><a href="#cb4-122" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-123"><a href="#cb4-123" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Reduce <span class="in">`micro_batch_size`</span></span>
|
||||
<span id="cb4-124"><a href="#cb4-124" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Reduce <span class="in">`eval_batch_size`</span></span>
|
||||
<span id="cb4-125"><a href="#cb4-125" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Adjust <span class="in">`gradient_accumulation_steps`</span></span>
|
||||
<span id="cb4-126"><a href="#cb4-126" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Consider using a higher ZeRO stage</span>
|
||||
<span id="cb4-127"><a href="#cb4-127" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-128"><a href="#cb4-128" aria-hidden="true" tabindex="-1"></a><span class="fu">## Training Instability</span></span>
|
||||
<span id="cb4-129"><a href="#cb4-129" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-130"><a href="#cb4-130" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Start with DeepSpeed ZeRO-2</span>
|
||||
<span id="cb4-131"><a href="#cb4-131" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Monitor loss values</span>
|
||||
<span id="cb4-132"><a href="#cb4-132" aria-hidden="true" tabindex="-1"></a><span class="ss">- </span>Check learning rates</span>
|
||||
<span id="cb4-133"><a href="#cb4-133" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-134"><a href="#cb4-134" aria-hidden="true" tabindex="-1"></a>:::</span>
|
||||
<span id="cb4-135"><a href="#cb4-135" aria-hidden="true" tabindex="-1"></a></span>
|
||||
<span id="cb4-136"><a href="#cb4-136" aria-hidden="true" tabindex="-1"></a>For more detailed troubleshooting, see our <span class="co">[</span><span class="ot">debugging guide</span><span class="co">](debugging.qmd)</span>.</span></code><button title="Copy to Clipboard" class="code-copy-button" data-in-quarto-modal=""><i class="bi"></i></button></pre></div>
|
||||
</div></div></div></div></div>
|
||||
</div> <!-- /content -->
|
||||
|
||||
|
||||
@@ -400,7 +400,7 @@
|
||||
"href": "docs/multi-gpu.html#sec-deepspeed",
|
||||
"title": "Multi-GPU",
|
||||
"section": "2 DeepSpeed",
|
||||
"text": "2 DeepSpeed\nDeepSpeed is the recommended approach for multi-GPU training due to its stability and performance. It provides various optimization levels through ZeRO stages.\n\n2.1 Configuration\nAdd to your YAML config:\ndeepspeed: deepspeed_configs/zero1.json\n\n\n2.2 Usage\n# Fetch deepspeed configs (if not already present)\naxolotl fetch deepspeed_configs\n\n# Passing arg via config\naxolotl train config.yml\n\n# Passing arg via cli\naxolotl train config.yml --deepspeed deepspeed_configs/zero1.json\n\n\n2.3 ZeRO Stages\nWe provide default configurations for:\n\nZeRO Stage 1 (zero1.json)\nZeRO Stage 1 with torch compile (zero1_torch_compile.json)\nZeRO Stage 2 (zero2.json)\nZeRO Stage 3 (zero3.json)\nZeRO Stage 3 with bf16 (zero3_bf16.json)\nZeRO Stage 3 with bf16 and CPU offload params(zero3_bf16_cpuoffload_params.json)\nZeRO Stage 3 with bf16 and CPU offload params and optimizer (zero3_bf16_cpuoffload_all.json)\n\n\n\n\n\n\n\nTip\n\n\n\nChoose the configuration that offloads the least amount to memory while still being able to fit on VRAM for best performance.\nStart from Stage 1 -> Stage 2 -> Stage 3.",
|
||||
"text": "2 DeepSpeed\nDeepSpeed is the recommended approach for multi-GPU training due to its stability and performance. It provides various optimization levels through ZeRO stages.\n\n2.1 Configuration\nAdd to your YAML config:\ndeepspeed: deepspeed_configs/zero1.json\n\n\n2.2 Usage\n# Fetch deepspeed configs (if not already present)\naxolotl fetch deepspeed_configs\n\n# Passing arg via config\naxolotl train config.yml\n\n# Passing arg via cli\naxolotl train config.yml --deepspeed deepspeed_configs/zero1.json\n\n\n2.3 ZeRO Stages\nWe provide default configurations for:\n\nZeRO Stage 1 (zero1.json)\nZeRO Stage 1 with torch compile (zero1_torch_compile.json)\nZeRO Stage 2 (zero2.json)\nZeRO Stage 3 (zero3.json)\nZeRO Stage 3 with bf16 (zero3_bf16.json)\nZeRO Stage 3 with bf16 and CPU offload params(zero3_bf16_cpuoffload_params.json)\nZeRO Stage 3 with bf16 and CPU offload params and optimizer (zero3_bf16_cpuoffload_all.json)\n\n\n\n\n\n\n\nTip\n\n\n\nChoose the configuration that offloads the least amount to memory while still being able to fit on VRAM for best performance.\nStart from Stage 1 -> Stage 2 -> Stage 3.\n\n\n\n\n\n\n\n\nTip\n\n\n\nUsing ZeRO Stage 3 with Single-GPU training\nZeRO Stage 3 can be used for training on a single GPU by manually setting the environment variables:\nWORLD_SIZE=1 LOCAL_RANK=0 MASTER_ADDR=0.0.0.0 MASTER_PORT=29500",
|
||||
"crumbs": [
|
||||
"Deployments",
|
||||
"Multi-GPU"
|
||||
@@ -2791,14 +2791,14 @@
|
||||
"href": "docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html",
|
||||
"title": "monkeypatch.gradient_checkpointing.offload_cpu",
|
||||
"section": "",
|
||||
"text": "monkeypatch.gradient_checkpointing.offload_cpu\nCPU offloaded checkpointing\n\n\n\n\n\nName\nDescription\n\n\n\n\nCPU_Offloaded_Gradient_Checkpointer\nSaves VRAM by smartly offloading to RAM.\n\n\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer(\n)\nSaves VRAM by smartly offloading to RAM.\nTiny hit to performance, since we mask the movement via non blocking calls."
|
||||
"text": "monkeypatch.gradient_checkpointing.offload_cpu\nCPU offloaded checkpointing\n\n\n\n\n\nName\nDescription\n\n\n\n\nCPU_Offloaded_Gradient_Checkpointer\nSaves VRAM by smartly offloading to RAM.\n\n\nCheckpointFunctionWithCPUOffload\nThis is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: ((100_000*4096)*2*32/2**30)\n\n\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer(\n)\nSaves VRAM by smartly offloading to RAM.\nTiny hit to performance, since we mask the movement via non blocking calls.\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload(\n)\nThis is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: ((100_000*4096)*2*32/2**30)\nIn the case of a very long seqlen 100k+ the copying to/from cpu overhead is not big, because dense quadratic attention compute will dominate."
|
||||
},
|
||||
{
|
||||
"objectID": "docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html#classes",
|
||||
"href": "docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html#classes",
|
||||
"title": "monkeypatch.gradient_checkpointing.offload_cpu",
|
||||
"section": "",
|
||||
"text": "Name\nDescription\n\n\n\n\nCPU_Offloaded_Gradient_Checkpointer\nSaves VRAM by smartly offloading to RAM.\n\n\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer(\n)\nSaves VRAM by smartly offloading to RAM.\nTiny hit to performance, since we mask the movement via non blocking calls."
|
||||
"text": "Name\nDescription\n\n\n\n\nCPU_Offloaded_Gradient_Checkpointer\nSaves VRAM by smartly offloading to RAM.\n\n\nCheckpointFunctionWithCPUOffload\nThis is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: ((100_000*4096)*2*32/2**30)\n\n\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CPU_Offloaded_Gradient_Checkpointer(\n)\nSaves VRAM by smartly offloading to RAM.\nTiny hit to performance, since we mask the movement via non blocking calls.\n\n\n\nmonkeypatch.gradient_checkpointing.offload_cpu.CheckpointFunctionWithCPUOffload(\n)\nThis is a torch/utils/checkpoint.py CheckpointFunction monkey patch that offloads the first tensor to cpu during forward and back to cuda during backward. This allows significant memory savings when using a very long seqlen. e.g. for llama 8b at 100k it’s 24GB saved per gpu: ((100_000*4096)*2*32/2**30)\nIn the case of a very long seqlen 100k+ the copying to/from cpu overhead is not big, because dense quadratic attention compute will dominate."
|
||||
},
|
||||
{
|
||||
"objectID": "docs/api/core.trainers.mamba.html",
|
||||
|
||||
378
sitemap.xml
378
sitemap.xml
@@ -2,758 +2,758 @@
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/unsloth.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.713Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/conversation.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/stepwise_supervised.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/tokenized.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/mac.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/nccl.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/multi-node.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/docker.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/lr_groups.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/inference.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.805Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/cli.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/config-reference.html</loc>
|
||||
<lastmod>2025-07-09T13:26:17.755Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:58.872Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/multi-gpu.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/debugging.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/multimodal.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.sweeps.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.594Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.406Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.llama3.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.923Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.735Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schedulers.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.310Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.123Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_xformers.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.123Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.935Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.cloud.modal_.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.643Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.454Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/kernels.geglu.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.076Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.888Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.716Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.530Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.datasets.chat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.432Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.256Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.peft.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.424Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.237Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.btlm_attn_hijack_flash.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.187Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.997Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.chat_template.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.825Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.638Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.kto.user_defined.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.956Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.768Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.cloud.base.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.636Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.448Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/kernels.swiglu.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.086Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.898Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.stepwise_supervised.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.890Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.702Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.bradley_terry.llama3.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.980Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.792Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.completion.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.879Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.692Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/kernels.utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.095Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.907Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/common.datasets.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.635Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.452Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.datasets.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.415Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.228Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.builders.rl.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.386Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.211Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/evaluate.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.291Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.116Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/kernels.quantize.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.094Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.906Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_flash.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.122Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.933Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.mixins.rng_state_loader.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.760Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.574Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.base.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.595Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.412Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.merge_lora.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.557Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.380Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.merge_sharded_fsdp_weights.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.569Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.392Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.transformers_fa_utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.204Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.014Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.llama2_chat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.873Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.686Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.collators.mm_chat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.664Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.482Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.data.sft.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.349Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.163Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_instruct.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.840Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.653Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.liger.args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.610Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.428Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.mistral_attn_hijack_flash.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.138Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.949Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.vllm_serve.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.633Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.445Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.185Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.996Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.patch_manager.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.750Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.563Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.integrations.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.444Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.258Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.perplexity.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.712Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.528Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.626Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.438Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.config.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.385Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.198Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.input_output.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.885Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.697Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.distributed.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.330Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.143Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_disk.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.238Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.050Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.trainer_fsdp_optim.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.198Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.008Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.builders.base.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.374Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.198Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.trl.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.675Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.486Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.evaluate.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.489Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.312Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.optimizers.adopt.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.341Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.155Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.qat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.731Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.547Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.dpo.trainer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.692Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.502Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.chat.format.shared.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.427Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.251Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.relora.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.147Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.957Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.config.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.533Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.356Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.preprocess.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.585Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.400Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.base.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.658Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.470Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/convert.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.315Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.141Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.pygmalion.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.907Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.719Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.trl.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.427Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.241Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.509Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.331Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chat_template.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.913Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.725Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.constants.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.751Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.565Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/logging_config.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.367Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.192Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.inference.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.548Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.371Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.ctx_managers.sequence_parallel.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.790Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.604Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.spectrum.args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.617Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.434Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.training.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.397Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.210Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.orcamini.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.900Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.713Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.freeze.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.268Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.081Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.tokenizer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.735Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.548Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.bench.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.260Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.073Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.quantization.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.371Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.184Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/batch_vs_grad.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/input_output.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/sequence_parallelism.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/reward_modelling.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/index.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.821Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.727Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/src/axolotl/integrations/LICENSE.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.825Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.731Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/FAQS.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.801Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.706Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/src/axolotl/integrations/cut_cross_entropy/ACKNOWLEDGEMENTS.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.825Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.731Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/TODO.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.801Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.707Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/examples/colab-notebooks/colab-axolotl-example.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.808Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.714Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/torchao.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/ray-integration.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/quantize.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/qat.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.lora.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.251Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.064Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_w_system.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.852Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.665Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.stablelm_attn_hijack_flash.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.194Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.004Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.collators.core.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.637Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.455Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.metharme.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.896Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.profiler.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.716Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.532Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.data.pretraining.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.343Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.156Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.lisa.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.717Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.533Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.trainer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.286Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.098Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.cut_cross_entropy.args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.598Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.416Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.model.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.392Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.205Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.data.batch_dataset_fetcher.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.207Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.017Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.zephyr.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.935Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.747Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/datasets.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.301Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.127Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.enums.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.455Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.269Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.kd.trainer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.607Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.424Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.lora_kernels.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.177Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.988Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.collators.batching.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.656Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.473Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.grpo.sampler.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.715Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.529Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.base.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.792Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.606Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.multipack.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.140Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.951Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.orpo.chat_template.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.976Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.788Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.passthrough.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.938Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.750Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.chat.format.chatml.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.424Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.248Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.mixins.scheduler.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.767Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.580Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.model_shard_quant.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.257Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.069Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.kto.chatml.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.954Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.766Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.tokenization.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.245Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.057Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.model.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.726Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.540Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.mlflow_.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.721Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.537Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.grpo.trainer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.702Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.517Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.main.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.472Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.295Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.callbacks.comet_.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.724Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.540Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.chat_templates.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.246Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.059Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.utils.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.460Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.275Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/common.architectures.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.618Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.436Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.llama_expand_mask.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.148Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.959Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_chat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.839Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.652Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.samplers.multipack.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.706Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.522Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.grokfast.optimizer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.599Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.417Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chatml.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.933Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.745Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.mixtral.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.209Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.019Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/train.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.280Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.106Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.llama_patch_multipack.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.188Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.999Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/index.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.218Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.043Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.adapter.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.742Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.555Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.schemas.multimodal.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.432Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.246Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/kernels.lora.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.065Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.877Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.kto.llama3.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.946Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.758Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.checks.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.515Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.338Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.quantize.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.648Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.459Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/integrations.lm_eval.args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.613Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.431Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.chat.messages.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.422Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.246Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.builders.causal.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.378Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.203Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.relora.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.685Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.496Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/models.mamba.modeling_mamba.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.636Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.453Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.212Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.024Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.mamba.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.680Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.491Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.datasets.transforms.chat_builder.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.440Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.264Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/loaders.processor.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.736Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.550Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.chat.format.llama3x.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.425Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.249Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.messages.chat.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.911Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.723Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/cli.train.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.481Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.303Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.trainers.mixins.optimizer.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.757Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.571Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.collators.mamba.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.660Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.477Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/monkeypatch.unsloth_.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.206Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.016Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/utils.dict.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.333Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.147Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.user_defined.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.861Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.673Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/core.training_args.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.399Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.223Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.user_defined.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.936Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.748Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/prompt_tokenizers.html</loc>
|
||||
<lastmod>2025-07-09T13:26:03.358Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:44.182Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/api/common.const.html</loc>
|
||||
<lastmod>2025-07-09T13:26:04.620Z</lastmod>
|
||||
<lastmod>2025-07-09T16:51:45.437Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/fsdp_qlora.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/custom_integrations.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/getting-started.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/faq.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.709Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/lora_optims.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/rlhf.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/amd_hpc.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/installation.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/multipack.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.806Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.712Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset_preprocessing.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset_loading.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.803Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/inst_tune.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/template_free.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/index.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
<url>
|
||||
<loc>https://docs.axolotl.ai/docs/dataset-formats/pretraining.html</loc>
|
||||
<lastmod>2025-07-09T13:22:46.802Z</lastmod>
|
||||
<lastmod>2025-07-09T16:48:34.708Z</lastmod>
|
||||
</url>
|
||||
</urlset>
|
||||
|
||||
Reference in New Issue
Block a user