tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Built site for gh-pages

Browse Source
This commit is contained in:
Quarto GHA Workflow Runner
2025-06-25 12:39:05 +00:00
parent 26c3e80b1f
commit 7e84479334
4 changed files with 200 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/api/utils.ctx_managers.sequence_parallel.html
View File

@@ -685,7 +685,8 @@ from the full gradient tensor.</p>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> gradient_accumulation_steps,</span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> ring_attn_func,</span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> heads_k_stride,</span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a> gather_outputs,</span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>Context manager for sequence parallelism operations.</p>
<p>This class provides a context that will automatically apply sequence parallelism
during model forward passes using a pre-forward hook, and gather outputs from
@@ -738,6 +739,12 @@ across the sequence parallelism group using a post-forward hook.</p>
<td>Sequence parallelism K head stride size. Passed through to <code>varlen_llama3</code> <code>ring_flash_attn</code> implementation.</td>
<td><em>required</em></td>
</tr>
<tr class="even">
<td>gather_outputs</td>
<td>bool</td>
<td>Whether to gather outputs after model forward pass across the sequence parallel group.</td>
<td><em>required</em></td>
</tr>
</tbody>
</table>
</section>