tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Built site for gh-pages

Browse Source
This commit is contained in:
Quarto GHA Workflow Runner
2026-02-25 04:38:55 +00:00
parent aaf47dc7ec
commit dafe30369e
6 changed files with 1024 additions and 1004 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/api/loaders.tokenizer.html
View File

@@ -800,7 +800,8 @@ gtag('config', 'G-9KYCVJBNMQ', { 'anonymize_ip': true});
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a> tokenizer_path,</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a> token_mappings,</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> output_dir,</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a>)</span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> revision<span class="op">=</span><span class="st">'main'</span>,</span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a>)</span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
<p>Modify tokenizer files to replace added_tokens strings, save to output directory,
and return the path to the modified tokenizer.</p>
<p>This only works with reserved tokens that were added to the tokenizer, not tokens
@@ -809,10 +810,10 @@ already part of the vocab.</p>
<h4 class="doc-section doc-section-parameters anchored" data-anchor-id="parameters">Parameters</h4>
<table class="caption-top table">
<colgroup>
<col style="width: 16%">
<col style="width: 18%">
<col style="width: 51%">
<col style="width: 12%">
<col style="width: 15%">
<col style="width: 17%">
<col style="width: 54%">
<col style="width: 11%">
</colgroup>
<thead>
<tr class="header">
@@ -841,6 +842,12 @@ already part of the vocab.</p>
<td>Directory to save the modified tokenizer</td>
<td><em>required</em></td>
</tr>
<tr class="even">
<td>revision</td>
<td>str</td>
<td>Model revision/branch/tag/commit to load from (HF Hub)</td>
<td><code>'main'</code></td>
</tr>
</tbody>
</table>
</section>