quartodoc integration

docs/api/utils.tokenization.qmd

@@ -0,0 +1,38 @@
+# utils.tokenization { #axolotl.utils.tokenization }
+
+`utils.tokenization`
+
+Module for tokenization utilities
+
+## Functions
+
+| Name | Description |
+| --- | --- |
+| [color_token_for_rl_debug](#axolotl.utils.tokenization.color_token_for_rl_debug) | Helper function to color tokens based on their type. |
+| [process_tokens_for_rl_debug](#axolotl.utils.tokenization.process_tokens_for_rl_debug) | Helper function to process and color tokens. |
+
+### color_token_for_rl_debug { #axolotl.utils.tokenization.color_token_for_rl_debug }
+
+```python
+utils.tokenization.color_token_for_rl_debug(
+    decoded_token,
+    encoded_token,
+    color,
+    text_only,
+)
+```
+
+Helper function to color tokens based on their type.
+
+### process_tokens_for_rl_debug { #axolotl.utils.tokenization.process_tokens_for_rl_debug }
+
+```python
+utils.tokenization.process_tokens_for_rl_debug(
+    tokens,
+    color,
+    tokenizer,
+    text_only,
+)
+```
+
+Helper function to process and color tokens.