39 lines
964 B
Plaintext
39 lines
964 B
Plaintext
# utils.tokenization { #axolotl.utils.tokenization }
|
|
|
|
`utils.tokenization`
|
|
|
|
Module for tokenization utilities
|
|
|
|
## Functions
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [color_token_for_rl_debug](#axolotl.utils.tokenization.color_token_for_rl_debug) | Helper function to color tokens based on their type. |
|
|
| [process_tokens_for_rl_debug](#axolotl.utils.tokenization.process_tokens_for_rl_debug) | Helper function to process and color tokens. |
|
|
|
|
### color_token_for_rl_debug { #axolotl.utils.tokenization.color_token_for_rl_debug }
|
|
|
|
```python
|
|
utils.tokenization.color_token_for_rl_debug(
|
|
decoded_token,
|
|
encoded_token,
|
|
color,
|
|
text_only,
|
|
)
|
|
```
|
|
|
|
Helper function to color tokens based on their type.
|
|
|
|
### process_tokens_for_rl_debug { #axolotl.utils.tokenization.process_tokens_for_rl_debug }
|
|
|
|
```python
|
|
utils.tokenization.process_tokens_for_rl_debug(
|
|
tokens,
|
|
color,
|
|
tokenizer,
|
|
text_only,
|
|
)
|
|
```
|
|
|
|
Helper function to process and color tokens.
|