# utils.tokenization { #axolotl.utils.tokenization }

`utils.tokenization`

Module for tokenization utilities

## Functions

| Name | Description |
| --- | --- |
| [color_token_for_rl_debug](#axolotl.utils.tokenization.color_token_for_rl_debug) | Helper function to color tokens based on their type. |
| [process_tokens_for_rl_debug](#axolotl.utils.tokenization.process_tokens_for_rl_debug) | Helper function to process and color tokens. |

### color_token_for_rl_debug { #axolotl.utils.tokenization.color_token_for_rl_debug }

```python
utils.tokenization.color_token_for_rl_debug(
    decoded_token,
    encoded_token,
    color,
    text_only,
)
```

Helper function to color tokens based on their type.

### process_tokens_for_rl_debug { #axolotl.utils.tokenization.process_tokens_for_rl_debug }

```python
utils.tokenization.process_tokens_for_rl_debug(
    tokens,
    color,
    tokenizer,
    text_only,
)
```

Helper function to process and color tokens.