# utils.tokenization

`utils.tokenization`

Module for tokenization utilities

## Functions

Name | Description
--- | ---
color_token_for_rl_debug | Helper function to color tokens based on their type.
process_tokens_for_rl_debug | Helper function to process and color tokens.

### color_token_for_rl_debug

    utils.tokenization.color_token_for_rl_debug(
        decoded_token,
        encoded_token,
        color,
        text_only,
    )

Helper function to color tokens based on their type.

### process_tokens_for_rl_debug

    utils.tokenization.process_tokens_for_rl_debug(
        tokens,
        color,
        tokenizer,
        text_only,
    )

Helper function to process and color tokens.