162 lines
5.1 KiB
Plaintext
162 lines
5.1 KiB
Plaintext
# utils.models { #axolotl.utils.models }
|
|
|
|
`utils.models`
|
|
|
|
Module for models and model loading
|
|
|
|
## Classes
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [ModelLoader](#axolotl.utils.models.ModelLoader) | ModelLoader: managing all the config and monkey patches while loading model |
|
|
|
|
### ModelLoader { #axolotl.utils.models.ModelLoader }
|
|
|
|
```python
|
|
utils.models.ModelLoader(
|
|
self,
|
|
cfg,
|
|
tokenizer,
|
|
*,
|
|
processor=None,
|
|
inference=False,
|
|
reference_model=False,
|
|
**kwargs,
|
|
)
|
|
```
|
|
|
|
ModelLoader: managing all the config and monkey patches while loading model
|
|
|
|
#### Attributes
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [has_flash_attn](#axolotl.utils.models.ModelLoader.has_flash_attn) | Check if flash attention is installed |
|
|
|
|
#### Methods
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [patch_llama_derived_model](#axolotl.utils.models.ModelLoader.patch_llama_derived_model) | Modify all llama derived models in one block |
|
|
| [patch_loss_llama](#axolotl.utils.models.ModelLoader.patch_loss_llama) | Patch loss functions and other optimizations |
|
|
| [set_attention_config](#axolotl.utils.models.ModelLoader.set_attention_config) | sample packing uses custom FA2 patch |
|
|
| [set_auto_model_loader](#axolotl.utils.models.ModelLoader.set_auto_model_loader) | set self.AutoModelLoader |
|
|
|
|
##### patch_llama_derived_model { #axolotl.utils.models.ModelLoader.patch_llama_derived_model }
|
|
|
|
```python
|
|
utils.models.ModelLoader.patch_llama_derived_model()
|
|
```
|
|
|
|
Modify all llama derived models in one block
|
|
|
|
##### patch_loss_llama { #axolotl.utils.models.ModelLoader.patch_loss_llama }
|
|
|
|
```python
|
|
utils.models.ModelLoader.patch_loss_llama()
|
|
```
|
|
|
|
Patch loss functions and other optimizations
|
|
|
|
##### set_attention_config { #axolotl.utils.models.ModelLoader.set_attention_config }
|
|
|
|
```python
|
|
utils.models.ModelLoader.set_attention_config()
|
|
```
|
|
|
|
sample packing uses custom FA2 patch
|
|
|
|
##### set_auto_model_loader { #axolotl.utils.models.ModelLoader.set_auto_model_loader }
|
|
|
|
```python
|
|
utils.models.ModelLoader.set_auto_model_loader()
|
|
```
|
|
|
|
set self.AutoModelLoader
|
|
- default value: AutoModelForCausalLM (set at __init__)
|
|
- when using a multi modality model, self.AutoModelLoader should
|
|
be set according to model type of the model
|
|
|
|
## Functions
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [get_module_class_from_name](#axolotl.utils.models.get_module_class_from_name) | Gets a class from a module by its name. |
|
|
| [load_model](#axolotl.utils.models.load_model) | Load a model for a given configuration and tokenizer. |
|
|
| [load_tokenizer](#axolotl.utils.models.load_tokenizer) | Load and configure the tokenizer based on the provided config. |
|
|
| [modify_tokenizer_files](#axolotl.utils.models.modify_tokenizer_files) | Modify tokenizer files to replace added_tokens strings, save to output directory, and return the path to the modified tokenizer. |
|
|
| [setup_quantized_meta_for_peft](#axolotl.utils.models.setup_quantized_meta_for_peft) | Replaces `quant_state.to` with a dummy function to prevent PEFT from moving `quant_state` to meta device |
|
|
| [setup_quantized_peft_meta_for_training](#axolotl.utils.models.setup_quantized_peft_meta_for_training) | Replaces dummy `quant_state.to` method with the original function to allow training to continue |
|
|
|
|
### get_module_class_from_name { #axolotl.utils.models.get_module_class_from_name }
|
|
|
|
```python
|
|
utils.models.get_module_class_from_name(module, name)
|
|
```
|
|
|
|
Gets a class from a module by its name.
|
|
|
|
Args:
|
|
module (`torch.nn.Module`): The module to get the class from.
|
|
name (`str`): The name of the class.
|
|
|
|
### load_model { #axolotl.utils.models.load_model }
|
|
|
|
```python
|
|
utils.models.load_model(
|
|
cfg,
|
|
tokenizer,
|
|
*,
|
|
processor=None,
|
|
inference=False,
|
|
reference_model=False,
|
|
**kwargs,
|
|
)
|
|
```
|
|
|
|
Load a model for a given configuration and tokenizer.
|
|
|
|
### load_tokenizer { #axolotl.utils.models.load_tokenizer }
|
|
|
|
```python
|
|
utils.models.load_tokenizer(cfg)
|
|
```
|
|
|
|
Load and configure the tokenizer based on the provided config.
|
|
|
|
### modify_tokenizer_files { #axolotl.utils.models.modify_tokenizer_files }
|
|
|
|
```python
|
|
utils.models.modify_tokenizer_files(tokenizer_path, token_mappings, output_dir)
|
|
```
|
|
|
|
Modify tokenizer files to replace added_tokens strings, save to output directory, and return the path to the modified tokenizer.
|
|
|
|
This only works with reserved tokens that were added to the tokenizer, not tokens already part of the vocab.
|
|
|
|
Args:
|
|
tokenizer_path: Path or name of the original tokenizer
|
|
token_mappings: Dict mapping {token_id (int): new_token_string}
|
|
output_dir: Directory to save the modified tokenizer
|
|
|
|
Returns:
|
|
Path to the modified tokenizer directory
|
|
|
|
Ref: https://github.com/huggingface/transformers/issues/27974#issuecomment-1854188941
|
|
|
|
### setup_quantized_meta_for_peft { #axolotl.utils.models.setup_quantized_meta_for_peft }
|
|
|
|
```python
|
|
utils.models.setup_quantized_meta_for_peft(model)
|
|
```
|
|
|
|
Replaces `quant_state.to` with a dummy function to prevent PEFT from moving `quant_state` to meta device
|
|
|
|
### setup_quantized_peft_meta_for_training { #axolotl.utils.models.setup_quantized_peft_meta_for_training }
|
|
|
|
```python
|
|
utils.models.setup_quantized_peft_meta_for_training(model)
|
|
```
|
|
|
|
Replaces dummy `quant_state.to` method with the original function to allow training to continue
|