98 lines
2.2 KiB
Plaintext
98 lines
2.2 KiB
Plaintext
---
|
||
title: "LLMCompressor Sparse Fine-tuning"
|
||
format:
|
||
html:
|
||
toc: true
|
||
toc-depth: 3
|
||
number-sections: true
|
||
execute:
|
||
enabled: false
|
||
---
|
||
|
||
# LLMCompressor Integration
|
||
|
||
Fine-tune sparsified models in Axolotl using [LLMCompressor](https://github.com/vllm-project/llm-compressor).
|
||
|
||
This integration enables fine-tuning of models **already sparsified** using LLMCompressor.
|
||
It hooks into Axolotl’s training pipeline using the plugin system and maintains sparsity throughout the fine-tuning process.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
- Install Axolotl with `llmcompressor` extras:
|
||
|
||
```bash
|
||
pip install "axolotl[llmcompressor]"
|
||
```
|
||
|
||
- Requires `llmcompressor >= 0.5.1`
|
||
|
||
This will install all required dependencies for sparse model fine-tuning.
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
To enable sparse fine-tuning with this integration, configure your Axolotl YAML like so:
|
||
|
||
```yaml
|
||
plugins:
|
||
- axolotl.integrations.llm_compressor.LLMCompressorPlugin
|
||
|
||
llmcompressor:
|
||
recipe:
|
||
finetuning_stage:
|
||
finetuning_modifiers:
|
||
ConstantPruningModifier:
|
||
targets: [
|
||
're:.*q_proj.weight',
|
||
're:.*k_proj.weight',
|
||
're:.*v_proj.weight',
|
||
're:.*o_proj.weight',
|
||
're:.*gate_proj.weight',
|
||
're:.*up_proj.weight',
|
||
're:.*down_proj.weight',
|
||
]
|
||
start: 0
|
||
# ... (other Axolotl training arguments)
|
||
```
|
||
|
||
::: {.callout-note}
|
||
This plugin **does not prune or sparsify the model**. It is only meant for **fine-tuning models that are already sparsified**.
|
||
:::
|
||
|
||
---
|
||
|
||
## Pre-Sparsified Checkpoints
|
||
|
||
You can use:
|
||
|
||
- Your own LLMCompressor-sparsified model
|
||
- Or one from [Neural Magic's Hugging Face page](https://huggingface.co/neuralmagic)
|
||
|
||
Refer to the [LLMCompressor README](https://github.com/vllm-project/llm-compressor/blob/main/README.md) to learn how to sparsify models or write custom recipes.
|
||
|
||
---
|
||
|
||
## Example Config
|
||
|
||
A full working example is provided at:
|
||
|
||
```bash
|
||
examples/llama-3/sparse-finetuning.yaml
|
||
```
|
||
|
||
Run fine-tuning using:
|
||
|
||
```bash
|
||
axolotl train examples/llama-3/sparse-finetuning.yaml
|
||
```
|
||
|
||
---
|
||
|
||
## Learn More
|
||
|
||
Explore LLMCompressor capabilities, supported modifiers, and detailed examples:
|
||
|
||
👉 [LLMCompressor GitHub](https://github.com/vllm-project/llm-compressor) |