90e5598930
* fix: lock version in gemma3n docs * feat: add sample configs and docs * chore: move mistraltokenizer into mistral folder * feat: update instructions * feat: add dynamic load voxtral * fix: remove incorrect vision config, add audio * fix: support voxtral processing strategy and address none in data * feat: patch mistraltokenizer subclass upstream and add missing * feat: update cce commit to include voxtral * fix: remove old comment * fix: gemma3 patch not needed anymore * fix: voxtral modeling code * fix: remove incorrect ds path * fix: adjust apply chat template parsing * feat: enable voxtral patch * fix: patch * feat: update example datasets * fix: target layer * feat: update gemma3n docs * feat: update voxtral docs * feat: revert assistant parsing to rely on new upstream changes * chore: skip test till next PR fix * fix: override upstream decode due to missing handling * feat: update readme * fix: update * feat: add magistral small think support * feat: update mistral-common dep * fix: lint * fix: remove optional dep * chore: typing * chore: simply import * feat(doc): update differences for 2507 * fix: coderrabbit comments * feat: update clarify docs on new transformers
77 lines
2.8 KiB
Markdown
77 lines
2.8 KiB
Markdown
# Finetune Voxtral with Axolotl
|
|
|
|
Voxtral is a [3B](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507)/[24B](https://huggingface.co/mistralai/Voxtral-Small-24B-2507) parameter opensource model from MistralAI found on HuggingFace. This guide shows how to fine-tune it with Axolotl.
|
|
|
|
Thanks to the team at MistralAI for giving us early access to prepare for this release.
|
|
|
|
## Getting started
|
|
|
|
1. Install Axolotl following the [installation guide](https://docs.axolotl.ai/docs/installation.html). You need to install from main as Voxtral is only on nightly or use our latest [Docker images](https://docs.axolotl.ai/docs/docker.html).
|
|
|
|
Here is an example of how to install from main for pip:
|
|
|
|
```bash
|
|
# Ensure you have Pytorch installed (Pytorch 2.6.0 min)
|
|
git clone https://github.com/axolotl-ai-cloud/axolotl.git
|
|
cd axolotl
|
|
|
|
pip3 install packaging==23.2 setuptools==75.8.0 wheel ninja
|
|
pip3 install --no-build-isolation -e '.[flash-attn]'
|
|
```
|
|
|
|
2. Please install the below.
|
|
|
|
```bash
|
|
# audio
|
|
pip3 install librosa==0.11.0
|
|
pip3 install 'mistral_common[audio]==1.8.3'
|
|
```
|
|
|
|
3. Run the finetuning example:
|
|
|
|
```bash
|
|
# text only
|
|
axolotl train examples/voxtral/voxtral-mini-qlora.yml
|
|
|
|
# text + audio
|
|
axolotl train examples/voxtral/voxtral-mini-audio-qlora.yml
|
|
```
|
|
|
|
These configs use about 4.8 GB VRAM.
|
|
|
|
Let us know how it goes. Happy finetuning! 🚀
|
|
|
|
### TIPS
|
|
|
|
- For inference, the official MistralAI team recommends `temperature: 0.2` and `top_p: 0.95` for audio understanding and `temperature: 0.0` for transcription.
|
|
- You can run a full finetuning by removing the `adapter: qlora` and `load_in_4bit: true` from the config.
|
|
- Read more on how to load your own dataset at [docs](https://docs.axolotl.ai/docs/dataset_loading.html).
|
|
- The text dataset format follows the OpenAI Messages format as seen [here](https://docs.axolotl.ai/docs/dataset-formats/conversation.html#chat_template).
|
|
- The multimodal dataset format follows the OpenAI multi-content Messages format as seen [here](https://docs.axolotl.ai/docs/multimodal.html#dataset-format).
|
|
|
|
|
|
## Optimization Guides
|
|
|
|
- [Multi-GPU Training](https://docs.axolotl.ai/docs/multi-gpu.html)
|
|
- [Multi-Node Training](https://docs.axolotl.ai/docs/multi-node.html)
|
|
- [LoRA Optimizations](https://docs.axolotl.ai/docs/lora_optims.html)
|
|
|
|
## Limitations
|
|
|
|
We only support the `mistral-common` tokenizer for Supervised Fine-tuning at the moment and for `type: chat_template` only.
|
|
|
|
In addition, we do not support overriding tokens yet.
|
|
|
|
## Related Resources
|
|
|
|
- [MistralAI Magistral Blog](https://mistral.ai/news/magistral/)
|
|
- [Axolotl Docs](https://docs.axolotl.ai)
|
|
- [Axolotl Website](https://axolotl.ai)
|
|
- [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl)
|
|
- [Axolotl Discord](https://discord.gg/7m9sfhzaf3)
|
|
|
|
## Future Work
|
|
|
|
- Add parity to Preference Tuning, RL, etc.
|
|
- Add parity to other tokenizer configs like overriding tokens.
|