243620394a
* fix: force train split for json,csv,txt for test_datasets * feat(doc): add info on mixing datasets for VLM * feat(doc): max memory * fix(doc): clarify lr groups * fix: add info on vision not being dropped * feat: add qwen3-vl to multimodal docs * fix: add moe blocks to arch list * feat(doc): improve mistral docs * chore: add helpful link [skip-e2e] * fix: add vram usage for mistral small * Update link in docs/faq.qmd Co-authored-by: salman <salman.mohammadi@outlook.com> --------- Co-authored-by: Wing Lian <wing@axolotl.ai> Co-authored-by: salman <salman.mohammadi@outlook.com>
1.9 KiB
1.9 KiB
Magistral Small Vision Fine-tuning
This guide covers fine-tuning Magistral Small 2509 with vision capabilities using Axolotl.
Prerequisites
Before starting, ensure you have:
- Installed Axolotl from source (see main README)
Getting started
-
Install the required vision lib:
pip install 'mistral-common[opencv]==1.8.5' -
Download the example dataset image:
wget https://huggingface.co/datasets/Nanobit/text-vision-2k-test/resolve/main/African_elephant.jpg -
Run the fine-tuning:
axolotl train examples/magistral/vision/magistral-small-vision-24B-qlora.yml
This config uses about 17GiB VRAM.
WARNING: The loss and grad norm will be much higher than normal at first. We suspect this to be inherent to the model as of the moment. If anyone would like to submit a fix for this, we are happy to take a look.
Tips
Key differences from text-only model:
max_tokens: 131072for inference- Multi-modal dataset format required
- Sample packing not supported
Dataset Format
The vision model requires multi-modal dataset format as documented here.
One exception is that, passing "image": PIL.Image is not supported. MistralTokenizer only supports path, url, and base64 for now.
Example:
{
"messages": [
{"role": "system", "content": [{ "type": "text", "text": "{SYSTEM_PROMPT}"}]},
{"role": "user", "content": [
{ "type": "text", "text": "What's in this image?"},
{"type": "image", "path": "path/to/image.jpg" }
]},
{"role": "assistant", "content": [{ "type": "text", "text": "..." }]},
],
}
Limitations
- Sample Packing is not supported for multi-modality training currently.