axolotl/examples/ministral3

Finetune Ministral3 with Axolotl

Ministral3 is a family of open-weight models from MistralAI found on HuggingFace. This guide shows how to fine-tune it with Axolotl with multi-turn conversations and proper masking.

Please see Thinking and Vision for their respective fine-tuning.

Thanks to the team at MistralAI for giving us early access to prepare for these releases.

Note: This is still experimental given it is based on transformers v5 RC.

Getting started

1. Install Axolotl from source following the installation guide.

2. Install Cut Cross Entropy to reduce training VRAM usage.

3. Swap to the Axolotl transformers v5 branch

   cp examples/ministral3/ministral3-3b-qlora.yaml ministral3-3b-qlora.yaml
   
   git fetch
   git checkout transformers-v5
   
   # Install packages for transformers v5
   pip install -e .
   

4. Run the fine-tuning:

   axolotl train ministral3-3b-qlora.yaml
   

Let us know how it goes. Happy finetuning! 🚀

Tips

• We recommend adding the same/similar SystemPrompt that the model is tuned for. You can find this within the repo's files titled SYSTEM_PROMPT.txt.
• You can run a full finetuning by removing the adapter: qlora and load_in_4bit: true from the config.
• Read more on how to load your own dataset at docs.
• The text dataset format follows the OpenAI Messages format as seen here.

Thinking

Ministral3 2512 model supports thinking capabilities, enabling Chain-of-Thought reasoning with explicit thinking steps.

📚 See the Thinking fine-tuning guide →

Vision

Ministral3 2512 model also supports vision capabilities.

📚 See the Vision fine-tuning guide →

Optimization Guides

Please check the Optimizations doc.

Limitations

We only support the mistral-common tokenizer for Supervised Fine-tuning at the moment and for type: chat_template only.

In addition, we do not support overriding tokens yet.

Related Resources

• MistralAI Mistral3 Blog
• Axolotl Docs
• Axolotl Website
• Axolotl GitHub
• Axolotl Discord

Future Work

• Add parity to Preference Tuning, RL, etc.
• Add parity to other tokenizer configs like overriding tokens.