Qwen3.5-MoE example config with lora_target_modules regex (#3515) [skip ci]

* lora target modules with regex

* updates

* fsdp for non moe

* update wording

* chore: cleanup and lint

* chore: cleanup docs from merge

---------

Co-authored-by: NanoCode012 <nano@axolotl.ai>
This commit is contained in:
Owen Arliawan
2026-03-20 02:52:46 -07:00
committed by GitHub
parent 038ffe3f26
commit c57acef2c7
8 changed files with 309 additions and 44 deletions

View File

@@ -26,8 +26,6 @@ lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
# Targets the language model attention and MLP layers.
# Qwen3.5 is early-fusion: all layers (including those seeing vision tokens) share
# the same transformer stack, so standard attention targets work for both modalities.
lora_target_modules:
- q_proj
- k_proj