* feat: systemic multimodal assistant-only loss masking + cfg.role_boundaries
Fixes silent ignoring of `cfg.train_on_inputs` / `cfg.roles_to_train` /
`cfg.train_on_eos` in the multimodal training path. Before this branch,
only Gemma 3n honored these knobs; every other VLM trained on the full
sequence regardless of config. Also adds `cfg.role_boundaries` YAML
override so users can declare per-role markers without subclassing.
What changed
------------
- `ProcessingStrategy` gains a declarative boundary scanner. Each
strategy declares per-role start/end markers via
`_build_role_boundaries`; the shared scanner honors
`train_on_inputs` / `roles_to_train` / `train_on_eos` (incl. "last").
- New per-template strategies: Gemma 4, Llama 3.2 Vision, Llama 4,
Pixtral, Mistral V7 Tekken.
- Refactored: Gemma 3 (previously no role masking), Gemma 3n
(previously ad-hoc scanner, now shared).
- Strategies whose boundary tokens couldn't be verified offline
(Voxtral, SmolVLM2, Mistral3, InternVL, GLM4V, llava/lfm2vl
fallback) retain legacy behavior and emit a one-shot warning. Users
can enable masking on them via `cfg.role_boundaries`.
- Pixtral / Mistral V7 Tekken correctly handle the shared `[/INST]`
token between user-end and assistant-start via `include_end=False`
+ scanner rewind.
See `docs/multimodal_assistant_mask.md` for the full audit table,
root-cause analysis, and design rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: systemic multimodal assistant-only loss masking + cfg.role_boundaries
Fixes silent ignoring of `cfg.train_on_inputs` / `cfg.roles_to_train` /
`cfg.train_on_eos` in the multimodal training path. Before this branch,
only Gemma 3n honored these knobs; every other VLM trained on the full
sequence regardless of config. Also adds `cfg.role_boundaries` YAML
override so users can declare per-role markers without subclassing.
What changed
------------
- `ProcessingStrategy` gains a declarative boundary scanner. Each
strategy declares per-role start/end markers via
`_build_role_boundaries`; the shared scanner honors
`train_on_inputs` / `roles_to_train` / `train_on_eos` (incl. "last").
- New per-template strategies: Gemma 4, Llama 3.2 Vision, Llama 4,
Pixtral, Mistral V7 Tekken.
- Refactored: Gemma 3 (previously no role masking), Gemma 3n
(previously ad-hoc scanner, now shared).
- Strategies whose boundary tokens couldn't be verified offline
(Voxtral, SmolVLM2, Mistral3, InternVL, GLM4V, llava/lfm2vl
fallback) retain legacy behavior and emit a one-shot warning. Users
can enable masking on them via `cfg.role_boundaries`.
- Pixtral / Mistral V7 Tekken correctly handle the shared `[/INST]`
token between user-end and assistant-start via `include_end=False`
+ scanner rewind.
See `docs/multimodal_assistant_mask.md` for the full audit table,
root-cause analysis, and design rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs+types: address CodeRabbit nitpicks on PR #7
- builders/causal.py: add inline NOTE that multi-dataset configs reuse
the first dataset's masking knobs (roles_to_train / train_on_eos) for
all datasets — heterogeneous per-dataset overrides are not supported
in the MM path today.
- processing_strategies.py: annotate inner scanner helpers
_match_prefix and _find_end with explicit types (Tensor, int,
list[int] → bool / tuple[int, bool]) for readability.
- docs/multimodal_assistant_mask.md: renumber the "Commits on this
branch" list to 1-7 consecutive (previously skipped 3).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mm-mask): address two CodeRabbit findings on PR #7
1. Schema rejected `train_on_eos: "none"` despite the scanner honoring it.
`_VALID_TRAIN_ON_EOS` accepts "none" and the design doc lists it, but
`SFTDataset.train_on_eos` was `Literal["all", "turn", "last"]`, so YAML
users hit a pydantic ValidationError at config load. Added "none" to
the Literal and updated the description.
2. `cfg.role_boundaries: []` had split-personality semantics: the strategy
ctor treated it as "replace built-ins with empty" while the collator
plumbing treated it as "unset", and both the design doc and the
MultiModalConfig schema help text promised wholesale replacement for
any set value. Aligned on opt-in semantics across all four surfaces —
a non-empty list replaces built-ins wholesale; unset or `[]` falls back
to built-ins. Rationale: honoring `[]` literally yields all-masked
labels and zero gradient, which is almost always a typo or leftover
rather than a deliberate user action. Users who want to disable role
masking should unset the field or use `train_on_inputs: true`.
Also sharpened the fallback one-shot warning for strategies without
built-in boundaries: names the consequence ("only pad and media tokens
are masked, every other token contributes to loss") and points users
at `cfg.role_boundaries` + docs/multimodal_assistant_mask.md instead
of "see axolotl/processing_strategies.py for how to declare
boundaries."
Files:
- src/axolotl/utils/schemas/datasets.py: Literal adds "none"
- src/axolotl/processing_strategies.py: ctor truthiness check on
role_boundaries_override; sharpened fallback warning
- src/axolotl/utils/schemas/multimodal.py: role_boundaries description
now calls out opt-in + empty-list fallback semantics
- docs/multimodal_assistant_mask.md: same clarification in the Semantics
block; updated the fallback-path detection paragraph to quote the new
warning text
- tests/test_processing_strategies.py: +2 regressions
(test_sft_dataset_schema_accepts_all_supported_train_on_eos_values,
test_empty_role_boundaries_override_falls_back_to_builtin); 63/63 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* doc cleanup
* fix(mm-mask): CodeRabbit findings + lint fix on PR #3625
Pre-commit failure: trailing newline missing on
docs/multimodal_assistant_mask.md (end-of-file-fixer hook).
Six CodeRabbit findings addressed:
1. Scanner: non-trainable role's end marker ignored ``include_end``.
Under ``train_on_eos="all"``, the shared ``[/INST]`` token (user-end
with ``include_end=False``, intentionally re-matched as assistant-start)
leaked into loss via the user branch on Pixtral / Mistral V7 Tekken.
Fix: gate the non-trainable branch on ``best_match.include_end`` to
mirror the trainable branch.
2. Gemma3 ``boi_token`` lookup used ``tokenizer.special_tokens_map.get("boi_token")``,
which never fires on real checkpoints (``special_tokens_map`` only
holds HF's standard slots — bos/eos/pad/unk/...). Swap to direct
attribute read ``getattr(tokenizer, "boi_token", None)``, matching
what ``transformers.models.gemma3.processing_gemma3`` itself does.
Updated the ``_gemma_tokenizer`` test fixture to mirror real-model
shape so the test exercises the production code path.
3. GLM dispatcher only registered ``Glm46VProcessor`` (GLM-4.6V /
GLM-4.7V). Real ``Glm4vProcessor`` (GLM-4V / GLM-4.1V) users fell
through to the base fallback. Both processors ship identical
media-token markers, so register both under the shared
``Glm4vProcessingStrategy`` with independent try/except import blocks.
Updated class docstring. +2 dispatcher regressions.
4. Gemma3 ``process_labels`` hardcoded 262144 for the soft image token.
Resolve dynamically via ``tokenizer.convert_tokens_to_ids("<image_soft_token>")``
with unk-id guard; fall back to 262144 only if the string isn't in
vocab. Mirrors ``Gemma4ProcessingStrategy.process_labels`` pattern.
5. ``build_collator`` was called twice per ``build()`` (eval + train
passes), producing two identical ``MM collator: ...`` INFO banners on
startup. Gate the log on ``is_eval=False`` so only the training pass
emits it.
6. Removed unused ``_mistral_common_stub`` pytest fixture (13 refs → 0,
always returned ``None``; the dispatcher already handles missing
``mistral_common`` via lazy import + ``try/except``). Added
``test_scanner_train_on_eos_all_with_non_trainable_include_end_false``
— a focused scanner-level lock-in for finding #1, independent of any
specific VLM strategy.
Test count: 63 → 68 passing. Local ``pre-commit run --all-files`` green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(mm-mask): hoist .tolist() out of scanner; shorten comments/docstrings
- Scanner perf: convert labels[i] to a Python list once per row so
_match_prefix / _find_end operate on list slices instead of
re-materializing Tensor slices via .tolist() on every probe. Cuts
O(n*boundaries) CPython↔C boundary crossings per batch.
- Markdown lint (MD001, MD040): promote two h3 section headings to h2
under the h1; add `text` language to the verify-at-runtime fenced block.
- Shorten verbose comments/docstrings added in recent commits to
bare-minimum "why" notes matching the repo's existing style.
68/68 tests, 8/8 pre-commit hooks still pass.
5352d41d32