NanoCode012 945c8aeb10 Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439) ...

* fix: saving clones state dict

* fix: apply fix for only CP mode

* fix: add dropout check when using lora target param

* fix: re-add patch from transformers PR #39866

* feat: add moe quant to test by ved

* fix: try match target param properly end with

* fix: clear cache per param quant

* fix: attempt on-load quantize experts instead of post-load

* fix: attempt disable async load

* chore: add log

* chore: adjust log

* fix: remove cuda alloc for moe and enable async load

* chore: remove leftover logs

* chore: add extra empty cache

* fix(doc): clarify support

* fix: handle fsdp2 for paramwrapper dtensor

* feat: attempt to quant experts in 8bit mode too

* feat: attempt to release bf16 experts from vram

* feat: upgrade cce

* fix: fsdp2 init_sharded_param load int8/uint4 dtensor as
require_grad=true on init

* fix: remove unnecessary gc and empty cache

* Revert "fix: remove unnecessary gc and empty cache"

This reverts commit 1d54518990.

* fix: do not call full_tensor on non-dtensors

* fix: attempt to address fsdp2 with quant exp high loss

* fix: attempt lora quant experts wrong dim

* fix: ensure require_grad patch applied for lora 8bit

* fix: attempt lora 8bit fsdp2

* fix: attribute access on save for lora 8bit fsdp2

* fix: wrong weight attrib access

* chore(refactor): add config, re-arrange position of patches, clean
comments

* feat: add example docs

* chore: cherry pick trinity fixes from PR 3399

* chore: comments refactor; add guards

* fix: guard using wrong key

* fix: mamba save does not accept main process param

* fix: guard prevent double hook

* fix: move gc to upper scope

* chore: add comment on proxy forward patch

* fix: add comment to clarify

* feat: add test idempotency

* fix: AttributeError: `e_score_correction_bias` is not an nn.Parameter

* fix: AttributeError: 'NoneType' object has no attribute 'to'

* fix: update docs on cpu_ram_efficient_loading

2026-03-03 10:06:23 -05:00

..

alst

feat(doc): add optimizations table of content to our improvements (#3175) [skip ci]

2025-09-24 16:13:49 -04:00

apertus

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

arcee

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

archived

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

cloud

deploy training jobs to baseten w truss in axolotl cli (#3086) [skip ci]

2025-08-26 09:29:50 -04:00

cohere

make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci]

2025-07-21 11:40:56 -04:00

colab-notebooks

Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439)

2026-03-03 10:06:23 -05:00

deepcogito

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

deepseek-v2

use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci]

2025-07-30 06:44:06 -04:00

devstral

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

distributed-parallel

feat: update nd parallelism readme (#3039)

2025-08-08 12:45:36 +01:00

eaft

EAFT (#3366) [skip ci]

2026-01-28 06:44:15 -05:00

falcon-h1

make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci]

2025-07-21 11:40:56 -04:00

gemma2

make pad_to_sequence_len default to the same value as sample_packing (#2941) [skip ci]

2025-07-21 11:40:56 -04:00

gemma3

fix: gemma3_text model loading vision config (#3354)

2026-01-13 09:49:23 -05:00

gemma3n

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

glm4

use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci]

2025-07-30 06:44:06 -04:00

glm4.7-flash

Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439)

2026-03-03 10:06:23 -05:00

glm46v

add glm support + patch (#3329) [skip ci]

2026-02-10 17:43:53 +07:00

gpt-oss

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

granite4

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

hunyuan

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

internvl3_5

feat: add internvl3_5 (#3141) [skip-ci]

2025-12-25 18:07:59 +07:00

jamba

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

kimi-linear

Feat: add kimi linear support (#3257)

2025-12-25 17:53:52 +07:00

LiquidAI

feat: add lfm2 family and latest moe model (#3208)

2025-10-09 10:47:41 -04:00

llama-2

Add FSDP v2 swap memory support + QLoRA compatibility fixes (#3167)

2025-09-26 10:23:59 +01:00

llama-3

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

llama-3-vision

fix: revert changing default optimizer to muon (#2965) [skip ci]

2025-07-22 10:00:30 -04:00

llama-4

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

llava

fix: revert changing default optimizer to muon (#2965) [skip ci]

2025-07-22 10:00:30 -04:00

magistral

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00

mamba

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

mimo

Feat: add MiMo and Plano (#3332) [skip-ci]

2025-12-25 18:09:03 +07:00

ministral

fix: improve ministral3 docs to be clearer (#3300)

2025-12-04 21:44:44 +07:00

ministral3

feat : scaled softmax support (#3338)

2026-01-13 14:33:11 +07:00

mistral

build examples readmes with quarto (#3046)

2025-12-25 19:17:25 +07:00

mistral-small

build examples readmes with quarto (#3046)

2025-12-25 19:17:25 +07:00

olmo3

feat: cleanup old flex mask patch, suppress Matmul bnb warn, and misc (#3330) [skip-ci]

2025-12-25 17:56:20 +07:00

orpheus

use warmup_ratio as a better default than warmup steps since it's data dependent (#2897) [skip ci]

2025-07-30 06:44:06 -04:00

phi

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

pixtral

Feat: add Magistral Small 2509 and native mistral3 tokenizer support (#3165)

2025-09-18 15:42:20 +07:00

plano

Feat: add MiMo and Plano (#3332) [skip-ci]

2025-12-25 18:09:03 +07:00

qat_nvfp4

Add QAT NVFP4 configs for blogpost (#3280) [skip ci]

2025-12-17 09:35:22 -05:00

qwen2

Distributed Muon Optimizer (#3264)

2025-12-19 10:43:47 -05:00

qwen2_5-vl

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

qwen2-vl

fix: unify default for conversations_field [skip-e2e] (#3070)

2025-09-23 21:22:15 +07:00

qwen3

fix: update qwen3 jinja tokenization off a few tokens (#3295)

2025-12-09 14:31:03 +07:00

qwen3-next

fix: qwen3-next to use fla causal-conv1d to support packing (#3437

2026-03-03 09:26:46 -05:00

seed-oss

Feat: add Olmo3 (BC with Olmo and Olmo2) (#3275)

2025-11-24 10:21:31 +07:00

slurm

Example for Slurm and various fixes (#3038) [skip ci]

2025-08-08 08:02:03 -04:00

smolvlm2

Feat: add Olmo3 (BC with Olmo and Olmo2) (#3275)

2025-11-24 10:21:31 +07:00

streaming

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

swanlab

feat: Add SwanLab integration for experiment tracking (#3334)

2026-01-06 09:19:18 -05:00

trinity

Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439)

2026-03-03 10:06:23 -05:00

voxtral

upgrade vllm to v0.14.0 (#3345)

2026-01-21 20:00:18 -05:00