NanoCode012 945c8aeb10 Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439) ...

* fix: saving clones state dict

* fix: apply fix for only CP mode

* fix: add dropout check when using lora target param

* fix: re-add patch from transformers PR #39866

* feat: add moe quant to test by ved

* fix: try match target param properly end with

* fix: clear cache per param quant

* fix: attempt on-load quantize experts instead of post-load

* fix: attempt disable async load

* chore: add log

* chore: adjust log

* fix: remove cuda alloc for moe and enable async load

* chore: remove leftover logs

* chore: add extra empty cache

* fix(doc): clarify support

* fix: handle fsdp2 for paramwrapper dtensor

* feat: attempt to quant experts in 8bit mode too

* feat: attempt to release bf16 experts from vram

* feat: upgrade cce

* fix: fsdp2 init_sharded_param load int8/uint4 dtensor as
require_grad=true on init

* fix: remove unnecessary gc and empty cache

* Revert "fix: remove unnecessary gc and empty cache"

This reverts commit 1d54518990.

* fix: do not call full_tensor on non-dtensors

* fix: attempt to address fsdp2 with quant exp high loss

* fix: attempt lora quant experts wrong dim

* fix: ensure require_grad patch applied for lora 8bit

* fix: attempt lora 8bit fsdp2

* fix: attribute access on save for lora 8bit fsdp2

* fix: wrong weight attrib access

* chore(refactor): add config, re-arrange position of patches, clean
comments

* feat: add example docs

* chore: cherry pick trinity fixes from PR 3399

* chore: comments refactor; add guards

* fix: guard using wrong key

* fix: mamba save does not accept main process param

* fix: guard prevent double hook

* fix: move gc to upper scope

* chore: add comment on proxy forward patch

* fix: add comment to clarify

* feat: add test idempotency

* fix: AttributeError: `e_score_correction_bias` is not an nn.Parameter

* fix: AttributeError: 'NoneType' object has no attribute 'to'

* fix: update docs on cpu_ram_efficient_loading

2026-03-03 10:06:23 -05:00

..

cli

feat: support dot-notation CLI args for nested config options (#3419)

2026-02-23 10:10:06 -05:00

core

upgrade transformers to 5.2.0 and torchao to 0.16.0 (#3407)

2026-02-19 18:27:27 -05:00

e2e

mark slow tests that are timing out in CI (#3428) [skip ci]

2026-03-02 12:26:30 -05:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

ScatterMoE LoRA support (#3410)

2026-02-24 14:59:55 -05:00

monkeypatch

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

patched

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

prompt_strategies

handle warnings from v5 upgrade (#3376)

2026-01-28 06:45:01 -05:00

telemetry

fix: remove telemetry warning (#3397) [skip ci]

2026-02-10 23:01:16 +07:00

utils

Fix: quantize and target moe layers in transformers v5 for adapters and many misc fixes (#3439)

2026-03-03 10:06:23 -05:00

__init__.py

fix: minor patches for multimodal (#2441)

2025-03-31 13:40:12 +07:00

conftest.py

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

constants.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

hf_offline_utils.py

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

test_chunked_xentropy.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_data.py

Fix: excess_length_strategy truncation method (#3401)

2026-02-25 11:31:11 +07:00

test_datasets.py

feature: raise on long sequence drop (#3321)

2025-12-22 13:59:49 -05:00

test_dict.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_exact_deduplication.py

feat:add support dataset_num_processes (#3129) [skip ci]

2025-10-13 17:18:12 +07:00

test_expand_mask.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_loaders.py

fix: transformers deprecate load_in_Xbit in model_kwargs (#3205)

2025-10-16 16:07:27 +07:00

test_logging_config_file_capture.py

Debug log, logging improvements (#3159)

2025-09-17 13:27:03 -04:00

test_lora.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_normalize_config.py

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

test_opentelemetry_callback.py

Feat/opentelemetry (#3215)

2025-10-22 19:16:55 -07:00

test_packed_batch_sampler.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_packed_dataset.py

feat:add support dataset_num_processes (#3129) [skip ci]

2025-10-13 17:18:12 +07:00

test_packed_pretraining.py

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

test_perplexity.py

transformers v5 upgrade (#3272)

2026-01-27 17:08:24 -05:00

test_prompt_tokenizers.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_revision_parameter.py

fix: pass revision parameter to tokenizer and processor loaders (#3388) [skip ci]

2026-02-25 11:11:20 +07:00

test_save_deduplicated.py

fix: Save de-duplicated dataset during pre-processing (#3427)

2026-03-02 12:55:59 -05:00

test_schedulers.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_streaming.py

text diffusion training plugin (#3067)

2025-09-10 20:27:00 -04:00

test_tokenizers.py

mark slow tests that are timing out in CI (#3428) [skip ci]

2026-03-02 12:26:30 -05:00

test_train.py

refactor dupes from merge/rebase (#2919) [skip ci]

2025-07-14 10:05:26 -04:00

test_utils_tee.py

Debug log, logging improvements (#3159)

2025-09-17 13:27:03 -04:00

test_validation_dataset.py

Distributed Muon Optimizer (#3264)

2025-12-19 10:43:47 -05:00