Dan Saunders 208fb7b8e7 basic torchao fp8 mixed precision training (#2926) ...

* debug

* debug

* debug

* revert unneeded change

* add accelerator config to base trainer builder

* add back accumulated_cache_size_limit setting

* lint

* accelerator constructor patch for single-GPU torch fp8

* lint

* re-using existing fp8 code

* lint

* remove accelerate patch now fix in latest release

* fix

* docs

* add fp8 + fsdp2 example

* remove unused config

* update config

* smoke tests

* add validator

* add 2.7.0 guard for fsdp2

* fix

* add config descriptions

* add FSDP doc link

* nit

* set force_recompute_fp8_weight_in_bwd with enable_fsdp_float8_all_gather

* better cfg for smoke tests

* add test for accelerate patching

* update fp8 validator

2025-07-22 16:27:47 -04:00

..

integrations

basic torchao fp8 mixed precision training (#2926)

2025-07-22 16:27:47 -04:00

kernels

mark flaky geglu tests and add torch seed (#2876) [skip ci]

2025-07-07 15:24:16 -04:00

multigpu

basic torchao fp8 mixed precision training (#2926)

2025-07-22 16:27:47 -04:00

patched

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

solo

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

__init__.py

missing dunder-init

2023-11-06 18:33:01 -05:00

.gitignore

Support Sample packing for phi arch (#586)

2023-09-15 15:46:54 -04:00

test_deepseekv3.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_dpo.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_embeddings_lr.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_evaluate.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_falcon.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_gemma2.py

Data loader refactor (#2707)

2025-06-10 19:53:07 -04:00

test_gemma3_text.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_imports.py

Fix: RL base feature parity (#2133)

2025-05-30 11:21:47 +07:00

test_llama_pretrain.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_llama_vision.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_llama.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_load_model.py

models.py -> loaders/ module refactor (#2680)

2025-05-23 15:51:11 -04:00

test_lora_llama.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_mamba.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_mistral.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_mixtral.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_optimizers.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_packing_loss.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_phi.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_preprocess.py

don't init distributed for deepspeed if preprocessing (#2920)

2025-07-14 14:19:19 -04:00

test_process_reward_model_smollm2.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_profiler.py

allow profiling in mid-training rather from the start (#2899) [skip ci]

2025-07-14 20:11:11 -04:00

test_qat.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_quantization.py

QAT (#2590)

2025-05-28 12:35:47 +01:00

test_qwen.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_reward_model_smollm2.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_save_first_step.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

test_schedulers.py

checkpoint model on first step callback (#2906)

2025-07-15 15:00:48 -04:00

utils.py

FSDP1 -> FSDP2 (#2760)

2025-07-12 15:18:01 +01:00