Wing Lian 78e12f8ca5 add basic support for the optimi adamw optimizer (#1727) ...

* add support for optimi_adamw optimizer w kahan summation

* pydantic validator for optimi_adamw

* workaround for setting optimizer for fsdp

* make sure to install optimizer packages

* make sure to have parity for model parameters passed to optimizer

* add smoke test for optimi_adamw optimizer

* don't use foreach optimi by default

2024-07-14 19:12:57 -04:00

..

patched

bump flash attention 2.5.8 -> 2.6.1 (#1738)

2024-07-14 19:11:31 -04:00

__init__.py

missing dunder-init

2023-11-06 18:33:01 -05:00

.gitignore

Support Sample packing for phi arch (#586)

2023-09-15 15:46:54 -04:00

test_dpo.py

bump trl and accelerate for latest releases (#1730)

2024-07-10 11:15:44 -04:00

test_falcon.py

Falcon embeddings (#1149) [skip docker]

2024-01-22 21:01:42 -05:00

test_lora_llama.py

add basic support for the optimi adamw optimizer (#1727)

2024-07-14 19:12:57 -04:00

test_mamba.py

attempt to also run e2e tests that needs gpus (#1070)

2024-01-09 21:23:23 -05:00

test_mistral.py

use temp_dir kwarg instead

2023-11-06 18:33:01 -05:00

test_mixtral.py

fix broken linting (#1541)

2024-04-19 01:03:04 -04:00

test_optimizers.py

add basic support for the optimi adamw optimizer (#1727)

2024-07-14 19:12:57 -04:00

test_phi.py

re-enable phi for tests in modal ci (#1373)

2024-05-29 15:41:46 -04:00

test_relora_llama.py

relora: magnitude pruning of the optimizer (#1245)

2024-02-06 00:35:30 -05:00

utils.py

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00