Wing Lian
9871fa060b
optim e2e tests to run a bit faster ( #2069 ) [skip ci]
...
* optim e2e tests to run a bit faster
* run prequant w/o lora_modules_to_save
* use smollm2
2024-11-18 12:35:31 -05:00
Wing Lian
0dabde1962
support for schedule free and e2e ci smoke test ( #2066 ) [skip ci]
...
* support for schedule free and e2e ci smoke test
* set default lr scheduler to constant in test
* ignore duplicate code
* fix quotes for config/dict
2024-11-15 19:10:14 -05:00
Sunny Liu
1d7aee0ad2
ADOPT optimizer integration ( #2032 ) [skip ci]
...
* adopt integration
* stuff
* doc and test for ADOPT
* rearrangement
* fixed formatting
* hacking pre-commit
* chore: lint
* update module doc for adopt optimizer
* remove un-necessary example yaml for adopt optimizer
* skip test adopt if torch<2.5.1
* formatting
* use version.parse
* specifies required torch version for adopt_adamw
---------
Co-authored-by: sunny <sunnyliu19981005@gmail.com >
Co-authored-by: Wing Lian <wing@axolotl.ai >
2024-11-13 17:10:17 -05:00
Wing Lian
78e12f8ca5
add basic support for the optimi adamw optimizer ( #1727 )
...
* add support for optimi_adamw optimizer w kahan summation
* pydantic validator for optimi_adamw
* workaround for setting optimizer for fsdp
* make sure to install optimizer packages
* make sure to have parity for model parameters passed to optimizer
* add smoke test for optimi_adamw optimizer
* don't use foreach optimi by default
2024-07-14 19:12:57 -04:00
