fix optimizer reset for relora sft (#1414)

* fix optimizer reset * set states to reset for 8bit optimizers and handle quantile runtime error for embeddings * fix relora test to check grad_norm * use flash attn for relora and tweak hyperparams for test * fix messages field for test dataset
2024-12-03 08:58:23 -05:00
parent 81ef3e45f7
commit 1ef70312ba
4 changed files with 64 additions and 30 deletions
--- a/requirements-tests.txt
+++ b/requirements-tests.txt
@@ -2,3 +2,4 @@ pytest
 pytest-xdist
 pytest-retry
 pytest-sugar
+tbparse