fdsp config dict fix, todo list, add torchdistx support

2023-04-30 13:32:07 -04:00
parent 9190ada23a
commit ad2b48c0fa
3 changed files with 24 additions and 3 deletions
--- a/TODO.md
+++ b/TODO.md
@@ -0,0 +1,10 @@
+# todo list
+
+- [] Validation of parameters for combinations that won't work
+
+
+
+## things that are known not to work
+
+- FSDP offload and gradient_checkpointing - https://github.com/pytorch/pytorch/issues/82203
+- adamw_bnb_8bit doesn't play well with FSDP offload