support for mamba (#915)

* support for mamba

* more mamba fixes

* use fork for mamba kwargs fix

* grad checkpointing doesn't work

* fix extras for mamaba

* mamba loss fix

* use fp32 and remove verbose logging

* mamba fixes

* fix collator for mamba

* set model_type on training_args

* don't save safetensors for mamba

* update mamba config to disable safetensor checkpooints, install for tests

* no evals for mamba tests

* handle save_pretrained

* handle unused safetensors arg

setup.py

@@ -51,5 +51,8 @@ setup(
         "deepspeed": [
             "deepspeed",
         ],
+        "mamba-ssm": [
+            "mamba-ssm==1.0.1",
+        ],
     },
 )