Tensor parallel w DeepSpeed AutoTP (#2574)

* support for deepspeed autotup

* bump to latest deepspeed that supports deepcompile too

* add deepcompile support too

* fix total steps calculation for TP

* setup fixture for tp

* update ds config to ensure weights are gathered for checkpoint

* fix duplicate validation names

* chore: lint

tests/core/test_builders.py

@@ -65,6 +65,7 @@ def fixture_base_cfg():
             "dataloader_pin_memory": True,
             "dataloader_prefetch_factor": 2,
             "sequence_parallel_degree": 1,
+            "tensor_parallel_size": 1,
             # Dtype
             "fp16": False,
             "bf16": False,