Mixtral fixes 20240124 (#1192) [skip ci]

* mixtral nccl fixes * make sure to patch for z3
2024-01-24 14:59:57 -05:00
parent af0243021c
commit 54d2ac155b
14 changed files with 71 additions and 13 deletions
--- a/examples/mistral/Mistral-7b-example/code.ipynb
+++ b/examples/mistral/Mistral-7b-example/code.ipynb
@@ -942,7 +942,7 @@
    "not only optimizer states but also gradients and parameters across GPUs. The bf16 indicate mixed precision training using bfloat16.\n",
    "For more information read axolotl's readme\n",
    "\"\"\"\n",
-    "!accelerate launch -m axolotl.cli.train /folder/config.yml --deepspeed deepspeed/zero3_bf16.json"
+    "!accelerate launch -m axolotl.cli.train /folder/config.yml --deepspeed deepspeed_configs/zero3_bf16.json"
   ]
  }
 ],