From 22d4a838dc27f19c5f08072dcaefeae3003d4028 Mon Sep 17 00:00:00 2001 From: NanoCode012 Date: Tue, 8 Jul 2025 01:13:37 +0700 Subject: [PATCH] feat(doc): add vllm and fa2 incompat error to faq (#2877) --- docs/faq.qmd | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/faq.qmd b/docs/faq.qmd index 59b06becd..57c2c81e6 100644 --- a/docs/faq.qmd +++ b/docs/faq.qmd @@ -55,6 +55,14 @@ description: Frequently asked questions > A: This is because you may be using `preprocess` CLI with `pretraining_dataset:` or `skip_prepare_dataset: true` respectively. Please use `axolotl train` CLI directly instead as these datasets are prepared on demand. +**Q: vLLM is not working with Axolotl** + +> A: We currently recommend torch 2.6.0 for use with `vllm`. Please ensure you use the right version. For Docker, please use the `main-py3.11-cu124-2.6.0` tag. + +**Q: FA2 2.8.0 `undefined symbol` runtime error on CUDA 12.4** + +> A: There seems to be a wheel issue with FA2 2.8.0 on CUDA 12.4. Try CUDA 12.6 instead or downgrade to FA2 2.7.4. Please refer to the upstream issue: https://github.com/Dao-AILab/flash-attention/issues/1717. + ### Chat templates **Q: `jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'content' / 'role' / ____`**