From 61825a464aeaea089448dfafc4714916b87e9207 Mon Sep 17 00:00:00 2001 From: NanoCode012 Date: Fri, 21 Mar 2025 22:59:22 +0700 Subject: [PATCH] chore(doc): add explanation on fsdp_transformer_layer_cls_to_wrap (#2429) [skip ci] --- docs/faq.qmd | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/faq.qmd b/docs/faq.qmd index acec1886e..1ce14681a 100644 --- a/docs/faq.qmd +++ b/docs/faq.qmd @@ -37,6 +37,10 @@ description: Frequently asked questions > A: Yes, since Axolotl is just Python, please see `src/axolotl/cli/main.py` on how each command is called. +**Q: How to know the value to use for `fsdp_transformer_layer_cls_to_wrap`?** + +> A: This is the class name of the transformer layer to wrap with FSDP. For example, for `LlamaForCausalLM`, the value is `LlamaDecoderLayer`. To find this for a specific model, check the model's `PreTrainedModel` definition and look for `_no_split_modules` variable in the `modeling_.py` file within `transformers` library. + ### Chat templates **Q: `jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'content' / 'role' / ____`**