Axolotl
Home
How-To Guides
Debugging
Multipack (Sample Packing)
FDSP + QLoRA
Template-free prompt construction
RLHF (Beta)
NCCL
Mac M-series
Multi Node
Dataset Formats
Pre-training
Instruction Tuning
Conversation
Template-Free
Custom Pre-Tokenized Dataset
Reference
Config options
FAQ
On this page
todo list
things that are known not to work
todo list
[] Validation of parameters for combinations that won’t work
things that are known not to work
FSDP offload and gradient_checkpointing - https://github.com/pytorch/pytorch/issues/82203
adamw_bnb_8bit doesn’t play well with FSDP offload