Axolotl CLI Documentation
+The Axolotl CLI provides a streamlined interface for training and fine-tuning large language models. This guide covers the CLI commands, their usage, and common examples.
+Table of Contents
+-
+
- Basic Commands +
- Command Reference
+
-
+
- fetch +
- preprocess +
- train +
- inference +
- merge-lora +
- merge-sharded-fsdp-weights +
- evaluate +
- lm-eval +
+ - Legacy CLI Usage +
- Remote Compute with Modal Cloud
+
-
+
- Cloud Configuration +
- Running on Modal Cloud +
- Cloud Configuration Options +
+
Basic Commands
+All Axolotl commands follow this general structure:
+axolotl <command> [config.yml] [options]The config file can be local or a URL to a raw YAML file.
+Command Reference
+fetch
+Downloads example configurations and deepspeed configs to your local machine.
+# Get example YAML files
+axolotl fetch examples
+
+# Get deepspeed config files
+axolotl fetch deepspeed_configs
+
+# Specify custom destination
+axolotl fetch examples --dest path/to/folderpreprocess
+Preprocesses and tokenizes your dataset before training. This is recommended for large datasets.
+# Basic preprocessing
+axolotl preprocess config.yml
+
+# Preprocessing with one GPU
+CUDA_VISIBLE_DEVICES="0" axolotl preprocess config.yml
+
+# Debug mode to see processed examples
+axolotl preprocess config.yml --debug
+
+# Debug with limited examples
+axolotl preprocess config.yml --debug --debug-num-examples 5Configuration options:
+dataset_prepared_path: Local folder for saving preprocessed data
+push_dataset_to_hub: HuggingFace repo to push preprocessed data (optional)train
+Trains or fine-tunes a model using the configuration specified in your YAML file.
+# Basic training
+axolotl train config.yml
+
+# Train and set/override specific options
+axolotl train config.yml \
+ --learning-rate 1e-4 \
+ --micro-batch-size 2 \
+ --num-epochs 3
+
+# Training without accelerate
+axolotl train config.yml --no-accelerate
+
+# Resume training from checkpoint
+axolotl train config.yml --resume-from-checkpoint path/to/checkpointinference
+Runs inference using your trained model in either CLI or Gradio interface mode.
+# CLI inference with LoRA
+axolotl inference config.yml --lora-model-dir="./outputs/lora-out"
+
+# CLI inference with full model
+axolotl inference config.yml --base-model="./completed-model"
+
+# Gradio web interface
+axolotl inference config.yml --gradio \
+ --lora-model-dir="./outputs/lora-out"
+
+# Inference with input from file
+cat prompt.txt | axolotl inference config.yml \
+ --base-model="./completed-model"merge-lora
+Merges trained LoRA adapters into the base model.
+# Basic merge
+axolotl merge-lora config.yml
+
+# Specify LoRA directory (usually used with checkpoints)
+axolotl merge-lora config.yml --lora-model-dir="./lora-output/checkpoint-100"
+
+# Merge using CPU (if out of GPU memory)
+CUDA_VISIBLE_DEVICES="" axolotl merge-lora config.ymlConfiguration options:
+gpu_memory_limit: Limit GPU memory usage
+lora_on_cpu: Load LoRA weights on CPUmerge-sharded-fsdp-weights
+Merges sharded FSDP model checkpoints into a single combined checkpoint.
+# Basic merge
+axolotl merge-sharded-fsdp-weights config.ymlevaluate
+Evaluates a model’s performance using metrics specified in the config.
+# Basic evaluation
+axolotl evaluate config.ymllm-eval
+Runs LM Evaluation Harness on your model.
+# Basic evaluation
+axolotl lm-eval config.yml
+
+# Evaluate specific tasks
+axolotl lm-eval config.yml --tasks arc_challenge,hellaswagConfiguration options:
+lm_eval_tasks: List of tasks to evaluate
+lm_eval_batch_size: Batch size for evaluation
+output_dir: Directory to save evaluation resultsLegacy CLI Usage
+While the new Click-based CLI is preferred, Axolotl still supports the legacy module-based CLI:
+# Preprocess
+python -m axolotl.cli.preprocess config.yml
+
+# Train
+accelerate launch -m axolotl.cli.train config.yml
+
+# Inference
+accelerate launch -m axolotl.cli.inference config.yml \
+ --lora_model_dir="./outputs/lora-out"
+
+# Gradio interface
+accelerate launch -m axolotl.cli.inference config.yml \
+ --lora_model_dir="./outputs/lora-out" --gradioRemote Compute with Modal Cloud
+Axolotl supports running training and inference workloads on Modal cloud infrastructure. This is configured using a cloud YAML file alongside your regular Axolotl config.
+Cloud Configuration
+Create a cloud config YAML with your Modal settings:
+# cloud_config.yml
+provider: modal
+gpu: a100 # Supported: l40s, a100-40gb, a100-80gb, a10g, h100, t4, l4
+gpu_count: 1 # Number of GPUs to use
+timeout: 86400 # Maximum runtime in seconds (24 hours)
+branch: main # Git branch to use (optional)
+
+volumes: # Persistent storage volumes
+ - name: axolotl-cache
+ mount: /workspace/cache
+
+env: # Environment variables
+ - WANDB_API_KEY
+ - HF_TOKENRunning on Modal Cloud
+Commands that support the –cloud flag:
+# Preprocess on cloud
+axolotl preprocess config.yml --cloud cloud_config.yml
+
+# Train on cloud
+axolotl train config.yml --cloud cloud_config.yml
+
+# Train without accelerate on cloud
+axolotl train config.yml --cloud cloud_config.yml --no-accelerate
+
+# Run lm-eval on cloud
+axolotl lm-eval config.yml --cloud cloud_config.ymlCloud Configuration Options
+provider: compute provider, currently only `modal` is supported
+gpu: GPU type to use
+gpu_count: Number of GPUs (default: 1)
+memory: RAM in GB (default: 128)
+timeout: Maximum runtime in seconds
+timeout_preprocess: Preprocessing timeout
+branch: Git branch to use
+docker_tag: Custom Docker image tag
+volumes: List of persistent storage volumes
+env: Environment variables to pass
+secrets: Secrets to inject