tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

feat: update readme with inference instructions

Browse Source
This commit is contained in:
NanoCode012
2025-02-06 21:29:36 +07:00
parent ebd406af1d
commit 13d458d0ae
1 changed files with 16 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
src/axolotl/integrations/lolcats/README.md
View File

@@ -4,7 +4,17 @@ https://github.com/HazyResearch/lolcats/
### Usage
TODO: Add instruction to install `causal_dot_product`.
Install `causal_dot_product` CUDA kernel (check the README in the `csrc` directory):
```bash
cd src/axolotl/integrations/lolcats/linear_llama/csrc
# Edit `setup.py` to point to the correct CUDA capabilities L40-44
# nano setup.py
# Build the CUDA kernel
python setup.py install
```
Step 1:
@@ -27,4 +37,8 @@ lora_target_modules: ["q_proj", "k_proj", "v_proj", "o_proj"]
# unfrozen_parameters: ['.*feature_map_q.mlp.layer.*', '.*feature_map_k.mlp.layer.*', '.*window_factors.*']
```
`axolotl train config.yaml --base-model={output_dir}/distilled --trust-remote-code`
`axolotl train config.yaml --base-model={output_dir}/distilled --trust-remote-code --learning-rate=0.0001 # --wandb-project="..."`
Step 3: Run inference on the finetuned model
`axolotl inference config.yaml --lora-model-dir="{output_dir}" --trust-remote-code # --prompter="AlpacaPrompter"`