add liger to readme (#1865)

* add liger to readme

* updates from PR feedback
This commit is contained in:
Wing Lian
2024-08-23 14:34:03 -04:00
committed by GitHub
parent da0d581a8c
commit 810ecd4e81

View File

@@ -55,6 +55,7 @@ Features:
- [FSDP + QLoRA](#fsdp--qlora) - [FSDP + QLoRA](#fsdp--qlora)
- [Weights \& Biases Logging](#weights--biases-logging) - [Weights \& Biases Logging](#weights--biases-logging)
- [Special Tokens](#special-tokens) - [Special Tokens](#special-tokens)
- [Liger Kernel](#liger-kernel)
- [Inference Playground](#inference-playground) - [Inference Playground](#inference-playground)
- [Merge LORA to base](#merge-lora-to-base) - [Merge LORA to base](#merge-lora-to-base)
- [Common Errors 🧰](#common-errors-) - [Common Errors 🧰](#common-errors-)
@@ -530,6 +531,25 @@ tokens: # these are delimiters
When you include these tokens in your axolotl config, axolotl adds these tokens to the tokenizer's vocabulary. When you include these tokens in your axolotl config, axolotl adds these tokens to the tokenizer's vocabulary.
##### Liger Kernel
Liger Kernel: Efficient Triton Kernels for LLM Training
https://github.com/linkedin/Liger-Kernel
Liger (LinkedIn GPU Efficient Runtime) Kernel is a collection of Triton kernels designed specifically for LLM training.
It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. The Liger Kernel
composes well and is compatible with both FSDP and Deepspeed.
```yaml
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true
```
### Inference Playground ### Inference Playground
Axolotl allows you to load your model in an interactive terminal playground for quick experimentation. Axolotl allows you to load your model in an interactive terminal playground for quick experimentation.