add liger to readme (#1865)
* add liger to readme * updates from PR feedback
This commit is contained in:
20
README.md
20
README.md
@@ -55,6 +55,7 @@ Features:
|
|||||||
- [FSDP + QLoRA](#fsdp--qlora)
|
- [FSDP + QLoRA](#fsdp--qlora)
|
||||||
- [Weights \& Biases Logging](#weights--biases-logging)
|
- [Weights \& Biases Logging](#weights--biases-logging)
|
||||||
- [Special Tokens](#special-tokens)
|
- [Special Tokens](#special-tokens)
|
||||||
|
- [Liger Kernel](#liger-kernel)
|
||||||
- [Inference Playground](#inference-playground)
|
- [Inference Playground](#inference-playground)
|
||||||
- [Merge LORA to base](#merge-lora-to-base)
|
- [Merge LORA to base](#merge-lora-to-base)
|
||||||
- [Common Errors 🧰](#common-errors-)
|
- [Common Errors 🧰](#common-errors-)
|
||||||
@@ -530,6 +531,25 @@ tokens: # these are delimiters
|
|||||||
|
|
||||||
When you include these tokens in your axolotl config, axolotl adds these tokens to the tokenizer's vocabulary.
|
When you include these tokens in your axolotl config, axolotl adds these tokens to the tokenizer's vocabulary.
|
||||||
|
|
||||||
|
##### Liger Kernel
|
||||||
|
|
||||||
|
Liger Kernel: Efficient Triton Kernels for LLM Training
|
||||||
|
|
||||||
|
https://github.com/linkedin/Liger-Kernel
|
||||||
|
|
||||||
|
Liger (LinkedIn GPU Efficient Runtime) Kernel is a collection of Triton kernels designed specifically for LLM training.
|
||||||
|
It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. The Liger Kernel
|
||||||
|
composes well and is compatible with both FSDP and Deepspeed.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
plugins:
|
||||||
|
- axolotl.integrations.liger.LigerPlugin
|
||||||
|
liger_rope: true
|
||||||
|
liger_rms_norm: true
|
||||||
|
liger_swiglu: true
|
||||||
|
liger_fused_linear_cross_entropy: true
|
||||||
|
```
|
||||||
|
|
||||||
### Inference Playground
|
### Inference Playground
|
||||||
|
|
||||||
Axolotl allows you to load your model in an interactive terminal playground for quick experimentation.
|
Axolotl allows you to load your model in an interactive terminal playground for quick experimentation.
|
||||||
|
|||||||
Reference in New Issue
Block a user