957c956f89
* rename jamba example * feat: change readme --------- Co-authored-by: Gal Cohen <galc@ai21.com>
11 lines
318 B
Markdown
11 lines
318 B
Markdown
# Jamba
|
|
|
|
- ✅ qlora w/ deepspeed Zero-2 needs at least 2x GPUs and
|
|
- 35GiB VRAM per GPU w minimal context length
|
|
- 56GiB VRAM per GPU (w multipack enabled)
|
|
- ✅ qlora w/ deepspeed Zero-3 needs at least 2x GPUs and 67GiB VRAM (wtf?)
|
|
- ✅ qlora single-gpu, ~51GiB VRAM
|
|
- ✅ multipack
|
|
- ✅ FSDP
|
|
- ❓ 8-bit LoRA
|