Wing Lian 68b227a7d8 Mixtral multipack (#928) ...

* mixtral multipack

* use mixtral model

* sample yml

* calculate cu_seqlens properly

* use updated flash ettention setting

* attn var checks

* force use of flash attention 2 for packing

* lint

* disable future fix for now

* update support table

2023-12-09 21:26:30 -05:00

..

cerebras

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

code-llama

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

falcon

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

gptj

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

jeopardy-bot

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

llama-2

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

mamba

support for mamba (#915)

2023-12-09 12:10:41 -05:00

mistral

Mixtral multipack (#928)

2023-12-09 21:26:30 -05:00

mpt-7b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

openllama-3b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

phi

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

pythia

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

pythia-12b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

qwen

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

redpajama

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

replit-3b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

xgen-7b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00