Wing Lian c7cf3810bd Pretrain transforms (#1261) ...

* wip for pretraining/iterable data with arbitrary prompt strategies

* more fixes, wip

* more fixes for custom pretraining

* iterable ds wrapper not needed

* remove extra features

* chore: lint

* update pretraning example yml

* fix order for partials

* fixup for tests

2024-02-06 00:37:03 -05:00

..

cerebras

Update qlora.yml - remove max_packed_sequence_len (#1210) [skip ci]

2024-01-26 07:43:05 -05:00

code-llama

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

colab-notebooks

add colab example (#1196) [skip ci]

2024-01-24 20:09:09 -05:00

falcon

Falcon embeddings (#1149) [skip docker]

2024-01-22 21:01:42 -05:00

gptj

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

jeopardy-bot

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

llama-2

Peft lotfq (#1222)

2024-01-28 18:50:08 -05:00

mamba

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

mistral

Mixtral fixes 20240124 (#1192) [skip ci]

2024-01-24 14:59:57 -05:00

mpt-7b

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

openllama-3b

Add shifted sparse attention (#973) [skip-ci]

2024-01-18 10:16:07 -05:00

phi

Mixtral fixes 20240124 (#1192) [skip ci]

2024-01-24 14:59:57 -05:00

pythia

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

pythia-12b

Feat(wandb): Refactor to be more flexible (#767)

2023-12-04 22:17:25 +09:00

qwen

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

redpajama

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

replit-3b

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

tiny-llama

Pretrain transforms (#1261)

2024-02-06 00:37:03 -05:00

xgen-7b

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

yi-34B-chat

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00