Wing Lian 1cf7075d18 support seperate lr for embeddings, similar to loraplus (#1910) [skip ci] ...

* support seperate lr for embeddings, similar to loraplus

* add test case for train w lr embedding scale

* use kwarg for optimizer

* make sure to handle the optimizer creation

* make sure to handle for embedding_lr too

* use smollm for e2e, check for embeddings lr first before wdecay

2024-11-29 20:38:20 -05:00

..

core

wip add new proposed message structure (#1904)

2024-10-13 12:15:18 -04:00

e2e

support seperate lr for embeddings, similar to loraplus (#1910) [skip ci]

2024-11-29 20:38:20 -05:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

upgrade liger to 0.4.0 (#1973)

2024-11-07 12:53:34 -05:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

prompt_strategies

move shared pytest conftest to top level tests (#2099) [skip ci]

2024-11-22 15:05:42 -05:00

utils

Refactor func load_model to class ModelLoader (#1909)

2024-10-25 09:06:56 -04:00

conftest.py

add finetome dataset to fixtures, check eval_loss in test (#2106) [skip ci]

2024-11-29 20:37:32 -05:00

test_data.py

pretrain: fix with sample_packing=false (#1841)

2024-08-21 13:36:51 -04:00

test_datasets.py

fix: loading locally downloaded dataset (#2056) [skip ci]

2024-11-15 20:35:26 -05:00

test_dict.py

Pydantic 2.x cfg (#1239)

2024-02-26 12:24:14 -05:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_normalize_config.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

test_packed_batch_sampler.py

Switch to parallel FFD bin packing algorithm. (#1619)

2024-05-23 17:32:14 -04:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

retry flaky test_packing_stream_dataset test that timesout on read (#2052) [skip ci]

2024-11-13 13:16:16 -05:00

test_perplexity.py

fix: ds3 and fsdp lmbench eval (#2102) [ski[p ci]

2024-11-29 20:37:49 -05:00

test_prompt_tokenizers.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

add optimizer step to prevent warning in tests (#1502) [skip ci]

2024-11-19 10:19:03 -05:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation_dataset.py

fix: handle sharegpt dataset missing (#2035)

2024-11-12 12:51:37 +07:00

test_validation.py

fix None-type not iterable error when deepspeed is left blank w/ use_… (#2087)

2024-11-21 13:36:51 -05:00