Wing Lian 1cf7075d18 support seperate lr for embeddings, similar to loraplus (#1910) [skip ci] ...

* support seperate lr for embeddings, similar to loraplus

* add test case for train w lr embedding scale

* use kwarg for optimizer

* make sure to handle the optimizer creation

* make sure to handle for embedding_lr too

* use smollm for e2e, check for embeddings lr first before wdecay

2024-11-29 20:38:20 -05:00

4.1 KiB

Raw Blame History

View Raw