18 lines
355 B
Plaintext
18 lines
355 B
Plaintext
# Optimizers
|
|
|
|
## Shampoo
|
|
|
|
```yaml
|
|
optimizer: shampoo
|
|
optim_shampoo_betas: [0.9, 0.999]
|
|
optim_args:
|
|
epsilon: 1e-12
|
|
max_preconditioner_dim: 8192
|
|
precondition_frequency: 100
|
|
use_decoupled_weight_decay: true
|
|
optim_shampoo_grafting_config_type: adam
|
|
optim_shampoo_grafting_config_kwargs:
|
|
beta2: 0.999
|
|
epsilon: 1e-12
|
|
```
|