tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Fix for setting adam_beta3 and adam_epsilon2 for CAME Optimizer (#2654) [skip ci]

Browse Source 
* make setting `adam_beta3` and `adam_epsilon2` work correctly

* update config docs so users know args are specific to CAME optim

---------

Co-authored-by: Wing Lian <wing@axolotl.ai>
This commit is contained in:
xzuyn
2025-05-16 15:46:50 -04:00
committed by GitHub
parent 288653adb6
commit 6cb07b9d12
3 changed files with 20 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/config.qmd
View File

@@ -633,7 +633,9 @@ weight_decay:
# adamw hyperparams
adam_beta1:
adam_beta2:
adam_beta3: # only used for CAME Optimizer
adam_epsilon:
adam_epsilon2: # only used for CAME Optimizer
# Gradient clipping max norm
max_grad_norm: