monkeypatch.relora

monkeypatch.relora

Implements the ReLoRA training procedure from https://arxiv.org/abs/2307.05695, minus the initial full fine-tune.

Classes

Name	Description
ReLoRACallback	Callback to merge LoRA weights into the base model and save full-weight checkpoints
ReLoRAScheduler	Wraps another scheduler to apply per-lora-restart learning rate warmups.

ReLoRACallback

monkeypatch.relora.ReLoRACallback(self, cfg)

Callback to merge LoRA weights into the base model and save full-weight checkpoints

ReLoRAScheduler

monkeypatch.relora.ReLoRAScheduler(
    self,
    optimizer,
    inner_schedule,
    relora_steps,
    warmup_steps,
    anneal_steps=1,
    min_lr_scale=0.001,
)

Wraps another scheduler to apply per-lora-restart learning rate warmups.