# monkeypatch.relora

`monkeypatch.relora`

Implements the ReLoRA training procedure from https://arxiv.org/abs/2307.05695, minus the initial full fine-tune.

## Classes

Name | Description
--- | ---
ReLoRACallback | Callback to merge LoRA weights into the base model and save full-weight checkpoints
ReLoRAScheduler | Wraps another scheduler to apply per-lora-restart learning rate warmups.

### ReLoRACallback

    monkeypatch.relora.ReLoRACallback(cfg)

Callback to merge LoRA weights into the base model and save full-weight checkpoints

### ReLoRAScheduler

    monkeypatch.relora.ReLoRAScheduler(
        optimizer,
        inner_schedule,
        relora_steps,
        warmup_steps,
        anneal_steps=1,
        min_lr_scale=0.001,
    )

Wraps another scheduler to apply per-lora-restart learning rate warmups.