Files

NanoCode012 a1da39cd48 Feat(wandb): Refactor to be more flexible (#767 )

* Feat: Update to handle wandb env better

* chore: rename wandb_run_id to wandb_name

* feat: add new recommendation and update config

* fix: indent and pop disabled env if project passed

* feat: test env set for wandb and recommendation

* feat: update to use wandb_name and allow id

* chore: add info to readme

2023-12-04 22:17:25 +09:00

Feat(wandb): Refactor to be more flexible (#767 )

2023-12-04 22:17:25 +09:00

13b

Feat(wandb): Refactor to be more flexible (#767 )

2023-12-04 22:17:25 +09:00

34b

Feat(wandb): Refactor to be more flexible (#767 )

2023-12-04 22:17:25 +09:00

README.md

Feat(cfg): Add code-llama configs for all sizes (#479 )

2023-08-27 10:20:17 +09:00

README.md

Overview

This is an example of CodeLLaMA configuration for 7b, 13b and 34b.

The 7b variant fits on any 24GB VRAM GPU and will take up about 17 GB of VRAM during training if using qlora and 20 GB if using lora. On a RTX 4090 it trains 3 epochs of the default dataset in about 15 minutes.

The 13b variant will fit if you change these settings to these values: gradient_accumulation_steps: 2 micro_batch_size: 1

The 34b variant does not fit on 24GB of VRAM - you will need something with +40 gb VRAM that also supports flash attention v2 - A6000 or A100 are good choices.

accelerate launch scripts/finetune.py examples/code-llama/[MODEL_SIZE]/qlora.yml

accelerate launch scripts/finetune.py examples/code-llama/[MODEL_SIZE]/lora.yml