Wing Lian 8c2e05ade3 relora: magnitude pruning of the optimizer (#1245) ...

* magnitude pruning of the optimizer

* add alpaca chat template and fix relora patch

* fix handling of lora adapter for relora

* fix merge and save call

* fixes for 8-bit lora merge

* save intermediate checkpoint adapters

* auto merge

* fix eval check

* handle relora annealing

* fix anneal step logic

* chore: lint

* misx fix

* fix types

* Update tests/e2e/test_relora_llama.py

* check for safetensors saved from relora

2024-02-06 00:35:30 -05:00

..

core

add gptneox embeddings, fix phi2 inputs, also fix the casting (#1083)

2024-01-10 22:32:43 -05:00

e2e

relora: magnitude pruning of the optimizer (#1245)

2024-02-06 00:35:30 -05:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

prompt_strategies

Feat/chatml add system message (#1117)

2024-01-25 08:24:27 +01:00

utils

Add shifted sparse attention (#973) [skip-ci]

2024-01-18 10:16:07 -05:00

test_data.py

Fix pretraining with iterable/streaming Dataset (#556)

2023-09-13 00:16:40 -04:00

test_dict.py

fix DefaultDict.__or__

2023-08-13 01:15:50 +00:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_normalize_config.py

set fp16 to false if bf16, update bf16: auto in example YAMLs (#1122) [skip ci]

2024-01-22 18:44:01 -05:00

test_packed_batch_sampler.py

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

test_prompt_tokenizers.py

fix mistral prompt assembly (#982)

2023-12-21 08:00:55 -08:00

test_prompters.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation.py

Peft lotfq (#1222)

2024-01-28 18:50:08 -05:00