Wing Lian 78e12f8ca5 add basic support for the optimi adamw optimizer (#1727) ...

* add support for optimi_adamw optimizer w kahan summation

* pydantic validator for optimi_adamw

* workaround for setting optimizer for fsdp

* make sure to install optimizer packages

* make sure to have parity for model parameters passed to optimizer

* add smoke test for optimi_adamw optimizer

* don't use foreach optimi by default

2024-07-14 19:12:57 -04:00

..

core

ORPO Trainer replacement (#1551)

2024-04-19 17:25:36 -04:00

e2e

add basic support for the optimi adamw optimizer (#1727)

2024-07-14 19:12:57 -04:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

prompt_strategies

Allow "weight: 0" in messages to mask them (#1703)

2024-06-20 10:05:16 -04:00

utils

Add shifted sparse attention (#973) [skip-ci]

2024-01-18 10:16:07 -05:00

test_data.py

Fix pretraining with iterable/streaming Dataset (#556)

2023-09-13 00:16:40 -04:00

test_datasets.py

wrap prepared_ds_path in str() to avoid TypeError in fsspec package (#1548)

2024-04-21 19:55:20 -04:00

test_dict.py

Pydantic 2.x cfg (#1239)

2024-02-26 12:24:14 -05:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_normalize_config.py

more fixes 20240228 (#1342) [skip ci]

2024-02-28 12:57:45 -05:00

test_packed_batch_sampler.py

Switch to parallel FFD bin packing algorithm. (#1619)

2024-05-23 17:32:14 -04:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

Switch to parallel FFD bin packing algorithm. (#1619)

2024-05-23 17:32:14 -04:00

test_perplexity.py

Phi-3 conversation format, example training script and perplexity metric (#1582)

2024-06-04 16:11:56 -04:00

test_prompt_tokenizers.py

fix broken linting (#1541)

2024-04-19 01:03:04 -04:00

test_prompters.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_schedulers.py

Scheduler implementation of Continual Pre-Training of Large Language Models: How to (re)warm your model? (#1273)

2024-02-12 21:23:28 -08:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation.py

Add KTO support (#1640)

2024-05-20 16:05:16 -04:00