NanoCode012 5c7e89105d Fix: modelloader handling of model_kwargs load_in*bit (#1999) ...

* fix: load_in_*bit not properly read

* fix: load_*bit check

* fix: typo

* refactor: load * bit handling

* feat: add test dpo lora multi-gpu

* fix: turn off sample packing for dpo

* fix: missing warmup_steps

* fix: test to load in 8bit for lora

* skip 8bit lora on h100, add 4bit lora on h100 to multi gpu tests

* chore: reduce max_steps

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>

2024-10-30 14:41:34 -04:00

..

core

wip add new proposed message structure (#1904)

2024-10-13 12:15:18 -04:00

e2e

Fix: modelloader handling of model_kwargs load_in*bit (#1999)

2024-10-30 14:41:34 -04:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

prompt_strategies

Feat: Add support for tokenizer’s or custom jinja chat_template (#1970)

2024-10-29 10:14:51 +07:00

utils

Refactor func load_model to class ModelLoader (#1909)

2024-10-25 09:06:56 -04:00

test_data.py

pretrain: fix with sample_packing=false (#1841)

2024-08-21 13:36:51 -04:00

test_datasets.py

remove skipped test (#2002)

2024-10-30 12:27:04 -04:00

test_dict.py

Pydantic 2.x cfg (#1239)

2024-02-26 12:24:14 -05:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_normalize_config.py

more fixes 20240228 (#1342) [skip ci]

2024-02-28 12:57:45 -05:00

test_packed_batch_sampler.py

Switch to parallel FFD bin packing algorithm. (#1619)

2024-05-23 17:32:14 -04:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

fixes to accelerator so that iterable pretraining datasets work (#1759)

2024-07-17 10:58:38 -04:00

test_perplexity.py

Phi-3 conversation format, example training script and perplexity metric (#1582)

2024-06-04 16:11:56 -04:00

test_prompt_tokenizers.py

fix broken linting (#1541)

2024-04-19 01:03:04 -04:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

Scheduler implementation of Continual Pre-Training of Large Language Models: How to (re)warm your model? (#1273)

2024-02-12 21:23:28 -08:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation_dataset.py

Feat: Add support for tokenizer’s or custom jinja chat_template (#1970)

2024-10-29 10:14:51 +07:00

test_validation.py

Log checkpoints as mlflow artifacts (#1976)

2024-10-22 08:52:21 -04:00