Dan Saunders 231a67e70b Streaming SFT support (#3101) ...

* working

* fixes

* deprecate --iterable; cleanup

* pretrain_multipack_buffer_size -> streaming_multipack_buffer_size

* improvements

* tests

* remove unused

* docs, examples

* nit

* nit

* add val_set_size validation

* val

* nit

* min

* coderabbito

* cleanup

* nit

* add depr warning, cleanup

* nit

* fix test, fix quarto

* fix

* review comments

* review comments

* fix

2025-09-02 12:08:44 -04:00

..

cli

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

core

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

e2e

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

monkeypatch

use nanmean for loss aggregation (CP fix) (#3033)

2025-08-08 08:15:17 -04:00

patched

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

prompt_strategies

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

utils

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

__init__.py

fix: minor patches for multimodal (#2441)

2025-03-31 13:40:12 +07:00

conftest.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

constants.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

hf_offline_utils.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_chunked_xentropy.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_data.py

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

test_datasets.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_dict.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_exact_deduplication.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_expand_mask.py

adding pre-commit auto-update GH action and bumping plugin versions (#2428)

2025-03-21 11:02:43 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_loaders.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_lora.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_normalize_config.py

FSDP2 fix validation and add tests (#2910)

2025-07-14 09:25:44 -04:00

test_packed_batch_sampler.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_packed_dataset.py

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

test_packed_pretraining.py

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

test_perplexity.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_prompt_tokenizers.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00

test_streaming.py

Streaming SFT support (#3101)

2025-09-02 12:08:44 -04:00

test_tokenizers.py

models.py -> loaders/ module refactor (#2680)

2025-05-23 15:51:11 -04:00

test_train.py

refactor dupes from merge/rebase (#2919) [skip ci]

2025-07-14 10:05:26 -04:00

test_validation_dataset.py

Add ruff, remove black, isort, flake8, pylint (#3092)

2025-08-23 23:37:33 -04:00