Dan Saunders fc973f4322 CLI Implementation with Click (#2107) ...

* Initial CLI implementation with click package

* Adding fetch command for pulling examples and deepspeed configs

* Automating default options for CliArgs classes

* Mimicking existing no config behavior

* bugfix in choose_config

* Updating fetch to sync instead of re-download

* bugfix

* isort fix

* fixing yaml isort order

* pre-commit fixes

* simplifying argument parsing -- pass through kwargs to do_cli

* make accelerate launch default for non-preprocess commands

* fixing arg handling

* testing None placeholder approach

* removing hacky --use-gpu argument to preprocess command

* Adding brief README documentation for CLI

* remove (New)

* Initial CLI pytest tests

* progress on CLI pytest

* adding inference CLI tests; cleanup

* Refactor train CLI tests to remove various mocking

* Major CLI test refator; adding remaining CLI codepath test coverage

* pytest fixes

* remove integration markers

* parallelizing examples, deepspeed config downloads; rename test to match other CLI test naming

* moving cli pytest due to isolation issues; cleanup

* testing fixes; various minor improvements

* fix

* tests fix

* Update tests/cli/conftest.py

Co-authored-by: Wing Lian <wing.lian@gmail.com>

---------

Co-authored-by: Dan Saunders <dan@axolotl.ai>
Co-authored-by: Wing Lian <wing.lian@gmail.com>

2024-12-05 22:11:48 -05:00

..

cli

CLI Implementation with Click (#2107)

2024-12-05 22:11:48 -05:00

core

various tests fixes for flakey tests (#2110)

2024-12-02 17:28:58 -05:00

e2e

replace tensorboard checks with helper function (#2120) [skip ci]

2024-12-03 21:06:20 -05:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

upgrade liger to 0.4.0 (#1973)

2024-11-07 12:53:34 -05:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

patched

CLI Implementation with Click (#2107)

2024-12-05 22:11:48 -05:00

prompt_strategies

add missing fixture decorator for predownload dataset (#2117) [skip ci]

2024-12-03 18:08:46 -05:00

utils

Refactor func load_model to class ModelLoader (#1909)

2024-10-25 09:06:56 -04:00

conftest.py

add missing fixture decorator for predownload dataset (#2117) [skip ci]

2024-12-03 18:08:46 -05:00

constants.py

Add Exact Deduplication Feature to Preprocessing Pipeline (#2072)

2024-12-02 08:47:10 -05:00

test_data.py

CLI Implementation with Click (#2107)

2024-12-05 22:11:48 -05:00

test_datasets.py

various tests fixes for flakey tests (#2110)

2024-12-02 17:28:58 -05:00

test_dict.py

Pydantic 2.x cfg (#1239)

2024-02-26 12:24:14 -05:00

test_exact_deduplication.py

Add Exact Deduplication Feature to Preprocessing Pipeline (#2072)

2024-12-02 08:47:10 -05:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_normalize_config.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

test_packed_batch_sampler.py

Switch to parallel FFD bin packing algorithm. (#1619)

2024-05-23 17:32:14 -04:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

retry flaky test_packing_stream_dataset test that timesout on read (#2052) [skip ci]

2024-11-13 13:16:16 -05:00

test_perplexity.py

various tests fixes for flakey tests (#2110)

2024-12-02 17:28:58 -05:00

test_prompt_tokenizers.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

add optimizer step to prevent warning in tests (#1502) [skip ci]

2024-11-19 10:19:03 -05:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation_dataset.py

Check torch version for ADOPT optimizer + integrating new ADOPT updates (#2104)

2024-12-02 20:15:39 -05:00