Dan Saunders
fc973f4322
CLI Implementation with Click ( #2107 )
...
* Initial CLI implementation with click package
* Adding fetch command for pulling examples and deepspeed configs
* Automating default options for CliArgs classes
* Mimicking existing no config behavior
* bugfix in choose_config
* Updating fetch to sync instead of re-download
* bugfix
* isort fix
* fixing yaml isort order
* pre-commit fixes
* simplifying argument parsing -- pass through kwargs to do_cli
* make accelerate launch default for non-preprocess commands
* fixing arg handling
* testing None placeholder approach
* removing hacky --use-gpu argument to preprocess command
* Adding brief README documentation for CLI
* remove (New)
* Initial CLI pytest tests
* progress on CLI pytest
* adding inference CLI tests; cleanup
* Refactor train CLI tests to remove various mocking
* Major CLI test refator; adding remaining CLI codepath test coverage
* pytest fixes
* remove integration markers
* parallelizing examples, deepspeed config downloads; rename test to match other CLI test naming
* moving cli pytest due to isolation issues; cleanup
* testing fixes; various minor improvements
* fix
* tests fix
* Update tests/cli/conftest.py
Co-authored-by: Wing Lian <wing.lian@gmail.com >
---------
Co-authored-by: Dan Saunders <dan@axolotl.ai >
Co-authored-by: Wing Lian <wing.lian@gmail.com >
2024-12-05 22:11:48 -05:00
Aman Gupta Karmani
649c19aba3
pretrain: fix with sample_packing=false ( #1841 )
2024-08-21 13:36:51 -04:00
Jan Philipp Harries
2f586d18db
Fix pretraining with iterable/streaming Dataset ( #556 )
...
* return without packing prep/len
* fix remove columns
* fix encode arguments
* add error when max steps not set
* fix test
---------
Co-authored-by: Jan Philipp Harries <jphme@users.noreply.github.com >
2023-09-13 00:16:40 -04:00
Wing Lian
0b4cf5bc8c
workaround for md5 variations ( #533 )
...
* workaround for md5 variations
* refactor the prepared hash too
2023-09-08 16:01:05 -04:00
