3 Commits

Author	SHA1	Message	Date
Thomas Cleberg	e73b8dff8d	Add Support for revision Dataset Parameter to specify reading from Huggingface Dataset Revision (#1912) ... * Add support for `revision` dataset parameter * only use revision on hf hub backed datasets * use revision tied to head * set download to use revision * feat: add config to model validator class * feat: add revision config to RL and tests for it --------- Co-authored-by: Wing Lian <wing.lian@gmail.com> Co-authored-by: NanoCode012 <nano@axolotl.ai>	2024-10-11 13:32:50 -04:00
Frank Ruis	7477a53287	wrap prepared_ds_path in str() to avoid TypeError in fsspec package (#1548) ... * wrap prepared_ds_path in str() to avoid TypeError in fsspec package `fsspec` calls `if "::" in path` on `prepared_ds_path`, which will throw an error if it is a `PosixPath` object. * update test too --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-04-21 19:55:20 -04:00
Keith Stevens	e634118f90	Support loading datasets saved via save_to_disk (#1432) ... * Support loading datasetes saved via save_to_disk * Adding comprehensive unittests * Fix dataset tests due to new hash changes	2024-03-29 00:19:36 -04:00