3 Commits

Author	SHA1	Message	Date
Aman Gupta Karmani	649c19aba3	pretrain: fix with sample_packing=false (#1841)	2024-08-21 13:36:51 -04:00
Jan Philipp Harries	2f586d18db	Fix pretraining with iterable/streaming Dataset (#556) ... * return without packing prep/len * fix remove columns * fix encode arguments * add error when max steps not set * fix test --------- Co-authored-by: Jan Philipp Harries <jphme@users.noreply.github.com>	2023-09-13 00:16:40 -04:00
Wing Lian	0b4cf5bc8c	workaround for md5 variations (#533) ... * workaround for md5 variations * refactor the prepared hash too	2023-09-08 16:01:05 -04:00