limit num_proc when saving datasets to disk (#2948) [skip ci]
* limit num_proc when saving datasets to disk * enforce at least 1 in case it rounds down to 0, and sane divisor is at least 8 rows per worker to save * update fixtures with dataset processes since that should never be NoneType * improve reusability for tests
This commit is contained in:
@@ -99,6 +99,7 @@ class TestPacking(unittest.TestCase):
|
||||
"type": "alpaca",
|
||||
},
|
||||
],
|
||||
"dataset_processes": 4,
|
||||
"num_epochs": 1,
|
||||
"max_steps": 20,
|
||||
"save_steps": 10,
|
||||
|
||||
Reference in New Issue
Block a user