limit num_proc when saving datasets to disk (#2948) [skip ci]
* limit num_proc when saving datasets to disk * enforce at least 1 in case it rounds down to 0, and sane divisor is at least 8 rows per worker to save * update fixtures with dataset processes since that should never be NoneType * improve reusability for tests
This commit is contained in:
@@ -210,6 +210,7 @@ class TestDeduplicateRLDataset:
|
||||
ALPACA_MESSAGES_CONFIG_REVISION,
|
||||
ALPACA_MESSAGES_CONFIG_REVISION,
|
||||
],
|
||||
"dataset_processes": 4,
|
||||
}
|
||||
)
|
||||
yield fixture
|
||||
|
||||
Reference in New Issue
Block a user