feat:add support dataset_num_processes (#3129) [skip ci]

* feat:add support dataset_num_processes * chore * required changes * requested chnages * required chnages * required changes * required changes * elif get_default_process_count() * add:del data * Update cicd/Dockerfile.jinja Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> * Update cicd/single_gpu.py Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> --------- Co-authored-by: salman <salman.mohammadi@outlook.com> Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>
2025-10-13 15:48:12 +05:30
parent 143dea4753
commit cd856b45b1
18 changed files with 57 additions and 34 deletions
--- a/tests/test_packed_dataset.py
+++ b/tests/test_packed_dataset.py
@@ -55,7 +55,7 @@ class TestPacking(unittest.TestCase):
                        "type": "alpaca",
                    },
                ],
-                "dataset_processes": 4,
+                "dataset_num_proc": 4,
                "num_epochs": 1,
                "max_steps": 20,
                "save_steps": 10,