tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Fix: ensure correct handling of val_set_size as float or int (#1655)

Browse Source 
* Fix: ensure correct handling of val_set_size as float or int

* chore: lint

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>
This commit is contained in:
Davide Caroselli
2024-05-28 18:00:32 +02:00
committed by GitHub
parent f5febc729a
commit 6a5a725f10
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/axolotl/utils/data/sft.py
View File

@@ -494,7 +494,9 @@ def load_prepare_datasets(
test_fingerprint = md5(to_hash_test) test_fingerprint = md5(to_hash_test)
dataset = dataset.train_test_split( dataset = dataset.train_test_split(
test_size=cfg.val_set_size, test_size=int(cfg.val_set_size)
if cfg.val_set_size == int(cfg.val_set_size)
else cfg.val_set_size,
shuffle=False, shuffle=False,
seed=cfg.seed or 42, seed=cfg.seed or 42,
train_new_fingerprint=train_fingerprint, train_new_fingerprint=train_fingerprint,