tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

allow for different sequence_len for evaluations (#2836) [skip ci]

Browse Source 
* allow for different sequence_len for evaluations

* reversed 🤦

* add more information to filter msg
This commit is contained in:
Wing Lian
2025-06-27 11:02:51 -04:00
committed by GitHub
parent d8280d45c1
commit 0a7a216b60
4 changed files with 17 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tests/test_packed_batch_sampler.py
View File

@@ -70,7 +70,7 @@ class TestBatchedSamplerPacking:
)
train_dataset = concatenate_datasets([dataset_wrapper])
train_dataset = drop_long_seq_in_dataset(train_dataset, cfg)
train_dataset = drop_long_seq_in_dataset(train_dataset, cfg.sequence_len, cfg)
lengths = get_dataset_lengths(train_dataset)
batch_sampler = MultipackBatchSampler(