tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Select input_ids explicitly after panda conversion (#2335)

Browse Source 
Without selecting the column, applying `len` counts the whole row as 1 which resulting the total number of the samples instead of the token counts.
This commit is contained in:
Seungduk Kim
2025-02-17 14:07:27 +09:00
committed by GitHub
parent a98526ef78
commit 97a2fa2781
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/axolotl/utils/trainer.py
View File

@@ -396,8 +396,8 @@ def calculate_total_num_steps(cfg, train_dataset, update=True):
): ):
total_num_tokens = np.sum( total_num_tokens = np.sum(
train_dataset.select_columns("input_ids") train_dataset.select_columns("input_ids")
.to_pandas() .to_pandas()["input_ids"]
.apply(lambda x: len(x)) # pylint: disable=unnecessary-lambda .apply(len)
.values .values
) )
LOG.debug(f"total_num_tokens: {total_num_tokens:_}", main_process_only=True) LOG.debug(f"total_num_tokens: {total_num_tokens:_}", main_process_only=True)