Tobias 8dfadc2b3c Fix sample packing producing longer sequences than specified by sequence_len (#2332) ...

* Extend MultiPackBatchSampler test to include shorter sequence length and drop long sequences filter

* Fix get_dataset_lengths for datasets that were previously filtered (e.g., with drop_long_seq_in_dataset)

* Update src/axolotl/utils/samplers/utils.py

Fix get_dataset_lengths for datasets that do not have position_ids or length attributes

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

2025-02-19 12:02:35 +07:00

..

cli

Move sweeps code to another module (#2338)

2025-02-17 15:46:04 -05:00

common

GRPO (#2307)

2025-02-13 16:01:01 -05:00

core

add support for include_tokens_per_second in training args (#2269)

2025-02-13 17:39:19 -05:00

integrations

batch add of spectrum snr results (#2320)

2025-02-07 21:33:14 -05:00

kernels

Activation function Triton kernels, LoRA custom autograd functions (#2324)

2025-02-17 14:23:15 -05:00

models

Phi2 multipack (#1173)

2024-01-23 12:54:36 -05:00

monkeypatch

Activation function Triton kernels, LoRA custom autograd functions (#2324)

2025-02-17 14:23:15 -05:00

prompt_strategies

feat: add config for optional parameters in a chat message (#2260)

2025-02-18 09:59:27 +07:00

utils

Fix sample packing producing longer sequences than specified by sequence_len (#2332)

2025-02-19 12:02:35 +07:00

__init__.py

bump dev version (#2342)

2025-02-18 04:30:59 -05:00

convert.py

Lint convert.py

2023-05-31 02:53:53 +09:00

datasets.py

KD Trainer w logprobs (#2303)

2025-01-31 20:18:52 -05:00

evaluate.py

CLI cleanup and documentation (#2244)

2025-01-13 17:55:29 +00:00

logging_config.py

Don't disable existing loggers when configuring axolotl logging (#1395)

2024-03-14 11:05:21 -04:00

prompt_tokenizers.py

KD Trainer w logprobs (#2303)

2025-01-31 20:18:52 -05:00

prompters.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

train.py

Activation function Triton kernels, LoRA custom autograd functions (#2324)

2025-02-17 14:23:15 -05:00