Tobias 8dfadc2b3c Fix sample packing producing longer sequences than specified by sequence_len (#2332) ...

* Extend MultiPackBatchSampler test to include shorter sequence length and drop long sequences filter

* Fix get_dataset_lengths for datasets that were previously filtered (e.g., with drop_long_seq_in_dataset)

* Update src/axolotl/utils/samplers/utils.py

Fix get_dataset_lengths for datasets that do not have position_ids or length attributes

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

2025-02-19 12:02:35 +07:00

..

cli

[feature] sweeps (#2171)

2025-02-01 21:11:18 -05:00

core

updating to fused (#2293)

2025-01-30 11:45:56 -05:00

e2e

feat: add config for optional parameters in a chat message (#2260)

2025-02-18 09:59:27 +07:00

fixtures

Respect sequence_len in config for type: llama2_chat (#926)

2023-12-12 09:39:22 -08:00

integrations

rename liger test so it properly runs in ci (#2246)

2025-01-09 17:31:43 -05:00

monkeypatch

support for true batches with multipack (#1230)

2024-02-01 10:18:42 -05:00

patched

support for latest transformers release 4.48.1 (#2256)

2025-01-23 21:17:57 -05:00

prompt_strategies

make sure chatml dpo dataset loading works (#2333)

2025-02-18 16:08:40 -05:00

utils

feat: add config for optional parameters in a chat message (#2260)

2025-02-18 09:59:27 +07:00

conftest.py

update upstream HF deps (#2239)

2025-01-09 21:01:59 +00:00

constants.py

Add Exact Deduplication Feature to Preprocessing Pipeline (#2072)

2024-12-02 08:47:10 -05:00

test_data.py

CLI Implementation with Click (#2107)

2024-12-05 22:11:48 -05:00

test_datasets.py

rename references to dpo dataset prep to pref data (#2258)

2025-01-14 22:07:55 -05:00

test_dict.py

Pydantic 2.x cfg (#1239)

2024-02-26 12:24:14 -05:00

test_exact_deduplication.py

rename references to dpo dataset prep to pref data (#2258)

2025-01-14 22:07:55 -05:00

test_expand_mask.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_freeze.py

Train parameters exclusively in specific ranges (#1390)

2024-03-14 11:05:42 -04:00

test_lora.py

assume empty lora dropout means 0.0 and add tests (#2243)

2025-01-13 10:44:11 -05:00

test_normalize_config.py

remove fastchat and sharegpt (#2021)

2024-11-08 13:45:49 -05:00

test_packed_batch_sampler.py

Fix sample packing producing longer sequences than specified by sequence_len (#2332)

2025-02-19 12:02:35 +07:00

test_packed_dataset.py

Attention mask and position id fixes for packing (#285)

2023-08-12 15:14:56 -04:00

test_packed_pretraining.py

Pretrain multipack (#2278)

2025-01-24 12:55:20 -05:00

test_perplexity.py

various tests fixes for flakey tests (#2110)

2024-12-02 17:28:58 -05:00

test_prompt_tokenizers.py

rename liger test so it properly runs in ci (#2246)

2025-01-09 17:31:43 -05:00

test_prompters.py

fix: prompt phi (#1845) [skip ci]

2024-08-22 11:46:57 -04:00

test_schedulers.py

add optimizer step to prevent warning in tests (#1502) [skip ci]

2024-11-19 10:19:03 -05:00

test_tokenizers.py

Support for additional_special_tokens (#1221) [skip ci]

2024-01-31 18:13:13 -05:00

test_validation_dataset.py

feat: add config for optional parameters in a chat message (#2260)

2025-02-18 09:59:27 +07:00