b194e17c28
* feat: add config for optional parameters in a chat message * chore: cleanup * chore: fix nits and add light docs * docs: update docs/dataset-formats/conversation.qmd Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> * feat: configurable message mappings, jinja template analyzer * chore: handle bradley terry * docs: update docs * refactor: change order of mappings, improve message transform * refactor: make chat awware of property mappings * chore: remove .python-version * chore: revert change * chore: add dataset validation to tests where appropriate * chore: add dataset validation to tests where appropriate * chore: clean up handling of ds_cfg * chore: recursively serialize config * make sure to use the return value from validate_config * DefaultDict pickle/unpickle fix * fix super call for override * refactor: message fields * chore: empty commit * tests: validate config before using * chore: add config validation to all e2e tests * chore: add unneeded logging * chore: add missed config validation * chore: pass field_messages to prompter * test: fix borked test * chore: remove uninteded file * chore: add deprecation warning and update chat_datasets script * chore: lint * refactor: message fields * feat: update axolotlinputconfig and test_models - add configdict import in axolotl/utils/config/models/input/v0_4_1/__init__.py - remove unnecessary line breaks in sftdataset, dpodataset, ktodataset, stepwisesuperviseddataset classes - update model_dump method in axolotlinputconfig to exclude none values - correct typo in test_models.py comment * feat: simplify dpodataset and ktodataset classes in config models removed several optional fields from dpodataset and ktodataset classes in axolotl/utils/config/models/input/v0_4_1. this simplifies the configuration subsets for these datasets. * feat: improve readability and structure in dataset configuration models this commit enhances the readability and structure of the dataset configuration models in the `axolotl/utils/config/models/input/v0_4_1` module. it removes unused `configdict` import and adds line breaks to separate class definitions for better clarity. additionally, a minor documentation fix is included to ensure a newline at the end of the `stepwise_supervised.qmd` file. * feat: change log level from info to debug in chattemplatestrategy * feat(prompt_strategies): refactor chattemplateprompter and chattemplatestrategy - Make `chat_template` a required parameter in `ChatTemplatePrompter` constructor - Add default value for `message_property_mappings` in `ChatTemplatePrompter` constructor - Add `messages_array_name` property to `ChatTemplatePrompter` - Change `processor` type to Optional in `ChatTemplatePrompter` - Add TypeError check for `processor` in `ChatTemplatePrompter.build_prompt` - Remove `_messages` property from `ChatTemplateStrategy` - Make `prompter` a required parameter and add type hint in `ChatTemplateStrategy` constructor - Remove `messages` getter and setter from `ChatTemplateStrategy` - Use `prompter.messages_array_name` in `ChatTemplateStrategy.get_conversation_thread` - Remove condition to set `messages` field in `load` function * feat(tests/utils): ignore type check in load_model call in test_models.py * feat: improve type handling and test structure in chat templates - Add return type hint for `get_chat_template` function in `chat_templates.py` - Remove unnecessary assignment of `strategy.messages` in several test cases - Add `messages_array_name` parameter to various test configurations in `test_chat_templates.py` and `test_chat_templates_advanced.py` - Remove redundant `strategy.messages` assignment in `test_chat_templates_advanced.py` * feat(axolotl): enhance chat strategy with datasetconfig support This commit introduces support for DatasetConfig in the ChatTemplateStrategy. It also refines the strategy loader to handle different types of ds_cfg inputs and improves the clarity of the code by formatting and reordering. The key changes include: - Importing Union from typing and BaseModel from pydantic. - Adding DatasetConfig as an optional type for ds_cfg in StrategyLoader. - Adjusting the handling of ds_cfg in StrategyLoader to account for BaseModel instances. - Refactoring the prompter_params and strategy_params for better readability. - Changing the reference from prompt[self.messages] to prompt[self.prompter.messages_array_name] in the is_prompt_batched method. * feat: update message handling in btchattemplatestrategy * Replace `self.messages` with direct string references to "chosen_messages" and "rejected_messages" * Append system, user, and assistant content directly to "chosen_messages" and "rejected_messages" * Add a new attribute "messages_array_name" to the `load` function parameters * Remove the conditional attribute assignment for "field_messages" in the `load` function * feat: add config validation in test_kd.py - Import `validate_config` from `axolotl.utils.config` - Validate the configuration in `test_llama_kd` and another function in `TestKnowledgeDistillation` class * feat: enhance config validation and capabilities handling * Import `EnvCapabilities` and `GPUCapabilities` from `axolotl.utils.config.models.internals` * Update `validate_config` function to create `KTODataset` and `SFTDataset` instances using `dict(ds_cfg)` * Replace `capabilities` and `env_capabilities` with instances of `GPUCapabilities` and `EnvCapabilities` respectively in `AxolotlConfigWCapabilities` model dump * feat: update config validation in axolotl utils - Remove import of `EnvCapabilities` and `GPUCapabilities` from `axolotl.utils.config.models.internals` - Update `validate_config` function to use `capabilities` and `env_capabilities` directly instead of creating new instances of `GPUCapabilities` and `EnvCapabilities` * feat: refactor strategyloader in chat_template.py - Extracted the creation of strategy parameters into a separate function, `_get_strategy_params(cfg, dataset_config)` - Created a new function, `_get_strategy_cls()`, to obtain the strategy class - Replaced `ChatTemplateStrategy` with `strategy_cls` for strategy instantiation * trigger CI * chore: revert dataset config changes for kto/dpo * subject: refactor: rename 'messages_array_name' to 'field_messages' Body: - Renamed 'messages_array_name' to 'field_messages' in 'ChatTemplatePrompter' class and its usages in 'chat_template.py' - Updated 'load' function in 'bradley_terry/chat_template.py' to reflect the change - Adjusted 'get_chat_template_msg_variables' and 'get_message_vars' methods in 'jinja_template_analyzer.py' to use the new variable name - Modified 'StrategyLoader' in 'chat_template.py' to use 'field_messages' - Updated tests in 'test_chat_templates.py' and 'test_chat_templates_advanced.py' to use 'field_messages' instead of 'messages_array_name' * feat: refactor prompt strategies and update config models * Remove redundant 'return None' in `axolotl/prompt_strategies/__init__.py` * Simplify message handling in `axolotl/prompt_strategies/bradley_terry/chat_template.py` by using a single 'messages' list instead of separate 'chosen_messages' and 'rejected_messages' lists * Update default 'message_property_mappings' in `axolotl/prompt_strategies/bradley_terry/chat_template.py` * Add 'field_messages' field to `axolotl/utils/config/models/input/v0_4_1/__init__.py` configuration model * chore: remove unused input * chore: remove redundant type ignore * fix: remove old configs and update examples * fix: type check * fix: remove loading old config in ChatMessage * fix: update faq with potential new undefinederror * fix: add debug if property mapped is not found * chore: improve explanation for unmapped properties * fix: update docs with new config * chore: add note for deprecation config and del old config from dict --------- Co-authored-by: NanoCode012 <kevinvong@rocketmail.com> Co-authored-by: Wing Lian <wing@axolotl.ai> Co-authored-by: NanoCode012 <nano@axolotl.ai>
324 lines
9.8 KiB
Python
324 lines
9.8 KiB
Python
"""Module for testing the validation module for the dataset config"""
|
|
|
|
import warnings
|
|
from typing import Optional
|
|
|
|
import pytest
|
|
|
|
from axolotl.utils.config import validate_config
|
|
from axolotl.utils.config.models.input.v0_4_1 import ChatTemplate
|
|
from axolotl.utils.dict import DictDefault
|
|
|
|
warnings.filterwarnings("error")
|
|
|
|
|
|
@pytest.fixture(name="minimal_cfg")
|
|
def fixture_cfg():
|
|
return DictDefault(
|
|
{
|
|
"base_model": "TinyLlama/TinyLlama-1.1B-Chat-v0.6",
|
|
"learning_rate": 0.000001,
|
|
"micro_batch_size": 1,
|
|
"gradient_accumulation_steps": 1,
|
|
}
|
|
)
|
|
|
|
|
|
# pylint: disable=too-many-public-methods (duplicate-code)
|
|
class BaseValidation:
|
|
"""
|
|
Base validation module to setup the log capture
|
|
"""
|
|
|
|
_caplog: Optional[pytest.LogCaptureFixture] = None
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def inject_fixtures(self, caplog):
|
|
self._caplog = caplog
|
|
|
|
|
|
class TestValidationCheckDatasetConfig(BaseValidation):
|
|
"""
|
|
Test the validation for the dataset config to ensure no correct parameters are dropped
|
|
"""
|
|
|
|
def test_dataset_config_no_drop_param(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"datasets": [
|
|
{
|
|
"path": "mhenrichsen/alpaca_2k_test",
|
|
"type": "alpaca",
|
|
"shards": 10,
|
|
}
|
|
]
|
|
}
|
|
)
|
|
|
|
checked_cfg = validate_config(cfg)
|
|
|
|
def _check_config():
|
|
assert checked_cfg.datasets[0].path == cfg.datasets[0].path
|
|
assert checked_cfg.datasets[0].type == cfg.datasets[0].type
|
|
assert checked_cfg.datasets[0].shards == cfg.datasets[0].shards
|
|
|
|
_check_config()
|
|
|
|
checked_cfg = validate_config(
|
|
cfg,
|
|
capabilities={
|
|
"bf16": "false",
|
|
"n_gpu": 1,
|
|
"compute_capability": "8.0",
|
|
},
|
|
env_capabilities={
|
|
"torch_version": "2.5.1",
|
|
},
|
|
)
|
|
|
|
_check_config()
|
|
|
|
def test_dataset_default_chat_template_no_drop_param(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"datasets": [
|
|
{
|
|
"path": "LDJnr/Puffin",
|
|
"type": "chat_template",
|
|
"field_messages": "conversations",
|
|
"shards": 10,
|
|
"message_field_role": "from",
|
|
"message_field_content": "value",
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
checked_cfg = validate_config(cfg)
|
|
|
|
def _check_config():
|
|
assert checked_cfg.datasets[0].path == cfg.datasets[0].path
|
|
assert checked_cfg.datasets[0].type == cfg.datasets[0].type
|
|
assert checked_cfg.chat_template is None
|
|
assert (
|
|
checked_cfg.datasets[0].chat_template == ChatTemplate.tokenizer_default
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].field_messages == cfg.datasets[0].field_messages
|
|
)
|
|
assert checked_cfg.datasets[0].shards == cfg.datasets[0].shards
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_role
|
|
== cfg.datasets[0].message_field_role
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_content
|
|
== cfg.datasets[0].message_field_content
|
|
)
|
|
|
|
_check_config()
|
|
|
|
checked_cfg = validate_config(
|
|
cfg,
|
|
capabilities={
|
|
"bf16": "false",
|
|
"n_gpu": 1,
|
|
"compute_capability": "8.0",
|
|
},
|
|
env_capabilities={
|
|
"torch_version": "2.5.1",
|
|
},
|
|
)
|
|
|
|
_check_config()
|
|
|
|
def test_dataset_partial_default_chat_template_no_drop_param(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"chat_template": "chatml",
|
|
"datasets": [
|
|
{
|
|
"path": "LDJnr/Puffin",
|
|
"type": "chat_template",
|
|
"field_messages": "conversations",
|
|
"shards": 10,
|
|
"message_field_role": "from",
|
|
"message_field_content": "value",
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
checked_cfg = validate_config(cfg)
|
|
|
|
def _check_config():
|
|
assert checked_cfg.datasets[0].path == cfg.datasets[0].path
|
|
assert checked_cfg.datasets[0].type == cfg.datasets[0].type
|
|
assert checked_cfg.chat_template == ChatTemplate.chatml
|
|
assert (
|
|
checked_cfg.datasets[0].chat_template == ChatTemplate.tokenizer_default
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].field_messages == cfg.datasets[0].field_messages
|
|
)
|
|
assert checked_cfg.datasets[0].shards == cfg.datasets[0].shards
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_role
|
|
== cfg.datasets[0].message_field_role
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_content
|
|
== cfg.datasets[0].message_field_content
|
|
)
|
|
|
|
_check_config()
|
|
|
|
checked_cfg = validate_config(
|
|
cfg,
|
|
capabilities={
|
|
"bf16": "false",
|
|
"n_gpu": 1,
|
|
"compute_capability": "8.0",
|
|
},
|
|
env_capabilities={
|
|
"torch_version": "2.5.1",
|
|
},
|
|
)
|
|
|
|
_check_config()
|
|
|
|
def test_dataset_chatml_chat_template_no_drop_param(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"chat_template": "chatml",
|
|
"datasets": [
|
|
{
|
|
"path": "LDJnr/Puffin",
|
|
"type": "chat_template",
|
|
"chat_template": "gemma",
|
|
"field_messages": "conversations",
|
|
"shards": 10,
|
|
"message_field_role": "from",
|
|
"message_field_content": "value",
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
checked_cfg = validate_config(cfg)
|
|
|
|
def _check_config():
|
|
assert checked_cfg.datasets[0].path == cfg.datasets[0].path
|
|
assert checked_cfg.datasets[0].type == cfg.datasets[0].type
|
|
assert checked_cfg.chat_template == cfg.chat_template
|
|
assert (
|
|
checked_cfg.datasets[0].chat_template == cfg.datasets[0].chat_template
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].field_messages == cfg.datasets[0].field_messages
|
|
)
|
|
assert checked_cfg.datasets[0].shards == cfg.datasets[0].shards
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_role
|
|
== cfg.datasets[0].message_field_role
|
|
)
|
|
assert (
|
|
checked_cfg.datasets[0].message_field_content
|
|
== cfg.datasets[0].message_field_content
|
|
)
|
|
|
|
_check_config()
|
|
|
|
checked_cfg = validate_config(
|
|
cfg,
|
|
capabilities={
|
|
"bf16": "false",
|
|
"n_gpu": 1,
|
|
"compute_capability": "8.0",
|
|
},
|
|
env_capabilities={
|
|
"torch_version": "2.5.1",
|
|
},
|
|
)
|
|
|
|
_check_config()
|
|
|
|
def test_dataset_sharegpt_deprecation(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"chat_template": "chatml",
|
|
"datasets": [
|
|
{
|
|
"path": "LDJnr/Puffin",
|
|
"type": "sharegpt",
|
|
"conversation": "chatml",
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
# Check sharegpt deprecation is raised
|
|
with pytest.raises(ValueError, match=r".*type: sharegpt.*` is deprecated.*"):
|
|
validate_config(cfg)
|
|
|
|
# Check that deprecation is not thrown for non-str type
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"datasets": [
|
|
{
|
|
"path": "mhenrichsen/alpaca_2k_test",
|
|
"type": {
|
|
"field_instruction": "instruction",
|
|
"field_output": "output",
|
|
"field_system": "system",
|
|
"format": "<|user|> {instruction} {input} <|model|>",
|
|
"no_input_format": "<|user|> {instruction} <|model|>",
|
|
"system_prompt": "",
|
|
},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
validate_config(cfg)
|
|
|
|
# Check that deprecation is not thrown for non-sharegpt type
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"datasets": [
|
|
{
|
|
"path": "mhenrichsen/alpaca_2k_test",
|
|
"type": "alpaca",
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
validate_config(cfg)
|
|
|
|
def test_message_property_mappings(self, minimal_cfg):
|
|
cfg = DictDefault(
|
|
minimal_cfg
|
|
| {
|
|
"datasets": [
|
|
{
|
|
"path": "mhenrichsen/alpaca_2k_test",
|
|
"type": "alpaca",
|
|
"message_property_mappings": {
|
|
"role": "role",
|
|
"content": "content",
|
|
},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
|
|
validate_config(cfg)
|