diff --git a/.nojekyll b/.nojekyll index 25de98414..5ff385712 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -cae327e5 \ No newline at end of file +b5d4de8f \ No newline at end of file diff --git a/docs/dataset-formats/conversation.html b/docs/dataset-formats/conversation.html index 82fd2e6b3..1196ca0ab 100644 --- a/docs/dataset-formats/conversation.html +++ b/docs/dataset-formats/conversation.html @@ -719,23 +719,25 @@ Tip "role": "assistant", // call the function via assistant "tool_calls": [ { - "type": "function", - "function": { - "name": "...", - "arguments": { - "...": "...", - } - } - } - ] - }, - { - "role": "tool", - "name": "...", - "content": "..." - }, - ], -} + "id": "...", // required only for mistral + "type": "function", + "function": { + "name": "...", + "arguments": { + "...": "...", + } + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "...", // required only for mistral + "name": "...", + "content": "..." + }, + ], +}
diff --git a/search.json b/search.json index 795916c28..b9c35b09b 100644 --- a/search.json +++ b/search.json @@ -15,7 +15,7 @@ "href": "docs/dataset-formats/conversation.html", "title": "Conversation", "section": "", - "text": "Chat Template strategy uses a jinja2 template that converts a list of messages into a prompt. Support using tokenizer’s template, a supported template, or custom jinja2.\n\n\ndata.jsonl\n\n{\"messages\": [{\"role\": \"...\", \"content\": \"...\"}, {\"role\": \"...\", \"content\": \"...\"}, ...]}\n\nSee configs for full configs and supported templates.\n\n\nMost configs can be adapted as follows:\n# old\nchat_template: chatml\ndatasets:\n - path: ...\n type: sharegpt\n conversation: chatml\n\n# new (if using tokenizer's chat_template)\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n\n# new (if setting a new chat_template like chatml, gemma, etc)\nchat_template: chatml\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\nWe recommend checking the below examples for other usecases.\n\n\n\n\n\n(Legacy) Using the default chat template in the tokenizer_config.json on OpenAI messages format, training on only last message.\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train:\n train_on_eos:\n\n\n\n\n\n\nTip\n\n\n\nIf you receive an error like “chat_template choice is tokenizer_default but tokenizer’s chat_template is null.”, it means the tokenizer does not have a default chat_template. Follow the examples below instead to set a custom chat_template.\n\n\n\n\n\nUsing the gemma chat template to override the tokenizer_config.json’s chat template on OpenAI messages format, training on all assistant messages.\nchat_template: gemma # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train: [\"assistant\"] # default value\n\n\n\n\n\n\nNote\n\n\n\nIf you want to use built-in chat_template, use chat_template: tokenizer_default (this is set by default).\n\n\n\n\n\nUsing the tokenizer_config.json’s chat template or chatml as fallback if the former’s chat template does not exist, on OpenAI messages format, training on all assistant messages.\nchat_template: tokenizer_default_fallback_chatml # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n\n\n\nUsing a custom jinja template on OpenAI messages format, training on all assistant messages.\n# chat_template: jinja # `jinja` will be implied if the `chat_template_jinja` is set and this field is empty\nchat_template_jinja: \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|system|>' + '\\n' + message['content'] + '<|end|>' + '\\n'}}{% elif (message['role'] == 'user') %}{{'<|user|>' + '\\n' + message['content'] + '<|end|>' + '\\n' + '<|assistant|>' + '\\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|end|>' + '\\n'}}{% endif %}{% endfor %}\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n\n\n\n\n\nImportant\n\n\n\nPlease make sure that your tokenizer.eos_token is same as EOS (End-of-Sequence) token in template. Otherwise, set eos_token under special_tokens:.\n\n\n\n\n\n\nIf you are using a template that has a different EOT (End-of-Turn) token from EOS token or multiple EOT tokens (like Mistral V7 Tekken), set the eot_tokens: config. The handling of EOT tokens follows train_on_eos: which defaults to turn.\n\neot_tokens:\n - \"[/INST]\"\n # - \"[/SYSTEM_PROMPT]\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n # optional\n train_on_eot: turn # defaults read from train_on_eos (which defaults to turn)\n\n\n\n\n\n\nTip\n\n\n\nSee config documentation for detailed explanations of “turn”, “last”, and “all” options for training on tokens.\n\n\n\n\n\n\n\n\nNote\n\n\n\nUsing eot_tokens requires each token that exists in chat_template to be a single token in the tokenizer. Otherwise, the tokenizer will split the token and cause unexpected behavior.\nYou can add those tokens as new tokens under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:. See config for more details.\n\n\n\nContinuing from the previous example, if you want to train on all EOT token trainable turns but only last EOS token, set train_on_eos: last.\n\neot_tokens:\n - \"[/INST]\"\n # ...\n\ndatasets:\n - path: ...\n type: chat_template\n\n train_on_eos: last\n train_on_eot: turn\n\n\n\n\n\n\nTip\n\n\n\nIf EOS token only appears at the end of a prompt, train_on_eos: last is equivalent to train_on_eos: turn. Therefore, generally, you can leave them to their defaults and omit them.\n\n\n\n\n\nInstead of passing tools via the system prompt, an alternative method would be to have the tools in a separate column and loaded via chat_template to let the template dynamically build it.\n{\n \"tools\": [\n {\n \"type\": \"...\",\n \"function\": {\n \"name\": \"...\",\n \"description\": \"...\",\n \"parameters\": {\n \"type\": \"...\",\n \"properties\": {\n // ...\n },\n \"required\": [\"...\"],\n },\n },\n },\n ],\n \"messages\": [\n // ...\n {\n \"role\": \"assistant\", // call the function via assistant\n \"tool_calls\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"...\",\n \"arguments\": {\n \"...\": \"...\",\n }\n }\n }\n ]\n },\n {\n \"role\": \"tool\",\n \"name\": \"...\",\n \"content\": \"...\"\n },\n ],\n}\n\n\n\n\n\n\nNote\n\n\n\nTools need to follow JSON schema.\n\n\nchat_template: llama4\ndatasets:\n - path: ...\n type: chat_template\n # field_tools: tools # default is `tools`\n\n\n\n\n\n\nTip\n\n\n\nLook into the chat_template you are using to see if it supports tools and what the expected role is for the tool answer. In the example above, the tool answer is expected to be in the tool or ipython role for llama4 template.\n\n\n\n\n\n(Advanced) Using fine-grained control over tokens and turns to train in a conversation\nFor a data sample that looks like:\n\n\ndata.jsonl\n\n{\n \"conversations\": [\n {\"from\": \"system\", \"value\": \"You are an AI assistant.\", \"train\": false},\n {\"from\": \"human\", \"value\": \"Hello\", \"train\": false},\n {\"from\": \"assistant\", \"value\": \"Hello\", \"train\": true},\n {\"from\": \"human\", \"value\": \"How are you?\", \"train\": true},\n {\n \"from\": \"assistant\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train_detail\": [\n {\"begin_offset\": 0, \"end_offset\": 8, \"train\": false},\n {\"begin_offset\": 9, \"end_offset\": 18, \"train\": true},\n {\"begin_offset\": 19, \"end_offset\": 30, \"train\": false},\n ],\n },\n {\n \"from\": \"human\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train\": true,\n },\n {\"from\": \"assistant\", \"value\": \"Hi there!\", \"train\": true}\n ]\n}\n\nThe configuration would look like:\ndatasets:\n - path: ...\n type: chat_template\n chat_template: tokenizer_default\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n roles_to_train: []\n train_on_eos: turn\n message_field_training: train\n message_field_training_detail: train_detail\n\n\n\n\n\n\nTip\n\n\n\nIt is not necessary to set both message_field_training and message_field_training_detail at once.\n\n\n\n\n\n(For Qwen3 template only) Enable reasoning split, where the reasoning is split from the content and passed as a separate field into the template.\ndatasets:\n - path: ...\n type: chat_template\n chat_template: qwen3\n split_thinking: true\nFor example, a content can look like:\n{\n \"content\": \"<think>Some thinking outputs</think>Output after thinking.\"\n}\nAfter split, it will look like:\n{\n \"reasoning_content\": \"Some thinking outputs\",\n \"content\": \"Output after thinking...\"\n}", + "text": "Chat Template strategy uses a jinja2 template that converts a list of messages into a prompt. Support using tokenizer’s template, a supported template, or custom jinja2.\n\n\ndata.jsonl\n\n{\"messages\": [{\"role\": \"...\", \"content\": \"...\"}, {\"role\": \"...\", \"content\": \"...\"}, ...]}\n\nSee configs for full configs and supported templates.\n\n\nMost configs can be adapted as follows:\n# old\nchat_template: chatml\ndatasets:\n - path: ...\n type: sharegpt\n conversation: chatml\n\n# new (if using tokenizer's chat_template)\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n\n# new (if setting a new chat_template like chatml, gemma, etc)\nchat_template: chatml\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\nWe recommend checking the below examples for other usecases.\n\n\n\n\n\n(Legacy) Using the default chat template in the tokenizer_config.json on OpenAI messages format, training on only last message.\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train:\n train_on_eos:\n\n\n\n\n\n\nTip\n\n\n\nIf you receive an error like “chat_template choice is tokenizer_default but tokenizer’s chat_template is null.”, it means the tokenizer does not have a default chat_template. Follow the examples below instead to set a custom chat_template.\n\n\n\n\n\nUsing the gemma chat template to override the tokenizer_config.json’s chat template on OpenAI messages format, training on all assistant messages.\nchat_template: gemma # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train: [\"assistant\"] # default value\n\n\n\n\n\n\nNote\n\n\n\nIf you want to use built-in chat_template, use chat_template: tokenizer_default (this is set by default).\n\n\n\n\n\nUsing the tokenizer_config.json’s chat template or chatml as fallback if the former’s chat template does not exist, on OpenAI messages format, training on all assistant messages.\nchat_template: tokenizer_default_fallback_chatml # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n\n\n\nUsing a custom jinja template on OpenAI messages format, training on all assistant messages.\n# chat_template: jinja # `jinja` will be implied if the `chat_template_jinja` is set and this field is empty\nchat_template_jinja: \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|system|>' + '\\n' + message['content'] + '<|end|>' + '\\n'}}{% elif (message['role'] == 'user') %}{{'<|user|>' + '\\n' + message['content'] + '<|end|>' + '\\n' + '<|assistant|>' + '\\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|end|>' + '\\n'}}{% endif %}{% endfor %}\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n\n\n\n\n\nImportant\n\n\n\nPlease make sure that your tokenizer.eos_token is same as EOS (End-of-Sequence) token in template. Otherwise, set eos_token under special_tokens:.\n\n\n\n\n\n\nIf you are using a template that has a different EOT (End-of-Turn) token from EOS token or multiple EOT tokens (like Mistral V7 Tekken), set the eot_tokens: config. The handling of EOT tokens follows train_on_eos: which defaults to turn.\n\neot_tokens:\n - \"[/INST]\"\n # - \"[/SYSTEM_PROMPT]\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n # optional\n train_on_eot: turn # defaults read from train_on_eos (which defaults to turn)\n\n\n\n\n\n\nTip\n\n\n\nSee config documentation for detailed explanations of “turn”, “last”, and “all” options for training on tokens.\n\n\n\n\n\n\n\n\nNote\n\n\n\nUsing eot_tokens requires each token that exists in chat_template to be a single token in the tokenizer. Otherwise, the tokenizer will split the token and cause unexpected behavior.\nYou can add those tokens as new tokens under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:. See config for more details.\n\n\n\nContinuing from the previous example, if you want to train on all EOT token trainable turns but only last EOS token, set train_on_eos: last.\n\neot_tokens:\n - \"[/INST]\"\n # ...\n\ndatasets:\n - path: ...\n type: chat_template\n\n train_on_eos: last\n train_on_eot: turn\n\n\n\n\n\n\nTip\n\n\n\nIf EOS token only appears at the end of a prompt, train_on_eos: last is equivalent to train_on_eos: turn. Therefore, generally, you can leave them to their defaults and omit them.\n\n\n\n\n\nInstead of passing tools via the system prompt, an alternative method would be to have the tools in a separate column and loaded via chat_template to let the template dynamically build it.\n{\n \"tools\": [\n {\n \"type\": \"...\",\n \"function\": {\n \"name\": \"...\",\n \"description\": \"...\",\n \"parameters\": {\n \"type\": \"...\",\n \"properties\": {\n // ...\n },\n \"required\": [\"...\"],\n },\n },\n },\n ],\n \"messages\": [\n // ...\n {\n \"role\": \"assistant\", // call the function via assistant\n \"tool_calls\": [\n {\n \"id\": \"...\", // required only for mistral\n \"type\": \"function\",\n \"function\": {\n \"name\": \"...\",\n \"arguments\": {\n \"...\": \"...\",\n }\n }\n }\n ]\n },\n {\n \"role\": \"tool\",\n \"tool_call_id\": \"...\", // required only for mistral\n \"name\": \"...\",\n \"content\": \"...\"\n },\n ],\n}\n\n\n\n\n\n\nNote\n\n\n\nTools need to follow JSON schema.\n\n\nchat_template: llama4\ndatasets:\n - path: ...\n type: chat_template\n # field_tools: tools # default is `tools`\n\n\n\n\n\n\nTip\n\n\n\nLook into the chat_template you are using to see if it supports tools and what the expected role is for the tool answer. In the example above, the tool answer is expected to be in the tool or ipython role for llama4 template.\n\n\n\n\n\n(Advanced) Using fine-grained control over tokens and turns to train in a conversation\nFor a data sample that looks like:\n\n\ndata.jsonl\n\n{\n \"conversations\": [\n {\"from\": \"system\", \"value\": \"You are an AI assistant.\", \"train\": false},\n {\"from\": \"human\", \"value\": \"Hello\", \"train\": false},\n {\"from\": \"assistant\", \"value\": \"Hello\", \"train\": true},\n {\"from\": \"human\", \"value\": \"How are you?\", \"train\": true},\n {\n \"from\": \"assistant\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train_detail\": [\n {\"begin_offset\": 0, \"end_offset\": 8, \"train\": false},\n {\"begin_offset\": 9, \"end_offset\": 18, \"train\": true},\n {\"begin_offset\": 19, \"end_offset\": 30, \"train\": false},\n ],\n },\n {\n \"from\": \"human\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train\": true,\n },\n {\"from\": \"assistant\", \"value\": \"Hi there!\", \"train\": true}\n ]\n}\n\nThe configuration would look like:\ndatasets:\n - path: ...\n type: chat_template\n chat_template: tokenizer_default\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n roles_to_train: []\n train_on_eos: turn\n message_field_training: train\n message_field_training_detail: train_detail\n\n\n\n\n\n\nTip\n\n\n\nIt is not necessary to set both message_field_training and message_field_training_detail at once.\n\n\n\n\n\n(For Qwen3 template only) Enable reasoning split, where the reasoning is split from the content and passed as a separate field into the template.\ndatasets:\n - path: ...\n type: chat_template\n chat_template: qwen3\n split_thinking: true\nFor example, a content can look like:\n{\n \"content\": \"<think>Some thinking outputs</think>Output after thinking.\"\n}\nAfter split, it will look like:\n{\n \"reasoning_content\": \"Some thinking outputs\",\n \"content\": \"Output after thinking...\"\n}", "crumbs": [ "Dataset Formats", "Conversation" @@ -26,7 +26,7 @@ "href": "docs/dataset-formats/conversation.html#chat_template", "title": "Conversation", "section": "", - "text": "Chat Template strategy uses a jinja2 template that converts a list of messages into a prompt. Support using tokenizer’s template, a supported template, or custom jinja2.\n\n\ndata.jsonl\n\n{\"messages\": [{\"role\": \"...\", \"content\": \"...\"}, {\"role\": \"...\", \"content\": \"...\"}, ...]}\n\nSee configs for full configs and supported templates.\n\n\nMost configs can be adapted as follows:\n# old\nchat_template: chatml\ndatasets:\n - path: ...\n type: sharegpt\n conversation: chatml\n\n# new (if using tokenizer's chat_template)\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n\n# new (if setting a new chat_template like chatml, gemma, etc)\nchat_template: chatml\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\nWe recommend checking the below examples for other usecases.\n\n\n\n\n\n(Legacy) Using the default chat template in the tokenizer_config.json on OpenAI messages format, training on only last message.\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train:\n train_on_eos:\n\n\n\n\n\n\nTip\n\n\n\nIf you receive an error like “chat_template choice is tokenizer_default but tokenizer’s chat_template is null.”, it means the tokenizer does not have a default chat_template. Follow the examples below instead to set a custom chat_template.\n\n\n\n\n\nUsing the gemma chat template to override the tokenizer_config.json’s chat template on OpenAI messages format, training on all assistant messages.\nchat_template: gemma # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train: [\"assistant\"] # default value\n\n\n\n\n\n\nNote\n\n\n\nIf you want to use built-in chat_template, use chat_template: tokenizer_default (this is set by default).\n\n\n\n\n\nUsing the tokenizer_config.json’s chat template or chatml as fallback if the former’s chat template does not exist, on OpenAI messages format, training on all assistant messages.\nchat_template: tokenizer_default_fallback_chatml # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n\n\n\nUsing a custom jinja template on OpenAI messages format, training on all assistant messages.\n# chat_template: jinja # `jinja` will be implied if the `chat_template_jinja` is set and this field is empty\nchat_template_jinja: \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|system|>' + '\\n' + message['content'] + '<|end|>' + '\\n'}}{% elif (message['role'] == 'user') %}{{'<|user|>' + '\\n' + message['content'] + '<|end|>' + '\\n' + '<|assistant|>' + '\\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|end|>' + '\\n'}}{% endif %}{% endfor %}\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n\n\n\n\n\nImportant\n\n\n\nPlease make sure that your tokenizer.eos_token is same as EOS (End-of-Sequence) token in template. Otherwise, set eos_token under special_tokens:.\n\n\n\n\n\n\nIf you are using a template that has a different EOT (End-of-Turn) token from EOS token or multiple EOT tokens (like Mistral V7 Tekken), set the eot_tokens: config. The handling of EOT tokens follows train_on_eos: which defaults to turn.\n\neot_tokens:\n - \"[/INST]\"\n # - \"[/SYSTEM_PROMPT]\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n # optional\n train_on_eot: turn # defaults read from train_on_eos (which defaults to turn)\n\n\n\n\n\n\nTip\n\n\n\nSee config documentation for detailed explanations of “turn”, “last”, and “all” options for training on tokens.\n\n\n\n\n\n\n\n\nNote\n\n\n\nUsing eot_tokens requires each token that exists in chat_template to be a single token in the tokenizer. Otherwise, the tokenizer will split the token and cause unexpected behavior.\nYou can add those tokens as new tokens under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:. See config for more details.\n\n\n\nContinuing from the previous example, if you want to train on all EOT token trainable turns but only last EOS token, set train_on_eos: last.\n\neot_tokens:\n - \"[/INST]\"\n # ...\n\ndatasets:\n - path: ...\n type: chat_template\n\n train_on_eos: last\n train_on_eot: turn\n\n\n\n\n\n\nTip\n\n\n\nIf EOS token only appears at the end of a prompt, train_on_eos: last is equivalent to train_on_eos: turn. Therefore, generally, you can leave them to their defaults and omit them.\n\n\n\n\n\nInstead of passing tools via the system prompt, an alternative method would be to have the tools in a separate column and loaded via chat_template to let the template dynamically build it.\n{\n \"tools\": [\n {\n \"type\": \"...\",\n \"function\": {\n \"name\": \"...\",\n \"description\": \"...\",\n \"parameters\": {\n \"type\": \"...\",\n \"properties\": {\n // ...\n },\n \"required\": [\"...\"],\n },\n },\n },\n ],\n \"messages\": [\n // ...\n {\n \"role\": \"assistant\", // call the function via assistant\n \"tool_calls\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"...\",\n \"arguments\": {\n \"...\": \"...\",\n }\n }\n }\n ]\n },\n {\n \"role\": \"tool\",\n \"name\": \"...\",\n \"content\": \"...\"\n },\n ],\n}\n\n\n\n\n\n\nNote\n\n\n\nTools need to follow JSON schema.\n\n\nchat_template: llama4\ndatasets:\n - path: ...\n type: chat_template\n # field_tools: tools # default is `tools`\n\n\n\n\n\n\nTip\n\n\n\nLook into the chat_template you are using to see if it supports tools and what the expected role is for the tool answer. In the example above, the tool answer is expected to be in the tool or ipython role for llama4 template.\n\n\n\n\n\n(Advanced) Using fine-grained control over tokens and turns to train in a conversation\nFor a data sample that looks like:\n\n\ndata.jsonl\n\n{\n \"conversations\": [\n {\"from\": \"system\", \"value\": \"You are an AI assistant.\", \"train\": false},\n {\"from\": \"human\", \"value\": \"Hello\", \"train\": false},\n {\"from\": \"assistant\", \"value\": \"Hello\", \"train\": true},\n {\"from\": \"human\", \"value\": \"How are you?\", \"train\": true},\n {\n \"from\": \"assistant\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train_detail\": [\n {\"begin_offset\": 0, \"end_offset\": 8, \"train\": false},\n {\"begin_offset\": 9, \"end_offset\": 18, \"train\": true},\n {\"begin_offset\": 19, \"end_offset\": 30, \"train\": false},\n ],\n },\n {\n \"from\": \"human\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train\": true,\n },\n {\"from\": \"assistant\", \"value\": \"Hi there!\", \"train\": true}\n ]\n}\n\nThe configuration would look like:\ndatasets:\n - path: ...\n type: chat_template\n chat_template: tokenizer_default\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n roles_to_train: []\n train_on_eos: turn\n message_field_training: train\n message_field_training_detail: train_detail\n\n\n\n\n\n\nTip\n\n\n\nIt is not necessary to set both message_field_training and message_field_training_detail at once.\n\n\n\n\n\n(For Qwen3 template only) Enable reasoning split, where the reasoning is split from the content and passed as a separate field into the template.\ndatasets:\n - path: ...\n type: chat_template\n chat_template: qwen3\n split_thinking: true\nFor example, a content can look like:\n{\n \"content\": \"<think>Some thinking outputs</think>Output after thinking.\"\n}\nAfter split, it will look like:\n{\n \"reasoning_content\": \"Some thinking outputs\",\n \"content\": \"Output after thinking...\"\n}", + "text": "Chat Template strategy uses a jinja2 template that converts a list of messages into a prompt. Support using tokenizer’s template, a supported template, or custom jinja2.\n\n\ndata.jsonl\n\n{\"messages\": [{\"role\": \"...\", \"content\": \"...\"}, {\"role\": \"...\", \"content\": \"...\"}, ...]}\n\nSee configs for full configs and supported templates.\n\n\nMost configs can be adapted as follows:\n# old\nchat_template: chatml\ndatasets:\n - path: ...\n type: sharegpt\n conversation: chatml\n\n# new (if using tokenizer's chat_template)\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n\n# new (if setting a new chat_template like chatml, gemma, etc)\nchat_template: chatml\ndatasets:\n - path: ...\n type: chat_template\n\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\nWe recommend checking the below examples for other usecases.\n\n\n\n\n\n(Legacy) Using the default chat template in the tokenizer_config.json on OpenAI messages format, training on only last message.\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train:\n train_on_eos:\n\n\n\n\n\n\nTip\n\n\n\nIf you receive an error like “chat_template choice is tokenizer_default but tokenizer’s chat_template is null.”, it means the tokenizer does not have a default chat_template. Follow the examples below instead to set a custom chat_template.\n\n\n\n\n\nUsing the gemma chat template to override the tokenizer_config.json’s chat template on OpenAI messages format, training on all assistant messages.\nchat_template: gemma # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n roles_to_train: [\"assistant\"] # default value\n\n\n\n\n\n\nNote\n\n\n\nIf you want to use built-in chat_template, use chat_template: tokenizer_default (this is set by default).\n\n\n\n\n\nUsing the tokenizer_config.json’s chat template or chatml as fallback if the former’s chat template does not exist, on OpenAI messages format, training on all assistant messages.\nchat_template: tokenizer_default_fallback_chatml # this overwrites the tokenizer's chat_template\ndatasets:\n - path: ...\n type: chat_template\n\n\n\nUsing a custom jinja template on OpenAI messages format, training on all assistant messages.\n# chat_template: jinja # `jinja` will be implied if the `chat_template_jinja` is set and this field is empty\nchat_template_jinja: \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|system|>' + '\\n' + message['content'] + '<|end|>' + '\\n'}}{% elif (message['role'] == 'user') %}{{'<|user|>' + '\\n' + message['content'] + '<|end|>' + '\\n' + '<|assistant|>' + '\\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|end|>' + '\\n'}}{% endif %}{% endfor %}\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n\n\n\n\n\nImportant\n\n\n\nPlease make sure that your tokenizer.eos_token is same as EOS (End-of-Sequence) token in template. Otherwise, set eos_token under special_tokens:.\n\n\n\n\n\n\nIf you are using a template that has a different EOT (End-of-Turn) token from EOS token or multiple EOT tokens (like Mistral V7 Tekken), set the eot_tokens: config. The handling of EOT tokens follows train_on_eos: which defaults to turn.\n\neot_tokens:\n - \"[/INST]\"\n # - \"[/SYSTEM_PROMPT]\"\n\ndatasets:\n - path: ...\n type: chat_template\n\n # optional\n train_on_eot: turn # defaults read from train_on_eos (which defaults to turn)\n\n\n\n\n\n\nTip\n\n\n\nSee config documentation for detailed explanations of “turn”, “last”, and “all” options for training on tokens.\n\n\n\n\n\n\n\n\nNote\n\n\n\nUsing eot_tokens requires each token that exists in chat_template to be a single token in the tokenizer. Otherwise, the tokenizer will split the token and cause unexpected behavior.\nYou can add those tokens as new tokens under tokens: or (recommended) override unused added_tokens via added_tokens_overrides:. See config for more details.\n\n\n\nContinuing from the previous example, if you want to train on all EOT token trainable turns but only last EOS token, set train_on_eos: last.\n\neot_tokens:\n - \"[/INST]\"\n # ...\n\ndatasets:\n - path: ...\n type: chat_template\n\n train_on_eos: last\n train_on_eot: turn\n\n\n\n\n\n\nTip\n\n\n\nIf EOS token only appears at the end of a prompt, train_on_eos: last is equivalent to train_on_eos: turn. Therefore, generally, you can leave them to their defaults and omit them.\n\n\n\n\n\nInstead of passing tools via the system prompt, an alternative method would be to have the tools in a separate column and loaded via chat_template to let the template dynamically build it.\n{\n \"tools\": [\n {\n \"type\": \"...\",\n \"function\": {\n \"name\": \"...\",\n \"description\": \"...\",\n \"parameters\": {\n \"type\": \"...\",\n \"properties\": {\n // ...\n },\n \"required\": [\"...\"],\n },\n },\n },\n ],\n \"messages\": [\n // ...\n {\n \"role\": \"assistant\", // call the function via assistant\n \"tool_calls\": [\n {\n \"id\": \"...\", // required only for mistral\n \"type\": \"function\",\n \"function\": {\n \"name\": \"...\",\n \"arguments\": {\n \"...\": \"...\",\n }\n }\n }\n ]\n },\n {\n \"role\": \"tool\",\n \"tool_call_id\": \"...\", // required only for mistral\n \"name\": \"...\",\n \"content\": \"...\"\n },\n ],\n}\n\n\n\n\n\n\nNote\n\n\n\nTools need to follow JSON schema.\n\n\nchat_template: llama4\ndatasets:\n - path: ...\n type: chat_template\n # field_tools: tools # default is `tools`\n\n\n\n\n\n\nTip\n\n\n\nLook into the chat_template you are using to see if it supports tools and what the expected role is for the tool answer. In the example above, the tool answer is expected to be in the tool or ipython role for llama4 template.\n\n\n\n\n\n(Advanced) Using fine-grained control over tokens and turns to train in a conversation\nFor a data sample that looks like:\n\n\ndata.jsonl\n\n{\n \"conversations\": [\n {\"from\": \"system\", \"value\": \"You are an AI assistant.\", \"train\": false},\n {\"from\": \"human\", \"value\": \"Hello\", \"train\": false},\n {\"from\": \"assistant\", \"value\": \"Hello\", \"train\": true},\n {\"from\": \"human\", \"value\": \"How are you?\", \"train\": true},\n {\n \"from\": \"assistant\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train_detail\": [\n {\"begin_offset\": 0, \"end_offset\": 8, \"train\": false},\n {\"begin_offset\": 9, \"end_offset\": 18, \"train\": true},\n {\"begin_offset\": 19, \"end_offset\": 30, \"train\": false},\n ],\n },\n {\n \"from\": \"human\",\n \"value\": \"I'm doing very well, thank you!\",\n \"train\": true,\n },\n {\"from\": \"assistant\", \"value\": \"Hi there!\", \"train\": true}\n ]\n}\n\nThe configuration would look like:\ndatasets:\n - path: ...\n type: chat_template\n chat_template: tokenizer_default\n field_messages: conversations\n message_property_mappings:\n role: from\n content: value\n roles_to_train: []\n train_on_eos: turn\n message_field_training: train\n message_field_training_detail: train_detail\n\n\n\n\n\n\nTip\n\n\n\nIt is not necessary to set both message_field_training and message_field_training_detail at once.\n\n\n\n\n\n(For Qwen3 template only) Enable reasoning split, where the reasoning is split from the content and passed as a separate field into the template.\ndatasets:\n - path: ...\n type: chat_template\n chat_template: qwen3\n split_thinking: true\nFor example, a content can look like:\n{\n \"content\": \"<think>Some thinking outputs</think>Output after thinking.\"\n}\nAfter split, it will look like:\n{\n \"reasoning_content\": \"Some thinking outputs\",\n \"content\": \"Output after thinking...\"\n}", "crumbs": [ "Dataset Formats", "Conversation" diff --git a/sitemap.xml b/sitemap.xml index 4e354d040..4169c364e 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,758 +2,758 @@ https://docs.axolotl.ai/docs/unsloth.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/dataset-formats/conversation.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/stepwise_supervised.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/tokenized.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/mac.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/nccl.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/multi-node.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/docker.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/lr_groups.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/inference.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/cli.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/config-reference.html - 2025-07-09T18:56:27.253Z + 2025-07-11T02:38:02.905Z https://docs.axolotl.ai/docs/multi-gpu.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/debugging.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/multimodal.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/api/cli.sweeps.html - 2025-07-09T18:56:13.622Z + 2025-07-11T02:37:49.470Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.llama3.html - 2025-07-09T18:56:13.952Z + 2025-07-11T02:37:49.795Z https://docs.axolotl.ai/docs/api/utils.schedulers.html - 2025-07-09T18:56:14.337Z + 2025-07-11T02:37:50.176Z https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_xformers.html - 2025-07-09T18:56:14.148Z + 2025-07-11T02:37:49.991Z https://docs.axolotl.ai/docs/api/cli.cloud.modal_.html - 2025-07-09T18:56:13.670Z + 2025-07-11T02:37:49.518Z https://docs.axolotl.ai/docs/api/kernels.geglu.html - 2025-07-09T18:56:14.102Z + 2025-07-11T02:37:49.945Z https://docs.axolotl.ai/docs/api/core.trainers.utils.html - 2025-07-09T18:56:13.743Z + 2025-07-11T02:37:49.590Z https://docs.axolotl.ai/docs/api/core.datasets.chat.html - 2025-07-09T18:56:13.473Z + 2025-07-11T02:37:49.322Z https://docs.axolotl.ai/docs/api/utils.schemas.peft.html - 2025-07-09T18:56:14.451Z + 2025-07-11T02:37:50.288Z https://docs.axolotl.ai/docs/api/monkeypatch.btlm_attn_hijack_flash.html - 2025-07-09T18:56:14.210Z + 2025-07-11T02:37:50.053Z https://docs.axolotl.ai/docs/api/prompt_strategies.chat_template.html - 2025-07-09T18:56:13.853Z + 2025-07-11T02:37:49.698Z https://docs.axolotl.ai/docs/api/prompt_strategies.kto.user_defined.html - 2025-07-09T18:56:13.984Z + 2025-07-11T02:37:49.827Z https://docs.axolotl.ai/docs/api/cli.cloud.base.html - 2025-07-09T18:56:13.664Z + 2025-07-11T02:37:49.511Z https://docs.axolotl.ai/docs/api/kernels.swiglu.html - 2025-07-09T18:56:14.112Z + 2025-07-11T02:37:49.956Z https://docs.axolotl.ai/docs/api/prompt_strategies.stepwise_supervised.html - 2025-07-09T18:56:13.918Z + 2025-07-11T02:37:49.762Z https://docs.axolotl.ai/docs/api/prompt_strategies.bradley_terry.llama3.html - 2025-07-09T18:56:14.008Z + 2025-07-11T02:37:49.851Z https://docs.axolotl.ai/docs/api/prompt_strategies.completion.html - 2025-07-09T18:56:13.908Z + 2025-07-11T02:37:49.751Z https://docs.axolotl.ai/docs/api/kernels.utils.html - 2025-07-09T18:56:14.121Z + 2025-07-11T02:37:49.965Z https://docs.axolotl.ai/docs/api/common.datasets.html - 2025-07-09T18:56:14.665Z + 2025-07-11T02:37:50.498Z https://docs.axolotl.ai/docs/api/utils.schemas.datasets.html - 2025-07-09T18:56:14.442Z + 2025-07-11T02:37:50.279Z https://docs.axolotl.ai/docs/api/core.builders.rl.html - 2025-07-09T18:56:13.429Z + 2025-07-11T02:37:49.277Z https://docs.axolotl.ai/docs/api/evaluate.html - 2025-07-09T18:56:13.335Z + 2025-07-11T02:37:49.184Z https://docs.axolotl.ai/docs/api/kernels.quantize.html - 2025-07-09T18:56:14.120Z + 2025-07-11T02:37:49.963Z https://docs.axolotl.ai/docs/api/monkeypatch.llama_attn_hijack_flash.html - 2025-07-09T18:56:14.147Z + 2025-07-11T02:37:49.990Z https://docs.axolotl.ai/docs/api/core.trainers.mixins.rng_state_loader.html - 2025-07-09T18:56:13.787Z + 2025-07-11T02:37:49.633Z https://docs.axolotl.ai/docs/api/integrations.base.html - 2025-07-09T18:56:14.624Z + 2025-07-11T02:37:50.459Z https://docs.axolotl.ai/docs/api/cli.merge_lora.html - 2025-07-09T18:56:13.596Z + 2025-07-11T02:37:49.444Z https://docs.axolotl.ai/docs/api/cli.merge_sharded_fsdp_weights.html - 2025-07-09T18:56:13.608Z + 2025-07-11T02:37:49.456Z https://docs.axolotl.ai/docs/api/monkeypatch.transformers_fa_utils.html - 2025-07-09T18:56:14.227Z + 2025-07-11T02:37:50.070Z https://docs.axolotl.ai/docs/api/prompt_strategies.llama2_chat.html - 2025-07-09T18:56:13.902Z + 2025-07-11T02:37:49.746Z https://docs.axolotl.ai/docs/api/utils.collators.mm_chat.html - 2025-07-09T18:56:14.695Z + 2025-07-11T02:37:50.527Z https://docs.axolotl.ai/docs/api/utils.data.sft.html - 2025-07-09T18:56:14.376Z + 2025-07-11T02:37:50.215Z https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_instruct.html - 2025-07-09T18:56:13.869Z + 2025-07-11T02:37:49.713Z https://docs.axolotl.ai/docs/api/integrations.liger.args.html - 2025-07-09T18:56:14.640Z + 2025-07-11T02:37:50.474Z https://docs.axolotl.ai/docs/api/monkeypatch.mistral_attn_hijack_flash.html - 2025-07-09T18:56:14.163Z + 2025-07-11T02:37:50.006Z https://docs.axolotl.ai/docs/api/cli.vllm_serve.html - 2025-07-09T18:56:13.661Z + 2025-07-11T02:37:49.508Z https://docs.axolotl.ai/docs/api/monkeypatch.utils.html - 2025-07-09T18:56:14.209Z + 2025-07-11T02:37:50.052Z https://docs.axolotl.ai/docs/api/loaders.patch_manager.html - 2025-07-09T18:56:13.777Z + 2025-07-11T02:37:49.623Z https://docs.axolotl.ai/docs/api/utils.schemas.integrations.html - 2025-07-09T18:56:14.472Z + 2025-07-11T02:37:50.308Z https://docs.axolotl.ai/docs/api/utils.callbacks.perplexity.html - 2025-07-09T18:56:14.742Z + 2025-07-11T02:37:50.575Z https://docs.axolotl.ai/docs/api/cli.utils.html - 2025-07-09T18:56:13.654Z + 2025-07-11T02:37:49.501Z https://docs.axolotl.ai/docs/api/utils.schemas.config.html - 2025-07-09T18:56:14.412Z + 2025-07-11T02:37:50.249Z https://docs.axolotl.ai/docs/api/prompt_strategies.input_output.html - 2025-07-09T18:56:13.914Z + 2025-07-11T02:37:49.757Z https://docs.axolotl.ai/docs/api/utils.distributed.html - 2025-07-09T18:56:14.357Z + 2025-07-11T02:37:50.196Z https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_disk.html - 2025-07-09T18:56:14.263Z + 2025-07-11T02:37:50.105Z https://docs.axolotl.ai/docs/api/monkeypatch.trainer_fsdp_optim.html - 2025-07-09T18:56:14.221Z + 2025-07-11T02:37:50.063Z https://docs.axolotl.ai/docs/api/core.builders.base.html - 2025-07-09T18:56:13.416Z + 2025-07-11T02:37:49.264Z https://docs.axolotl.ai/docs/api/core.trainers.trl.html - 2025-07-09T18:56:13.702Z + 2025-07-11T02:37:49.549Z https://docs.axolotl.ai/docs/api/cli.evaluate.html - 2025-07-09T18:56:13.529Z + 2025-07-11T02:37:49.377Z https://docs.axolotl.ai/docs/api/utils.optimizers.adopt.html - 2025-07-09T18:56:14.368Z + 2025-07-11T02:37:50.206Z https://docs.axolotl.ai/docs/api/utils.callbacks.qat.html - 2025-07-09T18:56:14.761Z + 2025-07-11T02:37:50.593Z https://docs.axolotl.ai/docs/api/core.trainers.dpo.trainer.html - 2025-07-09T18:56:13.718Z + 2025-07-11T02:37:49.565Z https://docs.axolotl.ai/docs/api/core.chat.format.shared.html - 2025-07-09T18:56:13.468Z + 2025-07-11T02:37:49.317Z https://docs.axolotl.ai/docs/api/monkeypatch.relora.html - 2025-07-09T18:56:14.171Z + 2025-07-11T02:37:50.014Z https://docs.axolotl.ai/docs/api/cli.config.html - 2025-07-09T18:56:13.573Z + 2025-07-11T02:37:49.421Z https://docs.axolotl.ai/docs/api/cli.preprocess.html - 2025-07-09T18:56:13.616Z + 2025-07-11T02:37:49.464Z https://docs.axolotl.ai/docs/api/core.trainers.base.html - 2025-07-09T18:56:13.686Z + 2025-07-11T02:37:49.533Z https://docs.axolotl.ai/docs/api/convert.html - 2025-07-09T18:56:13.359Z + 2025-07-11T02:37:49.207Z https://docs.axolotl.ai/docs/api/prompt_strategies.pygmalion.html - 2025-07-09T18:56:13.935Z + 2025-07-11T02:37:49.779Z https://docs.axolotl.ai/docs/api/utils.schemas.trl.html - 2025-07-09T18:56:14.454Z + 2025-07-11T02:37:50.291Z https://docs.axolotl.ai/docs/api/cli.args.html - 2025-07-09T18:56:13.548Z + 2025-07-11T02:37:49.397Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chat_template.html - 2025-07-09T18:56:13.941Z + 2025-07-11T02:37:49.785Z https://docs.axolotl.ai/docs/api/loaders.constants.html - 2025-07-09T18:56:13.778Z + 2025-07-11T02:37:49.625Z https://docs.axolotl.ai/docs/api/logging_config.html - 2025-07-09T18:56:13.410Z + 2025-07-11T02:37:49.258Z https://docs.axolotl.ai/docs/api/cli.inference.html - 2025-07-09T18:56:13.587Z + 2025-07-11T02:37:49.435Z https://docs.axolotl.ai/docs/api/utils.ctx_managers.sequence_parallel.html - 2025-07-09T18:56:13.818Z + 2025-07-11T02:37:49.664Z https://docs.axolotl.ai/docs/api/integrations.spectrum.args.html - 2025-07-09T18:56:14.647Z + 2025-07-11T02:37:50.480Z https://docs.axolotl.ai/docs/api/utils.schemas.training.html - 2025-07-09T18:56:14.424Z + 2025-07-11T02:37:50.261Z https://docs.axolotl.ai/docs/api/prompt_strategies.orcamini.html - 2025-07-09T18:56:13.929Z + 2025-07-11T02:37:49.772Z https://docs.axolotl.ai/docs/api/utils.freeze.html - 2025-07-09T18:56:14.295Z + 2025-07-11T02:37:50.135Z https://docs.axolotl.ai/docs/api/loaders.tokenizer.html - 2025-07-09T18:56:13.762Z + 2025-07-11T02:37:49.608Z https://docs.axolotl.ai/docs/api/utils.bench.html - 2025-07-09T18:56:14.287Z + 2025-07-11T02:37:50.127Z https://docs.axolotl.ai/docs/api/utils.quantization.html - 2025-07-09T18:56:14.397Z + 2025-07-11T02:37:50.235Z https://docs.axolotl.ai/docs/batch_vs_grad.html - 2025-07-09T18:52:52.310Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/input_output.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/sequence_parallelism.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/reward_modelling.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/index.html - 2025-07-09T18:52:52.329Z + 2025-07-11T02:34:39.214Z https://docs.axolotl.ai/src/axolotl/integrations/LICENSE.html - 2025-07-09T18:52:52.333Z + 2025-07-11T02:34:39.218Z https://docs.axolotl.ai/FAQS.html - 2025-07-09T18:52:52.309Z + 2025-07-11T02:34:39.194Z https://docs.axolotl.ai/src/axolotl/integrations/cut_cross_entropy/ACKNOWLEDGEMENTS.html - 2025-07-09T18:52:52.334Z + 2025-07-11T02:34:39.219Z https://docs.axolotl.ai/TODO.html - 2025-07-09T18:52:52.309Z + 2025-07-11T02:34:39.194Z https://docs.axolotl.ai/examples/colab-notebooks/colab-axolotl-example.html - 2025-07-09T18:52:52.316Z + 2025-07-11T02:34:39.202Z https://docs.axolotl.ai/docs/torchao.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/ray-integration.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/quantize.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/qat.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/api/utils.lora.html - 2025-07-09T18:56:14.277Z + 2025-07-11T02:37:50.118Z https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_w_system.html - 2025-07-09T18:56:13.881Z + 2025-07-11T02:37:49.725Z https://docs.axolotl.ai/docs/api/monkeypatch.stablelm_attn_hijack_flash.html - 2025-07-09T18:56:14.218Z + 2025-07-11T02:37:50.060Z https://docs.axolotl.ai/docs/api/utils.collators.core.html - 2025-07-09T18:56:14.667Z + 2025-07-11T02:37:50.501Z https://docs.axolotl.ai/docs/api/prompt_strategies.metharme.html - 2025-07-09T18:56:13.925Z + 2025-07-11T02:37:49.769Z https://docs.axolotl.ai/docs/api/utils.callbacks.profiler.html - 2025-07-09T18:56:14.746Z + 2025-07-11T02:37:50.578Z https://docs.axolotl.ai/docs/api/utils.data.pretraining.html - 2025-07-09T18:56:14.369Z + 2025-07-11T02:37:50.208Z https://docs.axolotl.ai/docs/api/utils.callbacks.lisa.html - 2025-07-09T18:56:14.747Z + 2025-07-11T02:37:50.580Z https://docs.axolotl.ai/docs/api/utils.trainer.html - 2025-07-09T18:56:14.312Z + 2025-07-11T02:37:50.152Z https://docs.axolotl.ai/docs/api/integrations.cut_cross_entropy.args.html - 2025-07-09T18:56:14.628Z + 2025-07-11T02:37:50.462Z https://docs.axolotl.ai/docs/api/utils.schemas.model.html - 2025-07-09T18:56:14.419Z + 2025-07-11T02:37:50.256Z https://docs.axolotl.ai/docs/api/monkeypatch.data.batch_dataset_fetcher.html - 2025-07-09T18:56:14.230Z + 2025-07-11T02:37:50.073Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.zephyr.html - 2025-07-09T18:56:13.963Z + 2025-07-11T02:37:49.807Z https://docs.axolotl.ai/docs/api/datasets.html - 2025-07-09T18:56:13.346Z + 2025-07-11T02:37:49.194Z https://docs.axolotl.ai/docs/api/utils.schemas.enums.html - 2025-07-09T18:56:14.482Z + 2025-07-11T02:37:50.319Z https://docs.axolotl.ai/docs/api/integrations.kd.trainer.html - 2025-07-09T18:56:14.636Z + 2025-07-11T02:37:50.470Z https://docs.axolotl.ai/docs/api/monkeypatch.lora_kernels.html - 2025-07-09T18:56:14.201Z + 2025-07-11T02:37:50.044Z https://docs.axolotl.ai/docs/api/utils.collators.batching.html - 2025-07-09T18:56:14.686Z + 2025-07-11T02:37:50.519Z https://docs.axolotl.ai/docs/api/core.trainers.grpo.sampler.html - 2025-07-09T18:56:13.742Z + 2025-07-11T02:37:49.588Z https://docs.axolotl.ai/docs/api/prompt_strategies.base.html - 2025-07-09T18:56:13.819Z + 2025-07-11T02:37:49.665Z https://docs.axolotl.ai/docs/api/monkeypatch.multipack.html - 2025-07-09T18:56:14.164Z + 2025-07-11T02:37:50.007Z https://docs.axolotl.ai/docs/api/prompt_strategies.orpo.chat_template.html - 2025-07-09T18:56:14.005Z + 2025-07-11T02:37:49.848Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.passthrough.html - 2025-07-09T18:56:13.966Z + 2025-07-11T02:37:49.810Z https://docs.axolotl.ai/docs/api/core.chat.format.chatml.html - 2025-07-09T18:56:13.465Z + 2025-07-11T02:37:49.314Z https://docs.axolotl.ai/docs/api/core.trainers.mixins.scheduler.html - 2025-07-09T18:56:13.794Z + 2025-07-11T02:37:49.640Z https://docs.axolotl.ai/docs/api/utils.model_shard_quant.html - 2025-07-09T18:56:14.283Z + 2025-07-11T02:37:50.124Z https://docs.axolotl.ai/docs/api/prompt_strategies.kto.chatml.html - 2025-07-09T18:56:13.983Z + 2025-07-11T02:37:49.826Z https://docs.axolotl.ai/docs/api/utils.tokenization.html - 2025-07-09T18:56:14.270Z + 2025-07-11T02:37:50.112Z https://docs.axolotl.ai/docs/api/loaders.model.html - 2025-07-09T18:56:13.753Z + 2025-07-11T02:37:49.600Z https://docs.axolotl.ai/docs/api/utils.callbacks.mlflow_.html - 2025-07-09T18:56:14.751Z + 2025-07-11T02:37:50.583Z https://docs.axolotl.ai/docs/api/core.trainers.grpo.trainer.html - 2025-07-09T18:56:13.729Z + 2025-07-11T02:37:49.576Z https://docs.axolotl.ai/docs/api/cli.main.html - 2025-07-09T18:56:13.512Z + 2025-07-11T02:37:49.361Z https://docs.axolotl.ai/docs/api/utils.callbacks.comet_.html - 2025-07-09T18:56:14.754Z + 2025-07-11T02:37:50.587Z https://docs.axolotl.ai/docs/api/utils.chat_templates.html - 2025-07-09T18:56:14.272Z + 2025-07-11T02:37:50.113Z https://docs.axolotl.ai/docs/api/utils.schemas.utils.html - 2025-07-09T18:56:14.488Z + 2025-07-11T02:37:50.325Z https://docs.axolotl.ai/docs/api/common.architectures.html - 2025-07-09T18:56:14.648Z + 2025-07-11T02:37:50.482Z https://docs.axolotl.ai/docs/api/monkeypatch.llama_expand_mask.html - 2025-07-09T18:56:14.172Z + 2025-07-11T02:37:50.015Z https://docs.axolotl.ai/docs/api/prompt_strategies.alpaca_chat.html - 2025-07-09T18:56:13.867Z + 2025-07-11T02:37:49.712Z https://docs.axolotl.ai/docs/api/utils.samplers.multipack.html - 2025-07-09T18:56:14.736Z + 2025-07-11T02:37:50.568Z https://docs.axolotl.ai/docs/api/integrations.grokfast.optimizer.html - 2025-07-09T18:56:14.629Z + 2025-07-11T02:37:50.463Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.chatml.html - 2025-07-09T18:56:13.962Z + 2025-07-11T02:37:49.805Z https://docs.axolotl.ai/docs/api/monkeypatch.mixtral.html - 2025-07-09T18:56:14.232Z + 2025-07-11T02:37:50.074Z https://docs.axolotl.ai/docs/api/train.html - 2025-07-09T18:56:13.324Z + 2025-07-11T02:37:49.173Z https://docs.axolotl.ai/docs/api/monkeypatch.llama_patch_multipack.html - 2025-07-09T18:56:14.212Z + 2025-07-11T02:37:50.054Z https://docs.axolotl.ai/docs/api/index.html - 2025-07-09T18:56:13.261Z + 2025-07-11T02:37:49.111Z https://docs.axolotl.ai/docs/api/loaders.adapter.html - 2025-07-09T18:56:13.769Z + 2025-07-11T02:37:49.615Z https://docs.axolotl.ai/docs/api/utils.schemas.multimodal.html - 2025-07-09T18:56:14.460Z + 2025-07-11T02:37:50.296Z https://docs.axolotl.ai/docs/api/kernels.lora.html - 2025-07-09T18:56:14.092Z + 2025-07-11T02:37:49.935Z https://docs.axolotl.ai/docs/api/prompt_strategies.kto.llama3.html - 2025-07-09T18:56:13.975Z + 2025-07-11T02:37:49.818Z https://docs.axolotl.ai/docs/api/cli.checks.html - 2025-07-09T18:56:13.555Z + 2025-07-11T02:37:49.403Z https://docs.axolotl.ai/docs/api/cli.quantize.html - 2025-07-09T18:56:13.675Z + 2025-07-11T02:37:49.522Z https://docs.axolotl.ai/docs/api/integrations.lm_eval.args.html - 2025-07-09T18:56:14.643Z + 2025-07-11T02:37:50.477Z https://docs.axolotl.ai/docs/api/core.chat.messages.html - 2025-07-09T18:56:13.464Z + 2025-07-11T02:37:49.313Z https://docs.axolotl.ai/docs/api/core.builders.causal.html - 2025-07-09T18:56:13.421Z + 2025-07-11T02:37:49.269Z https://docs.axolotl.ai/docs/api/core.trainers.relora.html - 2025-07-09T18:56:13.712Z + 2025-07-11T02:37:49.559Z https://docs.axolotl.ai/docs/api/models.mamba.modeling_mamba.html - 2025-07-09T18:56:14.666Z + 2025-07-11T02:37:50.499Z https://docs.axolotl.ai/docs/api/monkeypatch.gradient_checkpointing.offload_cpu.html - 2025-07-09T18:56:14.237Z + 2025-07-11T02:37:50.079Z https://docs.axolotl.ai/docs/api/core.trainers.mamba.html - 2025-07-09T18:56:13.707Z + 2025-07-11T02:37:49.554Z https://docs.axolotl.ai/docs/api/core.datasets.transforms.chat_builder.html - 2025-07-09T18:56:13.481Z + 2025-07-11T02:37:49.330Z https://docs.axolotl.ai/docs/api/loaders.processor.html - 2025-07-09T18:56:13.763Z + 2025-07-11T02:37:49.609Z https://docs.axolotl.ai/docs/api/core.chat.format.llama3x.html - 2025-07-09T18:56:13.467Z + 2025-07-11T02:37:49.316Z https://docs.axolotl.ai/docs/api/prompt_strategies.messages.chat.html - 2025-07-09T18:56:13.940Z + 2025-07-11T02:37:49.783Z https://docs.axolotl.ai/docs/api/cli.train.html - 2025-07-09T18:56:13.521Z + 2025-07-11T02:37:49.369Z https://docs.axolotl.ai/docs/api/core.trainers.mixins.optimizer.html - 2025-07-09T18:56:13.784Z + 2025-07-11T02:37:49.630Z https://docs.axolotl.ai/docs/api/utils.collators.mamba.html - 2025-07-09T18:56:14.690Z + 2025-07-11T02:37:50.523Z https://docs.axolotl.ai/docs/api/monkeypatch.unsloth_.html - 2025-07-09T18:56:14.229Z + 2025-07-11T02:37:50.071Z https://docs.axolotl.ai/docs/api/utils.dict.html - 2025-07-09T18:56:14.360Z + 2025-07-11T02:37:50.199Z https://docs.axolotl.ai/docs/api/prompt_strategies.user_defined.html - 2025-07-09T18:56:13.889Z + 2025-07-11T02:37:49.733Z https://docs.axolotl.ai/docs/api/core.training_args.html - 2025-07-09T18:56:13.441Z + 2025-07-11T02:37:49.290Z https://docs.axolotl.ai/docs/api/prompt_strategies.dpo.user_defined.html - 2025-07-09T18:56:13.965Z + 2025-07-11T02:37:49.808Z https://docs.axolotl.ai/docs/api/prompt_tokenizers.html - 2025-07-09T18:56:13.400Z + 2025-07-11T02:37:49.249Z https://docs.axolotl.ai/docs/api/common.const.html - 2025-07-09T18:56:14.650Z + 2025-07-11T02:37:50.483Z https://docs.axolotl.ai/docs/fsdp_qlora.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/custom_integrations.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/getting-started.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.197Z https://docs.axolotl.ai/docs/faq.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/lora_optims.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/rlhf.html - 2025-07-09T18:52:52.315Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/amd_hpc.html - 2025-07-09T18:52:52.310Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/installation.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.199Z https://docs.axolotl.ai/docs/multipack.html - 2025-07-09T18:52:52.314Z + 2025-07-11T02:34:39.200Z https://docs.axolotl.ai/docs/dataset_preprocessing.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset_loading.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/inst_tune.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/template_free.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/index.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z https://docs.axolotl.ai/docs/dataset-formats/pretraining.html - 2025-07-09T18:52:52.311Z + 2025-07-11T02:34:39.196Z