tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Fix bug when using pretokenized datasets (#652)

Browse Source 
* fix pretokenized datasets readme

* check if dataset type is not set to handle pretokenized datasets
This commit is contained in:
ich
2023-09-29 04:54:10 +02:00
committed by GitHub
parent 409ca0f21c
commit 590d6032fd
2 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -317,7 +317,7 @@ Using file:
#### How to use your custom pretokenized dataset #### How to use your custom pretokenized dataset
- Do not pass a `type:` - Do not pass a `type:`
- Dataset must contain `input_ids`, `attention_mask`, `labels` in columns - Columns in Dataset must be exactly `input_ids`, `attention_mask`, `labels`
### Config ### Config

2
src/axolotl/utils/config.py
View File

@@ -293,6 +293,8 @@ def validate_config(cfg):
if cfg.datasets: if cfg.datasets:
for idx, ds_cfg in enumerate(cfg.datasets): for idx, ds_cfg in enumerate(cfg.datasets):
if not ds_cfg.type:
continue
if ds_cfg.type == "sharegpt:chat": if ds_cfg.type == "sharegpt:chat":
LOG.warning( LOG.warning(
PendingDeprecationWarning( PendingDeprecationWarning(