axolotl

Files

TearGosling f4746507f6 feat: add Metharme prompt strategy (#446 )

* Add Metharme tokenizing strategy

This strategy accounts for how the Metharme JSONLs are formatted as well as adds duplicated EOS tokens which can help trim model output length.
I haven't gotten the chance to test this yet, and probably won't have the chance for quite a bit, so I'm committing this now.

* Redo Metharme tokenizing strategy

lol

* fix: oops

* Rearrange a conditional

* chore: reformat code in accordance with linter

* chore: Make lint not freak out

* chore: fix lint

---------

Co-authored-by: NanoCode012 <kevinvong@rocketmail.com>

2023-08-22 11:21:45 +09:00

__init__.py

support user defined prompters, pretokenized datasets in config, local parquet, local arrow files (#348 )

2023-08-20 09:17:49 -04:00

alpaca_chat.py

Merge pull request #224 from OpenAccess-AI-Collective/system-prompt-data

2023-06-27 17:57:43 -04:00

alpaca_instruct.py

skip the system prompt

2023-06-25 22:40:50 -04:00

alpaca_w_system.py

support user defined prompters, pretokenized datasets in config, local parquet, local arrow files (#348 )