Phi-3 conversation format, example training script and perplexity metric (#1582)

* phi-3 support and perplexity metric

* phi-3 chat template

* metrics updates

* chore: lint

* fix assertion on Tensor

* fix tests since tokenization happens in the metric

* fix perplexity value of shorter passage

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>
This commit is contained in:
Brian Fitzgerald
2024-06-04 15:11:56 -05:00
committed by GitHub
parent c996881ec2
commit cf64284a04
10 changed files with 243 additions and 26 deletions

6
.gitignore vendored
View File

@@ -176,3 +176,9 @@ qlora-out/*
mlruns/*
/.quarto/
prepared-datasets/
submit.sh
*.out*
typings/
out/