Brian Fitzgerald cf64284a04 Phi-3 conversation format, example training script and perplexity metric (#1582) ...

* phi-3 support and perplexity metric

* phi-3 chat template

* metrics updates

* chore: lint

* fix assertion on Tensor

* fix tests since tokenization happens in the metric

* fix perplexity value of shorter passage

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>

2024-06-04 16:11:56 -04:00

2.7 KiB

Raw Permalink Blame History

View Raw