axolotl

Files

Casper 15d3a654bf Implement fused modules (#747 )

* MLP: Memory saving

* Remove RMSNorm restrictions

* Map packed weights to original

* FusedAttention module

* Simplify code

* Move fused modules

* Fix critical typo

* Split inplace

* Add FFT config

* Add validation of fused arguments

* Add fused arguments to config

* Update docs

* Fix validation logic

* Add fused modules to flash attn

* Only fuse during training

* Remove timing

* Formatting

* Formatting

* Formatting

* chore: lint

* chore: lint

* add e2e tests for fused llama

* no lora for tests

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>

2023-10-21 16:08:25 -04:00

cerebras

prepared dataset caching, other misc fixes (#665 )

2023-10-02 21:07:24 -04:00

code-llama

prepared dataset caching, other misc fixes (#665 )

2023-10-02 21:07:24 -04:00

falcon

prepared dataset caching, other misc fixes (#665 )

2023-10-02 21:07:24 -04:00

gptj

prepared dataset caching, other misc fixes (#665 )