axolotl

Files

Casper 15d3a654bf Implement fused modules (#747 )

* MLP: Memory saving

* Remove RMSNorm restrictions

* Map packed weights to original

* FusedAttention module

* Simplify code

* Move fused modules

* Fix critical typo

* Split inplace

* Add FFT config

* Add validation of fused arguments

* Add fused arguments to config

* Update docs

* Fix validation logic

* Add fused modules to flash attn

* Only fuse during training

* Remove timing

* Formatting

* Formatting

* Formatting

* chore: lint

* chore: lint

* add e2e tests for fused llama

* no lora for tests

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>

2023-10-21 16:08:25 -04:00

e2e

Implement fused modules (#747 )

2023-10-21 16:08:25 -04:00

fixtures

misc sharegpt fixes (#723 )

2023-10-13 11:04:39 -04:00

monkeypatch

Attention mask and position id fixes for packing (#285 )

2023-08-12 15:14:56 -04:00

test_data.py

Fix pretraining with iterable/streaming Dataset (#556 )