2 Commits

Author	SHA1	Message	Date
NanoCode012	7da5f94379	feat: add FA4 (#3481) ... * feat: add FA4 * chore: update docs * fix: recommend FA4 for those with compatible devices * fix: adjust import check and add head_dim check * chore: add limitation to doc * fix: log warning and quit if cannot import validator * chore: simplify * fix: add caveat with FA2 shadow dir	2026-03-16 00:13:18 -04:00
NanoCode012	fcc4cfdb63	feat: add sageattention (#2823) [skip ci] ... * feat: add sageattention * feat: call path on pre model load * fix: patch to use register to correct var * fix: add strict check import at start * chore: fix comments * chore: refactor * feat: add capability check * fix: missed underscore * fix: let sageattention use FA backend in transformers * feat: update sage attention for attention mask and position ids * feat: allow sample packing but add warning without packing * fix: loss hitting 0 with packing and attention mask note * feat: downcast embeds if sage attention too * feat: add config validation * feat: add attention docs * chore: docs	2026-02-10 17:49:21 +07:00