Wing Lian
b3289fd190
feat: LoRA kernel support for bias, dropout, dora, embeddings ( #3528 ) [skip ci]
...
* feat: LoRA kernel support for bias, dropout, dora, embeddings
* chore: lint
* chore: lint
* address PR feedback, add regression tests, add fsdp2 tests for lora kernels
* update tests for new sigs
* update tests now that bias and dropout are supported
2026-03-22 13:53:19 -04:00
Dan Saunders
79ddaebe9a
Add ruff, remove black, isort, flake8, pylint ( #3092 )
...
* black, isort, flake8 -> ruff
* remove unused
* add back needed import
* fix
2025-08-23 23:37:33 -04:00
Dan Saunders
d09290f2f4
Lora kernels bias support ( #3025 )
...
* lora kernels bias support
* revert rename
* nit
* lint, tests
* satisfying the rabbit
2025-08-06 20:20:08 -04:00
Wing Lian
de2c5ba103
mark flaky geglu tests and add torch seed ( #2876 ) [skip ci]
...
* mark flaky geglu tests and add torch seed
* restore accidental removal of seed
2025-07-07 15:24:16 -04:00
Wing Lian
7909bfb076
add manual seed for flaky test_geglu_backward test ( #2763 ) [skip ci]
2025-06-05 09:23:17 -07:00
Dan Saunders
c907ac173e
adding pre-commit auto-update GH action and bumping plugin versions ( #2428 )
...
* adding pre-commit auto-update GH action and bumping plugin versions
* running updated pre-commit plugins
* sorry to revert, but pylint complained
* Update .pre-commit-config.yaml
Co-authored-by: Wing Lian <wing.lian@gmail.com >
---------
Co-authored-by: Dan Saunders <dan@axolotl.ai >
Co-authored-by: Wing Lian <wing.lian@gmail.com >
2025-03-21 11:02:43 -04:00
Dan Saunders
3d8425fa91
Activation function Triton kernels, LoRA custom autograd functions ( #2324 )
...
* LoRA + activation fn Triton kernels: initial commit
* implementing optims
* finalizing MLP LoRA kernels and progress on QKV / W kernels
* updates
* O projection optim
* adding monkey patching logic
* doc strings, typing, pre-commit fixes
* updates
* adding lora 8b kernels example
* working on fsdp support
* tests and fixes
* small fixes, getting tests to pass, adding doc strings
* integration tests for LoRA patching
* config.qmd
* remove unneeded pytest fixture
* fix
* review comments first pass
* improving tests, attention class agnostic patching
* adding support for more archs
* wip SiLU / GELU impls
* improved testing, small updates, etc.
* slightly updating docs
* rebase
* fixing test_attention_patching_integration
* additional review comments, fixing test in CI (hopefully)
* isolating problematic patching test
* relaxing allclose threshold to reduce flakiness
* fixing accidental change
* adding model arch agnostic attention class fetching
* removing unused activations
2025-02-17 14:23:15 -05:00
