Dan Saunders 3d8425fa91 Activation function Triton kernels, LoRA custom autograd functions (#2324) ...

* LoRA + activation fn Triton kernels: initial commit

* implementing optims

* finalizing MLP LoRA kernels and progress on QKV / W kernels

* updates

* O projection optim

* adding monkey patching logic

* doc strings, typing, pre-commit fixes

* updates

* adding lora 8b kernels example

* working on fsdp support

* tests and fixes

* small fixes, getting tests to pass, adding doc strings

* integration tests for LoRA patching

* config.qmd

* remove unneeded pytest fixture

* fix

* review comments first pass

* improving tests, attention class agnostic patching

* adding support for more archs

* wip SiLU / GELU impls

* improved testing, small updates, etc.

* slightly updating docs

* rebase

* fixing test_attention_patching_integration

* additional review comments, fixing test in CI (hopefully)

* isolating problematic patching test

* relaxing allclose threshold to reduce flakiness

* fixing accidental change

* adding model arch agnostic attention class fetching

* removing unused activations

2025-02-17 14:23:15 -05:00

..

dataset-formats

feat(doc): Improve guide to dataset types with better examples (#2286)

2025-02-13 16:01:41 -05:00

images

Ray Train Axolotl Integration (#2251)

2025-01-29 00:10:19 -05:00

.gitignore

Bootstrap Hosted Axolotl Docs w/Quarto (#1429)

2024-03-21 22:28:36 -07:00

amd_hpc.qmd

fix build w pyproject to respect insalled torch version (#2168)

2024-12-10 16:25:25 -05:00

batch_vs_grad.qmd

Feat: update doc (#1475) [skip ci]

2024-04-04 13:43:40 +09:00

cli.qmd

native support for modal cloud from CLI (#2237)

2025-01-30 11:34:02 -05:00

config.qmd

Activation function Triton kernels, LoRA custom autograd functions (#2324)

2025-02-17 14:23:15 -05:00

dataset_preprocessing.qmd

add docs around pre-processing (#1529)

2024-04-16 19:45:46 -04:00

debugging.qmd

fix build w pyproject to respect insalled torch version (#2168)

2024-12-10 16:25:25 -05:00

faq.qmd

lint docs (#2327)

2025-02-12 10:04:26 -05:00

fsdp_qlora.qmd

github urls (#1734)

2024-07-11 09:19:29 -04:00

getting-started.qmd

refactor README; hardcode links to quarto docs; add additional quarto doc pages (#2295)

2025-01-30 12:49:21 -05:00

inference.qmd

refactor README; hardcode links to quarto docs; add additional quarto doc pages (#2295)

2025-01-30 12:49:21 -05:00

input_output.qmd

Multimodal Vision Llama - rudimentary support (#1940)

2024-10-02 21:02:48 -04:00

installation.qmd

refactor README; hardcode links to quarto docs; add additional quarto doc pages (#2295)

2025-01-30 12:49:21 -05:00

lora_optims.qmd

Activation function Triton kernels, LoRA custom autograd functions (#2324)

2025-02-17 14:23:15 -05:00

lr_groups.qmd

support for custom lr groups for non-embedding modules (#2213)

2025-01-24 12:56:28 -05:00

mac.qmd

Bootstrap Hosted Axolotl Docs w/Quarto (#1429)

2024-03-21 22:28:36 -07:00

multi-gpu.qmd

refactor README; hardcode links to quarto docs; add additional quarto doc pages (#2295)

2025-01-30 12:49:21 -05:00

multi-node.qmd

feat(doc): Add multi-node torchrun info (#2304)

2025-02-08 06:02:02 -05:00

multimodal.qmd

Multimodal Vision Llama - rudimentary support (#1940)

2024-10-02 21:02:48 -04:00

multipack.qmd

Bootstrap Hosted Axolotl Docs w/Quarto (#1429)

2024-03-21 22:28:36 -07:00

nccl.qmd

Bootstrap Hosted Axolotl Docs w/Quarto (#1429)

2024-03-21 22:28:36 -07:00

ray-integration.qmd

Ray Train Axolotl Integration (#2251)

2025-01-29 00:10:19 -05:00

reward_modelling.qmd

Process reward models (#2241)

2025-01-29 00:08:33 -05:00

rlhf.qmd

feat(doc): Improve guide to dataset types with better examples (#2286)

2025-02-13 16:01:41 -05:00

torchao.qmd

bump transformers and set roundup_power2_divisions for more VRAM improvements, low bit ao optimizers (#1769)

2024-07-19 00:47:07 -04:00

unsloth.qmd

fix inference when no chat_template is set, fix unsloth dora check (#2092)

2024-11-20 14:07:54 -05:00