Dan Saunders 23f0c51d88 Sequence parallelism (#2412) ...

* adding easy_context as integration for now

* progress on ring attn impl

* progress on ring attn impl

* cleanup

* remove errant file

* fix req

* removing unused code

* updates

* pytest

* update

* updates

* fixes

* precommit fixes

* working multi-group SP

* fixing sample packing

* remove debug logs and simplify

* eval dataloader and sampler changes

* removing some obvious comments

* update config.qmd and rename option

* scoping down problematic import

* another import scoping change

* pernicious Fire CLI bugfix

* isolate cli tests

* actually isolate CLI tests

* gracefully handle no ring-flash-attn

* fix

* fix

* move ring flash attn to extras with flash-attn (#2414)

* removing flash-attn from requirements.txt (in setup.py extras already)

* rename file, delete another

* using field validator instead of model validator

* test fix

* sampler / dataloader refactor

* non-seq2se1 collator fix

* removing print statement

* bugfix

* add SP doc, review comments

* small changes

* review comments, docstrings

* refactors, SP mixin

* small updates

* fix tests

* precommit

* precommit

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>
Co-authored-by: Dan Saunders <dan@axolotl.ai>

2025-03-21 12:43:55 -04:00

..

dataset-formats

chore(doc): add faq when having no default chat_template (#2398)

2025-03-10 16:25:50 +07:00

images

Ray Train Axolotl Integration (#2251)

2025-01-29 00:10:19 -05:00

.gitignore

Autodoc generation with quartodoc (#2419)

2025-03-21 12:26:47 -04:00

amd_hpc.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

batch_vs_grad.qmd

Feat: update doc (#1475) [skip ci]

2024-04-04 13:43:40 +09:00

cli.qmd

Autodoc generation with quartodoc (#2419)

2025-03-21 12:26:47 -04:00

config.qmd

Sequence parallelism (#2412)

2025-03-21 12:43:55 -04:00

custom_integrations.qmd

chore(doc): add instructions on adding custom integrations (#2422) [skip ci]

2025-03-21 10:18:01 -04:00

dataset_preprocessing.qmd

Autodoc generation with quartodoc (#2419)

2025-03-21 12:26:47 -04:00

debugging.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

docker.qmd

feat(doc): add docker images explanation (#2379) [skip ci]

2025-03-05 10:01:00 -05:00

faq.qmd

chore(doc): add explanation on fsdp_transformer_layer_cls_to_wrap (#2429) [skip ci]

2025-03-21 11:59:22 -04:00

fsdp_qlora.qmd

github urls (#1734)

2024-07-11 09:19:29 -04:00

getting-started.qmd

Feat: minor docs improvements for RLHF and faq on embeddings (#2401) [skip ci]

2025-03-17 08:39:04 -04:00

inference.qmd

Feat: minor docs improvements for RLHF and faq on embeddings (#2401) [skip ci]

2025-03-17 08:39:04 -04:00

input_output.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

installation.qmd

add run on novita ai (#2421) [skip ci]

2025-03-21 10:17:47 -04:00

lora_optims.qmd

chore(docs): add cookbook/blog link to docs (#2410) [skip ci]

2025-03-17 08:38:19 -04:00

lr_groups.qmd

support for custom lr groups for non-embedding modules (#2213)

2025-01-24 12:56:28 -05:00

mac.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

multi-gpu.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

multi-node.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

multimodal.qmd

Multimodal Vision Llama - rudimentary support (#1940)

2024-10-02 21:02:48 -04:00

multipack.qmd

Bootstrap Hosted Axolotl Docs w/Quarto (#1429)

2024-03-21 22:28:36 -07:00

nccl.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

ray-integration.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

reward_modelling.qmd

chore(docs): add cookbook/blog link to docs (#2410) [skip ci]

2025-03-17 08:38:19 -04:00

rlhf.qmd

Feat: minor docs improvements for RLHF and faq on embeddings (#2401) [skip ci]

2025-03-17 08:39:04 -04:00

sequence_parallelism.qmd

Sequence parallelism (#2412)

2025-03-21 12:43:55 -04:00

torchao.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00

unsloth.qmd

Feat(doc): Reorganize documentation, fix broken syntax, update notes (#2348)

2025-02-25 16:09:37 +07:00