From 66a3de36292e6a6afa5e0916c8392e7db41ba822 Mon Sep 17 00:00:00 2001 From: Wing Lian Date: Thu, 25 Dec 2025 07:17:25 -0500 Subject: [PATCH] build examples readmes with quarto (#3046) * build examples readmes with quarto * chore: formatting * feat: dynamic build docs * feat: add more model guides * chore: format * fix: collapse sidebar completely to have space for model guides * fix: security protection for generated qmd * fix: adjust collapse level, add new models, update links --------- Co-authored-by: NanoCode012 --- README.md | 12 +- _quarto.yml | 44 +- docs/.gitignore | 2 + docs/scripts/examples-allowlist.yml | 90 ++++ docs/scripts/generate_examples_docs.py | 424 ++++++++++++++++++ examples/magistral/think/README.md | 1 + examples/magistral/vision/README.md | 3 +- examples/ministral3/think/README.md | 1 + examples/ministral3/vision/README.md | 3 +- .../{mistral => }/mistral-small/README.md | 1 + .../mistral-small-3.1-24B-lora.yml | 0 11 files changed, 572 insertions(+), 9 deletions(-) create mode 100644 docs/scripts/examples-allowlist.yml create mode 100755 docs/scripts/generate_examples_docs.py rename examples/{mistral => }/mistral-small/README.md (99%) rename examples/{mistral => }/mistral-small/mistral-small-3.1-24B-lora.yml (100%) diff --git a/README.md b/README.md index 6b9a6f84b..01e0c44d9 100644 --- a/README.md +++ b/README.md @@ -29,15 +29,15 @@ ## 🎉 Latest Updates -- 2025/12: Axolotl now includes support for [Kimi-Linear](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/kimi-linear), [Plano-Orchestrator](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/plano), [MiMo](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/mimo), [InternVL 3.5](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/internvl3_5), [Olmo3](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/olmo3), [Trinity](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/trinity), and [Ministral3](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/ministral3). -- 2025/10: New model support has been added in Axolotl for: [Qwen3 Next](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/qwen3-next), [Qwen2.5-vl, Qwen3-vl](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/qwen2_5-vl), [Qwen3, Qwen3MoE](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/qwen3), [Granite 4](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/granite4), [HunYuan](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/hunyuan), [Magistral 2509](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/magistral#vision), [Apertus](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/apertus), and [Seed-OSS](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/seed-oss). +- 2025/12: Axolotl now includes support for [Kimi-Linear](https://docs.axolotl.ai/docs/models/kimi-linear.html), [Plano-Orchestrator](https://docs.axolotl.ai/docs/models/plano.html), [MiMo](https://docs.axolotl.ai/docs/models/mimo.html), [InternVL 3.5](https://docs.axolotl.ai/docs/models/internvl3_5.html), [Olmo3](https://docs.axolotl.ai/docs/models/olmo3.html), [Trinity](https://docs.axolotl.ai/docs/models/trinity.html), and [Ministral3](https://docs.axolotl.ai/docs/models/ministral3.html). +- 2025/10: New model support has been added in Axolotl for: [Qwen3 Next](https://docs.axolotl.ai/docs/models/qwen3-next.html), [Qwen2.5-vl, Qwen3-vl](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/qwen2_5-vl), [Qwen3, Qwen3MoE](https://docs.axolotl.ai/docs/models/qwen3.html), [Granite 4](https://docs.axolotl.ai/docs/models/granite4.html), [HunYuan](https://docs.axolotl.ai/docs/models/hunyuan.html), [Magistral 2509](https://docs.axolotl.ai/docs/models/magistral/vision.html), [Apertus](https://docs.axolotl.ai/docs/models/apertus.html), and [Seed-OSS](https://docs.axolotl.ai/docs/models/seed-oss.html). - 2025/09: Axolotl now has text diffusion training. Read more [here](https://github.com/axolotl-ai-cloud/axolotl/tree/main/src/axolotl/integrations/diffusion). - 2025/08: QAT has been updated to include NVFP4 support. See [PR](https://github.com/axolotl-ai-cloud/axolotl/pull/3107). - 2025/07: - ND Parallelism support has been added into Axolotl. Compose Context Parallelism (CP), Tensor Parallelism (TP), and Fully Sharded Data Parallelism (FSDP) within a single node and across multiple nodes. Check out the [blog post](https://huggingface.co/blog/accelerate-nd-parallel) for more info. - - Axolotl adds more models: [GPT-OSS](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/gpt-oss), [Gemma 3n](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/gemma3n), [Liquid Foundation Model 2 (LFM2)](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/lfm2), and [Arcee Foundation Models (AFM)](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/afm). + - Axolotl adds more models: [GPT-OSS](https://docs.axolotl.ai/docs/models/gpt-oss.html), [Gemma 3n](https://docs.axolotl.ai/docs/models/gemma3n.html), [Liquid Foundation Model 2 (LFM2)](https://docs.axolotl.ai/docs/models/LiquidAI.html), and [Arcee Foundation Models (AFM)](https://docs.axolotl.ai/docs/models/arcee.html). - FP8 finetuning with fp8 gather op is now possible in Axolotl via `torchao`. Get started [here](https://docs.axolotl.ai/docs/mixed_precision.html#sec-fp8)! - - [Voxtral](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/voxtral), [Magistral 1.1](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/magistral), and [Devstral](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/devstral) with mistral-common tokenizer support has been integrated in Axolotl! + - [Voxtral](https://docs.axolotl.ai/docs/models/voxtral.html), [Magistral 1.1](https://docs.axolotl.ai/docs/models/magistral.html), and [Devstral](https://docs.axolotl.ai/docs/models/devstral.html) with mistral-common tokenizer support has been integrated in Axolotl! - TiledMLP support for single-GPU to multi-GPU training with DDP, DeepSpeed and FSDP support has been added to support Arctic Long Sequence Training. (ALST). See [examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/alst) for using ALST with Axolotl! - 2025/05: Quantization Aware Training (QAT) support has been added to Axolotl. Explore the [docs](https://docs.axolotl.ai/docs/qat.html) to learn more! @@ -46,8 +46,8 @@ Expand older updates - 2025/03: Axolotl has implemented Sequence Parallelism (SP) support. Read the [blog](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl) and [docs](https://docs.axolotl.ai/docs/sequence_parallelism.html) to learn how to scale your context length when fine-tuning. -- 2025/06: Magistral with mistral-common tokenizer support has been added to Axolotl. See [examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/magistral) to start training your own Magistral models with Axolotl! -- 2025/04: Llama 4 support has been added in Axolotl. See [examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/llama-4) to start training your own Llama 4 models with Axolotl's linearized version! +- 2025/06: Magistral with mistral-common tokenizer support has been added to Axolotl. See [docs](https://docs.axolotl.ai/docs/models/magistral.html) to start training your own Magistral models with Axolotl! +- 2025/04: Llama 4 support has been added in Axolotl. See [docs](https://docs.axolotl.ai/docs/models/llama-4.html) to start training your own Llama 4 models with Axolotl's linearized version! - 2025/03: (Beta) Fine-tuning Multimodal models is now supported in Axolotl. Check out the [docs](https://docs.axolotl.ai/docs/multimodal.html) to fine-tune your own! - 2025/02: Axolotl has added LoRA optimizations to reduce memory usage and improve training speed for LoRA and QLoRA in single GPU and multi-GPU training (DDP and DeepSpeed). Jump into the [docs](https://docs.axolotl.ai/docs/lora_optims.html) to give it a try. - 2025/02: Axolotl has added GRPO support. Dive into our [blog](https://huggingface.co/blog/axolotl-ai-co/training-llms-w-interpreter-feedback-wasm) and [GRPO example](https://github.com/axolotl-ai-cloud/grpo_code) and have some fun! diff --git a/_quarto.yml b/_quarto.yml index c97b9838e..fe3a76e53 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -1,6 +1,8 @@ project: type: website - pre-render: docs/scripts/generate_config_docs.py + pre-render: + - docs/scripts/generate_config_docs.py + - docs/scripts/generate_examples_docs.py quartodoc: dir: docs/api @@ -240,6 +242,46 @@ website: - docs/getting-started.qmd - docs/installation.qmd - docs/inference.qmd + - section: "Model Guides" + contents: + - docs/models/kimi-linear.qmd + - docs/models/plano.qmd + - docs/models/mimo.qmd + - docs/models/internvl3_5.qmd + - docs/models/olmo3.qmd + - docs/models/trinity.qmd + - docs/models/arcee.qmd + - docs/models/mistral.qmd + - section: "Ministral3" + contents: + - docs/models/ministral3.qmd + - docs/models/ministral3/think.qmd + - docs/models/ministral3/vision.qmd + - section: "Magistral" + contents: + - docs/models/magistral.qmd + - docs/models/magistral/think.qmd + - docs/models/magistral/vision.qmd + - docs/models/ministral.qmd + - docs/models/mistral-small.qmd + - docs/models/voxtral.qmd + - docs/models/devstral.qmd + - docs/models/llama-4.qmd + - docs/models/llama-2.qmd + - docs/models/qwen3-next.qmd + - docs/models/qwen3.qmd + - docs/models/gemma3n.qmd + - docs/models/apertus.qmd + - docs/models/gpt-oss.qmd + - docs/models/seed-oss.qmd + - docs/models/phi.qmd + - docs/models/smolvlm2.qmd + - docs/models/granite4.qmd + - docs/models/LiquidAI.qmd + - docs/models/hunyuan.qmd + - docs/models/jamba.qmd + - docs/models/orpheus.qmd + - docs/cli.qmd - docs/telemetry.qmd - docs/config-reference.qmd diff --git a/docs/.gitignore b/docs/.gitignore index 89407326f..94e5b88fb 100644 --- a/docs/.gitignore +++ b/docs/.gitignore @@ -3,3 +3,5 @@ _site/ /api/*.qmd /api/*.html config-reference.qmd +models/**/*.qmd +models/**/*.html diff --git a/docs/scripts/examples-allowlist.yml b/docs/scripts/examples-allowlist.yml new file mode 100644 index 000000000..50acaea8e --- /dev/null +++ b/docs/scripts/examples-allowlist.yml @@ -0,0 +1,90 @@ +examples: + # December 2025 + - name: kimi-linear + title: Kimi Linear + - name: plano + title: Plano Orchestrator + - name: mimo + title: MiMo + - name: internvl3_5 + title: InternVL 3.5 + + # AllenAI + - name: olmo3 + title: OLMo 3 + + # ArceeAI + - name: trinity + title: Trinity + - name: arcee + title: Arcee AFM + + # MistralAI + - name: ministral3/think + title: Ministral 3 Thinking + - name: ministral3/vision + title: Ministral 3 Vision + - name: magistral/think + title: Magistral Thinking + - name: magistral/vision + title: Magistral Vision + - name: ministral + title: Ministral + - name: mistral-small + title: Mistral Small 3.1/3.2 + - name: voxtral + title: Voxtral + - name: devstral + title: Devstral + - name: mistral + title: Mistral 7B + + # Meta + - name: llama-4 + title: Llama 4 + - name: llama-2 + title: Llama 2 + + # Alibaba + - name: qwen3-next + title: Qwen 3 Next + - name: qwen3 + title: Qwen 3 + + # Google + - name: gemma3n + title: Gemma 3n + + # Swiss AI + - name: apertus + title: Apertus + + # GPT-OSS + - name: gpt-oss + title: GPT-OSS + - name: seed-oss + title: Seed-OSS + + # Microsoft + - name: phi + title: Phi + + # SmolVLM + - name: smolvlm2 + title: SmolVLM 2 + + # IBM + - name: granite4 + title: Granite 4 + + # LiquidAI + - name: LiquidAI + title: Liquid Foundation Models 2 + + # Other + - name: hunyuan + title: Hunyuan + - name: jamba + title: Jamba + - name: orpheus + title: Orpheus diff --git a/docs/scripts/generate_examples_docs.py b/docs/scripts/generate_examples_docs.py new file mode 100755 index 000000000..ba01088f1 --- /dev/null +++ b/docs/scripts/generate_examples_docs.py @@ -0,0 +1,424 @@ +""" +auto generate example docs from allowlist +""" + +import re +import shutil +import sys +from pathlib import Path + +import yaml + +# Paths +THIS = Path(__file__).resolve() +ROOT = THIS.parents[2] # repo root (docs/scripts -> docs -> ROOT) +EXAMPLES_DIR = ROOT / "examples" +OUTPUT_DIR = ROOT / "docs" / "models" +ALLOWLIST_YML = THIS.parent / "examples-allowlist.yml" + + +def slugify(name: str) -> str: + """Convert a name to a slug (lowercase, hyphens for spaces).""" + s = re.sub(r"[^a-zA-Z0-9\s\-]+", "", name.strip()) + s = re.sub(r"\s+", "-", s).strip("-").lower() + return s or "example" + + +def read_allowlist(): + with open(ALLOWLIST_YML, "r", encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + items = data.get("examples", []) + if not isinstance(items, list): + raise ValueError("`examples` must be a list in examples-allowlist.yml") + return items + + +def find_readme(folder: Path) -> Path | None: + for name in ("README.md", "Readme.md", "readme.md"): + p = folder / name + if p.exists(): + return p + return None + + +def remove_first_h1(md: str) -> tuple[str, str | None]: + """ + Remove the first H1 from markdown and return (modified_md, h1_title). + The H1 is removed since we use the frontmatter title instead. + """ + lines = md.splitlines() + result = [] + h1_title = None + skipped_first = False + + for line in lines: + if not skipped_first and line.startswith("# "): + h1_title = line[2:].strip() + skipped_first = True + continue + result.append(line) + + return "\n".join(result), h1_title + + +IMG_RE = re.compile(r"!\[[^\]]*\]\(([^)]+)\)") +LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") + + +def rewrite_and_copy_assets(md: str, src_dir: Path, dest_assets_root: Path) -> str: + """ + Copy local image assets referenced in markdown to + docs/examples/assets/... and rewrite the links. + """ + dest_assets = dest_assets_root / "assets" + + def repl(m): + url = m.group(1).strip() + if re.match(r"^(https?:)?//", url): + return m.group(0) # leave remote URLs + src_path = (src_dir / url).resolve() + if not src_path.exists(): + return m.group(0) # leave as-is if not found + rel = src_path.relative_to(src_dir) + # Create a unique asset path based on source directory name + asset_name = src_dir.name.replace("/", "-") + dest_path = dest_assets / asset_name / rel + dest_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src_path, dest_path) + new_rel = f"assets/{asset_name}/{rel.as_posix()}" + return m.group(0).replace(url, new_rel) + + return IMG_RE.sub(repl, md) + + +def rewrite_readme_links( + md: str, + src_dir: Path, + examples_dir: Path, + parent_index_only: set, + current_src_path: str, + allowlist_entries: set, + current_output_path: str, +) -> str: + """ + Rewrite links between README.md files to point to the correct .qmd files. + """ + + def repl(m): + text = m.group(1) + url = m.group(2).strip() + + # Skip remote URLs and anchor links + if re.match(r"^(https?:)?//", url) or url.startswith("#"): + return m.group(0) + + # Skip non-markdown files + if not url.lower().endswith(".md"): + return m.group(0) + + # Resolve the target path + try: + target_path = (src_dir / url).resolve() + + # Check if target is outside examples_dir + try: + rel_path = target_path.relative_to(examples_dir) + except ValueError: + # Target is outside examples_dir, leave as-is + return m.group(0) + + parts = list(rel_path.parts) + + # Determine the output path for the target + if len(parts) > 0 and parts[-1].lower() in ("readme.md", "readme"): + # This is a README link + if len(parts) == 1: + # Link to root README -> index.qmd + target_output = "index.qmd" + elif len(parts) == 2: + if parts[0] == ".": + # Current directory README + target_output = "index.qmd" + else: + # subdir/README.md + parent_dir = parts[0] + if parent_dir in parent_index_only: + target_output = f"{parent_dir}/index.qmd" + else: + target_output = f"{parent_dir}.qmd" + else: + # Deeper nesting: parent/subdir/README.md + # Build the full path like "parent/subdir" + full_path = "/".join(parts[:-1]) # Remove README.md + # Check if this exact path is in allowlist + if full_path in allowlist_entries: + # This is a sub-entry with its own entry -> use .qmd + target_output = f"{full_path}.qmd" + elif parts[0] == ".": + # ./subdir/README.md -> check if subdir has own entry + subdir = parts[1] + if subdir in parent_index_only: + target_output = f"{subdir}/index.qmd" + else: + target_output = f"{subdir}.qmd" + else: + # parent/subdir where parent doesn't have own entry + target_output = f"{full_path}/index.qmd" + else: + # Regular .md file -> convert to .qmd, keep path structure + target_output = "/".join(parts)[:-2] + "qmd" + + # Compute relative path from current output file to target + current_parts = current_output_path.split("/") + target_parts = target_output.split("/") + + # Special case: if current is a subdir file and target is a single-component file at root + # Example: current="magistral/vision", target="magistral.qmd" + if len(current_parts) > 1 and len(target_parts) == 1: + # Current is in subdir, target is at root level + # Go up to root: ../ for each level + up_count = len(current_parts) - 1 + rel_parts = [".."] * up_count + [target_parts[0]] + new_url = "/".join(rel_parts) + else: + # Find common prefix + i = 0 + while ( + i < min(len(current_parts) - 1, len(target_parts)) + and current_parts[i] == target_parts[i] + ): + i += 1 + + # Build relative path: go up (../) then down to target + up_count = len(current_parts) - 1 - i + rel_parts = [".."] * up_count + target_parts[i:] + + if not rel_parts or rel_parts == [".."]: + # Points to same directory or parent + new_url = "/".join(rel_parts) if rel_parts else "." + else: + new_url = "/".join(rel_parts) + + return f"[{text}]({new_url})" + except (ValueError, IndexError): + return m.group(0) + + return LINK_RE.sub(repl, md) + + +def write_qmd(out_path: Path, title: str, body_md: str): + out_path.parent.mkdir(parents=True, exist_ok=True) + fm = f"---\ntitle: {title!r}\nexecute:\n eval: false\nformat:\n html:\n toc: true\n---\n\n" + out_path.write_text(fm + body_md, encoding="utf-8") + + +def update_quarto_yml(generated: list[tuple[str, str, str]]): + """ + Update _quarto.yml with the generated example files in the correct order. + This keeps the sidebar in sync with the allowlist. + + Model Guides is now nested under "Getting Started" section. + Creates nested sections for models with sub-entries (e.g., magistral, ministral3). + Parent pages are now flat files (e.g., ministral3.qmd) with sub-pages in subdirs. + """ + quarto_yml = ROOT / "_quarto.yml" + if not quarto_yml.exists(): + print(f"[WARN] {quarto_yml} not found, skipping update", file=sys.stderr) + return + + content = quarto_yml.read_text(encoding="utf-8") + + # First pass: find all parents that have sub-entries + parents_with_subs = set() + for path, _name, _title in generated: + if "/" in path: + parent = path.split("/")[0] + parents_with_subs.add(parent) + + # Build the YAML contents while preserving allowlist order + lines = [] + processed_sections = set() + + for path, _name, title in generated: + # Check if this is a parent page that has sub-pages + if path in parents_with_subs: + # This is a parent page with sub-pages - create a nested section + if path not in processed_sections: + processed_sections.add(path) + section_title = ( + title or path.replace("-", " ").replace("_", " ").title() + ) + lines.append(f' - section: "{section_title}"') + lines.append(" contents:") + # Add the parent page first + lines.append(f" - docs/models/{path}.qmd") + # Then add all sub-pages + for sub_path, _sub_name, _sub_title in generated: + if "/" in sub_path and sub_path.split("/")[0] == path: + lines.append( + f" - docs/models/{sub_path}.qmd" + ) + elif "/" not in path: + # This is a flat item with no sub-pages + # Skip if it was already included as part of a parent section + if path not in processed_sections: + lines.append(f" - docs/models/{path}.qmd") + + yaml_content = "\n".join(lines) + "\n" + + # Pattern to match only the Model Guides contents, stopping at the next item + # in Getting Started (lines starting with 12 spaces: same level as the section) + pattern = r'( - section: "Model Guides"\n contents:)([^\n]*|.*?)(?=\n - |\n - section:|\n\nformat:)' + + def replacement(match): + prefix = match.group(1) + return prefix + "\n" + yaml_content + + new_content = re.sub(pattern, replacement, content, flags=re.DOTALL) + + if new_content != content: + quarto_yml.write_text(new_content, encoding="utf-8") + print(f"Updated {quarto_yml}") + else: + print(f"No changes needed for {quarto_yml}") + + +def main(): + allow = read_allowlist() + if not EXAMPLES_DIR.exists(): + print(f"[WARN] {EXAMPLES_DIR} not found", file=sys.stderr) + return + + (OUTPUT_DIR / "assets").mkdir(parents=True, exist_ok=True) + + # First pass: identify which parents have their own entry vs only sub-entries + parent_entries = set() # Parents that have their own entry + parent_with_subs = set() # Parents that have sub-entries + allowlist_entries = set() # All entries in allowlist + + for item in allow: + if isinstance(item, str): + name = item + else: + name = item.get("name") + + allowlist_entries.add(name) + + if "/" in name: + parent = name.split("/")[0] + parent_with_subs.add(parent) + else: + parent_entries.add(name) + + # Parents with subs that DON'T have their own entry -> use index.qmd + parent_index_only = parent_with_subs - parent_entries + + generated = [] + seen_dirs = set() # Track which parent directories we've created index for + + for item in allow: + if isinstance(item, str): + name = item + title = None + else: + name = item.get("name") + title = item.get("title") + + if not name: + print(f"[WARN] Skipping item without name: {item}", file=sys.stderr) + continue + + src_dir = EXAMPLES_DIR / name + if not src_dir.exists() or not src_dir.is_dir(): + print(f"[WARN] Skipping {name} (not a directory)", file=sys.stderr) + continue + + readme = find_readme(src_dir) + if not readme: + print(f"[WARN] Skipping {name} (no README.md)", file=sys.stderr) + continue + + md = readme.read_text(encoding="utf-8") + + # Determine output path first (needed for link rewriting) + parts = name.split("/") + if len(parts) == 1: + # Simple case: no subdirectory + out_path = OUTPUT_DIR / f"{parts[0]}.qmd" + sidebar_path = parts[0] + else: + # Has subdirectory: e.g., magistral/think + parent = parts[0] + child = "-".join(parts[1:]) # handle nested subdirs + out_path = OUTPUT_DIR / parent / f"{child}.qmd" + sidebar_path = f"{parent}/{child}" + + # Remove the first H1 (we use frontmatter title instead) + md, _ = remove_first_h1(md) + # Rewrite links between README files + md = rewrite_readme_links( + md, + src_dir, + EXAMPLES_DIR, + parent_index_only, + name, + allowlist_entries, + sidebar_path, + ) + md = rewrite_and_copy_assets(md, src_dir, OUTPUT_DIR) + + # Handle parent page generation for sub-entries + if len(parts) > 1: + # Has subdirectory: e.g., magistral/think + parent = parts[0] + + # Create parent.qmd if not already done and parent doesn't have own entry + if parent not in seen_dirs and parent in parent_index_only: + parent_readme = find_readme(EXAMPLES_DIR / parent) + if parent_readme: + parent_md = parent_readme.read_text(encoding="utf-8") + parent_md, _ = remove_first_h1(parent_md) + parent_md = rewrite_readme_links( + parent_md, + EXAMPLES_DIR / parent, + EXAMPLES_DIR, + parent_index_only, + parent, + allowlist_entries, + parent, + ) + parent_md = rewrite_and_copy_assets( + parent_md, EXAMPLES_DIR / parent, OUTPUT_DIR + ) + parent_title = parent.replace("-", " ").replace("_", " ").title() + write_qmd(OUTPUT_DIR / f"{parent}.qmd", parent_title, parent_md) + generated.append((parent, parent, parent_title)) + seen_dirs.add(parent) + + if not title: + title = name.replace("/", " ").replace("-", " ").title() + + write_qmd(out_path, title, md) + generated.append((sidebar_path, name, title)) + + # Index page - preserve allowlist order + if generated: + listing = "\n".join( + [f"- [{title}]({path}.qmd)" for path, name, title in generated] + ) + index_md = ( + "# Model Guides\n\nBelow are the curated examples for training various model architectures:\n\n" + + listing + + "\n" + ) + index_fm = ( + "---\nexecute:\n eval: false\nformat:\n html:\n toc: true\n---\n\n" + ) + (OUTPUT_DIR / "index.qmd").write_text(index_fm + index_md, encoding="utf-8") + + # Auto-update _quarto.yml to keep sidebar in sync + update_quarto_yml(generated) + + +if __name__ == "__main__": + main() diff --git a/examples/magistral/think/README.md b/examples/magistral/think/README.md index a87579775..adb22bbba 100644 --- a/examples/magistral/think/README.md +++ b/examples/magistral/think/README.md @@ -5,6 +5,7 @@ This guide covers fine-tuning [Magistral Small 2507](https://huggingface.co/mist ## Prerequisites Before starting, ensure you have: + - Installed Axolotl (see [main README](../README.md)) ## Getting Started diff --git a/examples/magistral/vision/README.md b/examples/magistral/vision/README.md index fc614c850..72a8a2215 100644 --- a/examples/magistral/vision/README.md +++ b/examples/magistral/vision/README.md @@ -5,7 +5,8 @@ This guide covers fine-tuning [Magistral Small 2509](https://huggingface.co/mist ## Prerequisites Before starting, ensure you have: -- Installed Axolotl from source (see [main README](../README.md#getting-started)) + +- Installed Axolotl from source (see [main README](../README.md)) ## Getting started diff --git a/examples/ministral3/think/README.md b/examples/ministral3/think/README.md index 8c40adbb9..a116dd5dd 100644 --- a/examples/ministral3/think/README.md +++ b/examples/ministral3/think/README.md @@ -5,6 +5,7 @@ This guide covers fine-tuning [Ministral3 2512](https://huggingface.co/collectio ## Prerequisites Before starting, ensure you have: + - Installed Axolotl (see [main README](../README.md)) ## Getting Started diff --git a/examples/ministral3/vision/README.md b/examples/ministral3/vision/README.md index 369b0116a..8193573eb 100644 --- a/examples/ministral3/vision/README.md +++ b/examples/ministral3/vision/README.md @@ -5,7 +5,8 @@ This guide covers fine-tuning [Ministral3 2512](https://huggingface.co/collectio ## Prerequisites Before starting, ensure you have: -- Installed Axolotl from source (see [main README](../README.md#getting-started)) + +- Installed Axolotl from source (see [main README](../README.md)) ## Getting started diff --git a/examples/mistral/mistral-small/README.md b/examples/mistral-small/README.md similarity index 99% rename from examples/mistral/mistral-small/README.md rename to examples/mistral-small/README.md index 3c606a897..7f7ec91e6 100644 --- a/examples/mistral/mistral-small/README.md +++ b/examples/mistral-small/README.md @@ -5,6 +5,7 @@ This guide covers fine-tuning [Mistral Small 3.1](mistralai/Mistral-Small-3.1-24 ## Prerequisites Before starting, ensure you have: + - Installed Axolotl (see [Installation docs](https://docs.axolotl.ai/docs/installation.html)) ## Getting Started diff --git a/examples/mistral/mistral-small/mistral-small-3.1-24B-lora.yml b/examples/mistral-small/mistral-small-3.1-24B-lora.yml similarity index 100% rename from examples/mistral/mistral-small/mistral-small-3.1-24B-lora.yml rename to examples/mistral-small/mistral-small-3.1-24B-lora.yml