# Seed-OSS

Seed-OSS are a series of 36B parameter open source models trained by ByteDance’s Seed Team.

This guide shows how to fine-tune it with Axolotl with multi-turn conversations and proper masking.

## Getting started

1. Install Axolotl following the installation guide.

   Here is an example of how to install from pip:

       # Ensure you have a compatible version of Pytorch installed
       pip3 install packaging setuptools wheel ninja
       pip3 install --no-build-isolation 'axolotl[flash-attn]>=0.12.0'

       # Install Cut Cross Entropy
       python scripts/cutcrossentropy_install.py | sh

2. Run the finetuning example:

    axolotl train examples/seed-oss/seed-oss-36b-qlora.yaml

This config uses about 27.7 GiB VRAM.

Let us know how it goes. Happy finetuning! 🚀

### TIPS

- For inference, the official Seed Team recommends `top_p=0.95` and `temperature=1.1`.
- You can run a full finetuning by removing the `adapter: qlora` and `load_in_4bit: true` from the config.
- Read more on how to load your own dataset at docs.
- The dataset format follows the OpenAI Messages format as seen here.

## Optimization Guides

Please check the Optimizations doc.

## Related Resources

- ByteDance Seed Website
- Axolotl Docs
- Axolotl Website
- Axolotl GitHub
- Axolotl Discord