# Differential Transformer

### Usage

```yaml
plugins:
  - axolotl.integrations.diff_transformer.DifferentialTransformerPlugin

diff_attention: true
diff_attn_zero_init: false
diff_attn_sublayer_norm: true
diff_attn_split_heads: false
```
