---
datasets:
- EleutherAI/pile
language:
- en
---

Based model but uses layernorm instead of QK.sum(-1) for the normalization, for better hardware efficiency.