---
tags:
- llama
---
252M custom transformer architecture with two transformations from the llama2-70b embeddings to 1024-dimensional from 8192-d and back from 1024-d to 8192-d for the llama2-70b language modelling head.