togethercomputer
/

StripedHyena-Hessian-7B

Text Generation

Model card Files Files and versions Community

Zymrael commited on Nov 28, 2023

Commit

af9abca

•

1 Parent(s): 6be2da5

chore: info

Files changed (1) hide show

README.md +1 -3

README.md CHANGED Viewed

@@ -9,9 +9,7 @@ language:
 ### Model Architecture
-The architecture of StripedHyena-Hessian-7B is different from traditional decoder-only Transformers.
-StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks.
   - Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
   - Lower latency to preprocess long prompts.
   - Improvements to training and inference compute-optimal scaling laws, compared to Transformers.

 ### Model Architecture
+StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks, different from traditional decoder-only Transformers.
   - Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
   - Lower latency to preprocess long prompts.
   - Improvements to training and inference compute-optimal scaling laws, compared to Transformers.