|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
## Cat Embeddings |
|
|
|
A set of embedding model trained for study embedding quality vs model architecture (width/depth) given a size constraint (12M params). |
|
|
|
- **cat-emb-2-128**: 2 layers/hidden size 128/4.4m |
|
- **cat-emb-4-128**: 4 layers/H 128/4.8m |
|
- **cat-emb-8-128**: 8 layers/H 128/5.6m |
|
- **cat-emb-12-128**: 12 layers/H 128/6.4m |
|
- **cat-emb-2-256**: 2 layers/H 256/9.7m |
|
- **cat-emb-4-256**: 4 layers/H 256/11.3m |
|
|
|
### Training |
|
|
|
- stage 1: seq 192, batch size 2048, 50k steps, sentence pairs. |
|
- stage 2: seq 512, batch size 64, 5k steps, sentence triplets. |
|
|
|
### Perf |
|
|
|
|
|
|
