| datasets: | |
| - JeanKaddour/minipile | |
| language: | |
| - en | |
| license: mpl-2.0 | |
| # RWKV Minipile | |
| ## Model Specifications | |
| - **Architecture**: RWKV | |
| - **Vocabulary Size**: 65,536 | |
| - **Embedding Size**: 768 | |
| - **Number of Layers**: 12 | |
| - **Context Length**: 512 | |
| - **Data Type**: bfloat16 | |
| - **Dataset**: Minipile | |
| - **Tokens**: 20,643,840 (20 Million) | |
| The model underwent a rigorous training regimen, completing 30 epochs to optimize performance. | |
| ## Inference | |
| ```bash | |
| pip install torch numpy | |
| python inference.py | |
| ``` |