File size: 1,833 Bytes
26c02f0 32d7e50 26c02f0 8014230 9d6957e 8014230 759a8f9 eb818d2 8014230 0e6972b 8014230 98f54e2 2e4b794 8014230 9d6957e 98f54e2 759a8f9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
---
license: apache-2.0
datasets:
- Norquinal/claude_multiround_chat_30k
- OpenLeecher/Teatime
---
We proudly announce this is the world first 128k context model based on RWKV architecture today, 2023-08-10.
This model trained with instructions datasets and chinese web novel and tradition wuxia,
more trainning details would be updated.
Test input 67k tokens to summary ,can find in example folders ,more cases are coming.
Full finetuned using this repo to train 128k context model , 4*A800 40hours with 1.3B tokens.
https://github.com/SynthiaDL/TrainChatGalRWKV/blob/main/train_world.sh
Using RWKV Runner https://github.com/josStorer/RWKV-Runner to test this ， use temp 0.1-0.2 topp 0.7 for more precise answer ,temp between 1-2.x is more creatively.
67k input test
 |