| license: apache-2.0 | |
| Mamba Bit! | |
| Mamba with vocab size 2 bites again! This time we bite at tiny stories. | |
| I didn't bother preprocess them at all, during a training model took random char offset, converted it to bit string and fed to mamba. This time I didn't forget about residual connections nor about norm. As the result model was trained in BF16. | |
| Training code included. | |
| Example to run a model from CLI: | |
| $ python mambabit.py "Run, kitten, run" | |
| Run, kitten, running and jumping. She saw a big tree and thought it would be fun to share the tree. So, she went to the tree and started to climb the tree. She saw a big tree and thought it would be fun to share the tree. So, she went to the tree and saw a big red ball. |