New Level
I return once again to talk about this model: Rocinante. For quite some time, I thought smaller models like this couldn’t improve much more. I considered Stheno to be the peak for small models. Running a 12B model like Rocinante pushes my system quite a bit, and I only get about 1.5 tokens per second. While that's not great, it's tolerable.
That said, I tolerate the slower speeds because this model has something I’ve seen even much larger models struggle with—a beautiful level of “understanding” in what it does. It’s by no means the smartest model out there, but it knows what its doing. To explain this better, think of the difference between street smarts and book smarts. For ages, I focused on finding the smartest model: one capable of handling complex statistics and calculations. But what I never expected was a model capable of evaluating a situation and demonstrating a nuanced understanding of what’s happening in that context.
While larger models like 70B or 150B may outshine this one in raw capability, and future smaller models may surpass it, for now, I can’t find a better, more soothing model to use as my daily driver.
To summarize, I wouldn’t recommend Rocinante for statistical or mathematical tasks. But if you’re looking for a situationally aware and intelligent model with a genuine grasp of context, this one comes with my full recommendation.
what instruct template are you using? Mistral or chatml?