Best medium model?
What do you regard as the best medium model, for use with you vectors? I did like Qwen 2.5, as it's been one of the few models of that size that has been able to pull character speech patterns from the world info. But I've seen you have said it's not great, but that is an important capability for me. Cheers
What do you regard as the best medium model, for use with you vectors? I did like Qwen 2.5, as it's been one of the few models of that size that has been able to pull character speech patterns from the world info. But I've seen you have said it's not great, but that is an important capability for me. Cheers
I think the (older) command-r:35b (uses a lot of VRAM for context) or gemma-2:27b (only 8k context) are likely the best, but I'm not really sure if you need smaller than that as haven't really used many (possiblly gemma-2:9b or one of its SPPO/SimPO fine-tunes: https://eqbench.com/creative_writing.html).
All of the SPPO-Iter3, SimPO and SPPO seem to score well on writing benchmarks, so probably they are the best to try first.
34b or there abouts is my sweet spot as I have 24gb. I'll check out some of the Gemma fine tunes. One has been extended to 34k? I think (I thought wrong), context. But I think they rope out quite well a little anyway. Thanks man!