---
base_model: microsoft/rho-math-7b-interpreter-v0.1
license: mit
model_creator: Microsoft
model_name: rho-math-7b-interpreter-v0.1
model_type: mistral
library_name: gguf
tags:
- math
quantized_by: arzeth
---

Author of this model: Microsoft, 2024. License: MIT.

Link to the original card: https://huggingface.co/microsoft/rho-math-7b-interpreter-v0.1

Prompt template: ChatML (according to llama.cpp's `server`)? Mistral (according to `tokenizer_config.json`)? Alpaca (according to text-generation-webui)? All three seem to work.

Context length: ?

According to their [paper on arXiv](https://arxiv.org/abs/2404.07965), rho-math-7b-v0.1 is a continued pretraining on Mistral-7B, while their 1B model is a continued pretraining on TinyLlama-1.1B.

# My experience

Unlike [the non-interpreter variant](https://huggingface.co/microsoft/rho-math-7b-v0.1) of this model, this `-interpreter` variant always answers with Python code, example:

Input: `Width of circle is 3cm, what is its area?`

with settings `{
  presence_penalty: 0,
  frequency_penalty: 0,
  top_p: 1,
  min_p: 0,
  top_k: 0,
  temperature: 0.8
}`

outputs (using unquantized gguf):

<pre>
```python
from sympy import pi, Rational

def circle_area():
    """Width of circle is 3cm, what is its area?"""
    r = Rational(3, 2)  # Radius of the circle
    area = pi * r**2  # Area of the circle

    return area

result = circle_area()
print(result)
```
```output
27*pi/4
```
The area of the circle is $\boxed{\frac{27\pi}{4}}$ square cm.
</pre>

??? It should have been `9*pi/4`. Am I using this model wrong? Same result with temperature=0.0,top_k=1.

In comparison, the Q6_K of the non-interpreter variant of this model with temp=0.0 outputs:
```
The area of a circle is given by the formula A = πr^2, where r is the radius of the circle. Since the width of the circle is 3cm, the radius is half of that, or 1.5cm. Substituting this value into the formula, we get:

A = π(1.5)^2
A = π(2.25)
A = 6.9887654321

So, the area of the circle is approximately 6.99 cm^2.
```
(close to the expected 7.06 which `deepseek-math-7b-rl.Q8_0.gguf` outputs)

# imatrix

I created imatrix with
```
./imatrix --mlock --verbosity 2 -m /tmp/rho-math-7b-interpreter-v0.1.f32.gguf -f ~/Downloads/groups_merged_forkOfArzeth.txt -c 32768 -o rho-math-7b-interpreter-v0.1.f32.ctx32768imatrix.dat
```
which took 1665 seconds (28 minutes) on my GTX 1660 Super and used only 1 thread on Ryzen 2600 downclocked to 3000MHz. `imatrix` consumed 35685 MiB of RAM (3200MHz) and 3158 MiB of VRAM.

# quantize

Quantized with llama.cpp b2661 (2024-04-12), compiled with `LLAMA_CUDA_FORCE_MMQ=1` (full cmd: `make -j6 LLAMA_CUDA_FORCE_MMQ=1 LLAMA_CUDA=1 LLAMA_FAST=1 LLAMA_OPENBLAS=1 LLAMA_BLAS_VENDOR=OpenBLAS`) for a big speed up (GTX 1660 Super doesn't have tensor cores, so it's better to use MMQ than nothing).