bartowski

/

Nemotron-Mini-4B-Instruct-GGUF

like 8

What do you mean by llama.cpp doesn't support it? It certainly does 🤔 which version of lmstudio are you using? It's possible it needs an update to its binary but definitely the main llamacpp has support

some of these need remote code true ,

I dont know how to quantize a model which uses this requirement as it always fails for me !

But if its quantized generally it works , But Yes also sometimes we can quantize something once and it does not work ! ( i made a merg like this and it was a llama+Mistral merge , even now i cannot delete it it was doi...
but if its quantized it should run on all the platforms , you just need to update...

i find that KoboloCPP Works the best with Strange GGUFS ! << ie you could have created the model with a mad chat template requiring more the 1 input .... ie alpaca , so the model does not work right .. hence you would need to use a code ui manually created so you can load it and prompt it to get the required response !
not all models are FIT to be Hosted on LMstudio or gpt4all or ollama etc , but as a chat model they work ...again messy prompt templates can mess you up also !

so some models that do not work in these dockers .... will work locally with the right UI ~ or Code access ... even Ctransformers ... but also hugging face transformers can now load gguf and extract back to safetensors !

I don't know how to attach a screenshot, but I'll try to do it in text:

You are on the latest version.
Current Version: 0.2.31

bartowski/Nemotron-Mini-4B-Instruct-GGUF
September 13, 2024 (yesterday)

Nemotron-Mini-4B-Instruct-Q8_0.gguf

{
  "title": "Failed to load model",
  "cause": "llama.cpp error: 'error loading model architecture: unknown model architecture: 'nemotron''",
  "errorData": {
    "n_ctx": 512,
    "n_batch": 512,
    "n_gpu_layers": 33
  },
  "data": {
    "memory": {
      "ram_capacity": "8.00 GB",
      "ram_unused": "3.40 GB"
    },
    "gpu": {
      "gpu_names": [
        "Apple Silicon"
      ],
      "vram_recommended_capacity": "5.33 GB",
      "vram_unused": "2.83 GB"
    },
    "os": {
      "platform": "darwin",
      "version": "15.1"
    },
    "app": {
      "version": "0.2.31",
      "downloadsDir": "/Users/alex/.cache/lm-studio/models"
    },
    "model": {}
  }
}```

0.2.31 isn't latest, you can grab the newer 0.3.2 from https://lmstudio.ai/

Not sure when if ever auto update will come

Ok, thank you that you pointed me out to the newest version. I was sure that I can update it from inside the LM Studio interface. Now the model is started to work.

P.s.: The interface on this new LM Studio is worse. I can't even find how to set up the model parameters. But it doesn't relate to this quant. The quant works with this version.

Unfortunately, the model is bad compared to its competitors, especially in general knowledge.

As for the rest. I asked the model to explain me many-to-many relationship in Django, and it explained me many-to-one... So, in my case, the model is useless.

Yes it also has a different server setup ! bloddy people made a booby ! ... I had to revert to the previous version because the models was not accpting the endpoint 2