README.md

---
base_model: infosys/NT-Java-1.1B
inference: false
widget:
- text: "public class HelloWorld {\n    public static void main(String[] args) {"
  example_title: Hello world
  group: Java
language:
- code
tags:
- NarrowTransformer
license: bigcode-openrail-m
library_name: transformers
model_creator: Infosys
model_name: infosys/NT-Java-1.1B
model_type: gpt_bigcode
prompt_template: |
  {prompt}
quantized_by: Infsys
pipeline_tag: text-generation
---

# NT-Java-1.1B-GGUF
- Model creator: [Infosys](https://huggingface.co/infosys)
- Original model: [NT-Java-1.1B](https://huggingface.co/infosys/NT-Java-1.1B)

<!-- description start -->
## Description

This repo contains GGUF format model files for [Infosys's NT-Java-1.1B](https://huggingface.co/infosys/NT-Java-1.1B).

<!-- description end -->
<!-- README_GGUF.md-about-gguf start -->
### About GGUF

GGUF, introduced by the llama.cpp team on August 21st, 2023, is a new format designed to replace the outdated GGML, which is no longer maintained by llama.cpp. GGUF boasts several improvements over GGML, such as enhanced tokenization, support for special tokens, and metadata capabilities. It is also designed with extensibility in mind.

Below is a partial list of clients and libraries known to support GGUF:

* [llama.cpp](https://github.com/ggerganov/llama.cpp). The foundational project for GGUF, featuring both a command-line interface (CLI) and server options.
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui), A highly utilized web UI offering extensive features and robust extensions, supporting GPU acceleration.
* [KoboldCpp](https://github.com/LostRuins/koboldcpp), A comprehensive web UI with full GPU acceleration across all platforms and architectures, particularly effective for storytelling.
* [LM Studio](https://lmstudio.ai/), An intuitive and powerful local GUI designed for Windows and macOS (Silicon), featuring GPU acceleration for enhanced performance.
* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), A notable web UI with distinctive features, including a comprehensive model library for easy model selection.
* [Faraday.dev](https://faraday.dev/), A user-friendly character-based chat GUI for Windows and macOS (both Silicon and Intel), boasting GPU acceleration for smooth operation.
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), A Python library featuring GPU acceleration, LangChain support, and an OpenAI-compatible API server.
* [candle](https://github.com/huggingface/candle), A Rust-based ML framework prioritizing performance, equipped with GPU support and designed for ease of use.

<!-- README_GGUF.md-about-gguf end -->

<!-- prompt-template start -->
## Prompt template: None

```
{prompt}

```

<!-- prompt-template end -->


<!-- compatibility_gguf start -->
## Compatibility

The NT-Java-1.1B GGUFs are supported by llama.cpp and are compatible with a range of third-party user interfaces and libraries. For a detailed list, please refer to the beginning of this README.
<!-- compatibility_gguf end -->

<!-- README_GGUF.md-provided-files start -->
## Provided files

| Name | Quant method | Bits | Size | Max RAM required | Model Evaluation | Application Scenarios |
| ---- | ---- | ---- | ---- | ---- | ----- | ----- |
| [NT-Java-1.1B_Q2_K.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q2_K.gguf) | Q2_K | 2 | 511 MB| 764 MB | poor quality | not advised for usage | 
| [NT-Java-1.1B_Q3_K_M.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q3_K_M.gguf) | Q3_K_M | 3 | 663 MB| 912 MB | moderate quality | suitable for environments with low RAM |
| [NT-Java-1.1B_Q4_0.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q4_0.gguf) | Q4_0 | 4 | 726 MB| 1021 MB | moderate quality, prefer using Q3_K_M | not recommended, prefer Q3_K_M |
| [NT-Java-1.1B_Q4_K_M.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q4_K_M.gguf) | Q4_K_M | 4 | 792 MB| 1.1 GB | good quality | top recommendation due to optimal size and quality |
| [NT-Java-1.1B_Q5_0.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q5_0.gguf) | Q5_0 | 5 | 868 MB| 1.08 GB | good quality, prefer Q4_K_M | not recommended, prefer Q4_K_M |
| [NT-Java-1.1B_Q5_K_M.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q5_K_M.gguf) | Q5_K_M | 5 | 910 MB| 1.13 GB | excellent quality | recommended, second-best choice |
| [NT-Java-1.1B_Q6_K.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q6_K.gguf) | Q6_K | 6 | 1.02 GB| 1.24 GB | excellent quality | generally not suggested due to size compared to Q5_K_M |
| [NT-Java-1.1B_Q8_0.gguf](https://huggingface.co/infosys/NT-Java-1.1B-GGUF/blob/main/NT-Java-1.1B_Q8_0.gguf) | Q8_0 | 8 | 1.32 GB| 1.54 GB | top-tier quality, near flawless | preferred in environments with sufficient RAM |

**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.



<!-- README_GGUF.md-provided-files end -->

<!-- README_GGUF.md-how-to-download start -->
## How to download GGUF files

**On the command line, including multiple files at once**

The use of the Huggingface Hub Python library is recommended:

```shell
pip3 install huggingface-hub
```

Then you can download any individual model file to the current directory, at high speed, with a command like this:

```shell
huggingface-cli download infosys/NT-Java-1.1B-GGUF NT-Java-1.1B_Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
```

<details>
  <summary>More advanced huggingface-cli download usage (click to read)</summary>

You can also download multiple files at once with a pattern:

```shell
huggingface-cli download infosys/NT-Java-1.1B-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
```

For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).

To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:

```shell
pip3 install hf_transfer
```

And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:

```shell
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download infosys/NT-Java-1.1B-GGUF NT-Java-1.1B_Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
```

Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
</details>
<!-- README_GGUF.md-how-to-download end -->

<!-- README_GGUF.md-how-to-run start -->
## Example `llama.cpp` command

Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.

```shell
./main -ngl 35 -m NT-Java-1.1B_Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"
```

Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically. Note that longer sequence lengths require much more resources, so you may need to reduce this value.

If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`

For other parameters and how to use them, please refer to [the llama.cpp documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)


## How to use with Ollama

1. **Install Ollama:**

```
curl -fsSL https://ollama.com/install.sh | sh
```

2. **Run the *nt-java* model:**

```
ollama run infosys/nt-java
```

### Building from `Modelfile`

Assuming that you have already downloaded GGUF files, here is how you can use them with [Ollama](https://ollama.com/):

1. **Get the Modelfile:**

```
huggingface-cli download infosys/NT-Java-1.1B-GGUF Modelfile_q4_k_m --local-dir /path/to/your/local/dir
```

2. Build the Ollama Model:
Use the Ollama CLI to create your model with the following command:

```
ollama create NT-Java -f Modelfile_q4_k_m
```

3. **Run the *NT-Java* model:** 

Now you can run the NT-Java model with Ollama using the following command:

```
ollama run NT-Java "Your prompt here"
```

Replace "Your prompt here" with the actual prompt you want to use for generating responses from the model.

## How to use with Llamafile:

Assuming that you already have GGUF files downloaded. Here is how you can use the GGUF model with [Llamafile](https://github.com/Mozilla-Ocho/llamafile):

1. **Download Llamafile-0.7.3**
```
wget https://github.com/Mozilla-Ocho/llamafile/releases/download/0.7.3/llamafile-0.7.3
```
2. **Run the model with prompt:**


```markdown
public class HelloWorld {\n    public static void main(String[] args) {
```


```
./llamafile-0.7.3 -ngl 9999 -m NT-Java-1.1B_Q4_K_M.gguf --temp 0.6 -p "public class HelloWorld {\n    public static void main(String[] args) {"
```

3. **Run with a chat interface:**

```
./llamafile-0.7.3 -ngl 9999 -m NT-Java-1.1B_Q4_K_M.gguf
```

Your browser should open automatically and display a chat interface. (If it doesn't, just open your browser and point it at http://localhost:8080)


## How to run from Python code

You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) version 0.2.23 and later.

### How to load this model in Python code, using llama-cpp-python

For full documentation, please see: [llama-cpp-python docs](https://abetlen.github.io/llama-cpp-python/).

#### First install the package

Run one of the following commands, according to your system:

```shell
# Base ctransformers with no GPU acceleration
pip install llama-cpp-python
# With NVidia CUDA acceleration
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
# Or with OpenBLAS acceleration
CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python
# Or with CLBLast acceleration
CMAKE_ARGS="-DLLAMA_CLBLAST=on" pip install llama-cpp-python
# Or with AMD ROCm GPU acceleration (Linux only)
CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python
# Or with Metal GPU acceleration for macOS systems only
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python

# In windows, to set the variables CMAKE_ARGS in PowerShell, follow this format; eg for NVidia CUDA:
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on"
pip install llama-cpp-python
```

### Simple llama-cpp-python example code

```python
from llama_cpp import Llama

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
  model_path="./NT-Java-1.1B_Q4_K_M.gguf",  # Download the model file first
  n_ctx=2048,  # The max sequence length to use - note that longer sequence lengths require much more resources
  n_threads=8,            # The number of CPU threads to use, tailor to your system and the resulting performance
  n_gpu_layers=35         # The number of layers to offload to GPU, if you have GPU acceleration available
)

# Simple inference example
output = llm(
  "{prompt}", # Prompt
  max_tokens=512,  # Generate up to 512 tokens
  stop=["</s>"],   # Example stop token - not necessarily correct for this specific model! Please check before using.
  echo=True        # Whether to echo the prompt
)
```

## How to use with LangChain

Here are guides on using llama-cpp-python and ctransformers with LangChain:

* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)

<!-- README_GGUF.md-how-to-run end -->

<!-- footer start -->
<!-- 200823 -->

# Citation
```
@article{rathinasamy2024narrow,
  title={Narrow Transformer: Starcoder-Based Java-LM For Desktop},
  author={Kamalkumar Rathinasamy and Balaji A J and Rajab Ali Mondal and Ankush Kumar and Harshini K and Gagan Gayari and Sreenivasa Raghavan Karumboor Seshadri and Swayam Singh},
  journal={arXiv preprint arXiv:2407.03941},
  year={2024}
}
```

# Original model card: Infosys's [NT-Java-1.1B](https://huggingface.co/infosys/NT-Java-1.1B)

# **NT-Java-1.1B**

The Narrow Transformer (NT) model NT-Java-1.1B is an open-source specialized code model built by extending pre-training on StarCoderBase-1B, designed for coding tasks in Java programming. The model is a decoder-only transformer with Multi-Query Attention and with a context length of 8192 tokens. The model was trained with Java subset of the StarCoderData dataset, which is ~22B tokens.