initial commit of 8-bit quantized GGUF

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text

LLama-2-MedText-13b-q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f12f95b748feba8d616514c759e969098339b41da11ad9c173a655a3b48ec19
+size 13831319392

Readme.md

@@ -0,0 +1,52 @@
+## LLama-2-MedText-13b-GGUF
+
+Quantized GGUF of https://huggingface.co/truehealth/LLama-2-MedText-13b
+
+## Usage
+
+Interactive [llama.cpp](https://github.com/ggerganov/llama.cpp/) session:
+
+```bash
+llama-cpp \
+  --instruct \
+  --color \
+  --in-prefix "[INST] " \
+  --in-suffix "[\INST] " \
+  --model LLama-2-MedText-13b-q8_0.gguf
+
+== Running in interactive mode. ==
+ - Press Ctrl+C to interject at any time.
+ - Press Return to return control to LLaMa.
+ - To return control without starting a new line, end your input with '/'.
+ - If you want to submit another line, end your input with '\'.
+
+
+> [INST] How confident are you in your knowledge and abilities?
+[\INST] [RSP] As an AI language model, I can provide information to the best of my ability based on the resources I was trained on, which were primarily before <DATE>. While I strive to provide useful and accurate responses, my knowledge is not infinite, and I might not be able to provide professional medical advice or predictions in all cases. Additionally, healthcare decisions should always be evaluated in the context of an individual's unique circumstances and should be evaluated by a healthcare professional.
+```
+
+## Model card from truehealth/Llama-2-MedText-Delta-Preview
+
+Trained on https://huggingface.co/datasets/BI55/MedText.
+
+These are PEFT delta weights and need to be merged into LLama-2-13b to be used for inference.
+
+library_name: peft
+
+Training procedure
+
+The following bitsandbytes quantization config was used during training:
+
+- load_in_8bit: False
+- load_in_4bit: True
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: nf4
+- bnb_4bit_use_double_quant: True
+- bnb_4bit_compute_dtype: float16
+
+Framework versions
+
+- PEFT 0.5.0.dev0