add model files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,140 @@
+---
+language:
+- en
+license: llama2
+tags:
+- meta
+- llama-2
+- wasmedge
+- second-state
+- llama.cpp
+model_name: Llama 2 GGUF
+inference: false
+model_creator: Meta Llama 2
+model_type: llama
+pipeline_tag: text-generation
+prompt_template: '[INST] <<SYS>>
+
+  You are a helpful, respectful and honest assistant. Always answer as helpfully as
+  possible, while being safe.  Your answers should not include any harmful, unethical,
+  racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses
+  are socially unbiased and positive in nature. If a question does not make any sense,
+  or is not factually coherent, explain why instead of answering something not correct.
+  If you don''t know the answer to a question, please don''t share false information.
+
+  <</SYS>>
+
+  {prompt}[/INST]
+
+  '
+quantized_by: wasmedge
+---
+
+This repo contains GGUF model files for cross-platform AI inference using the [WasmEdge Runtime](https://github.com/WasmEdge/WasmEdge). 
+[Learn more](https://medium.com/stackademic/fast-and-portable-llama2-inference-on-the-heterogeneous-edge-a62508e82359) on why and how. 
+
+
+## Prerequisite
+
+
+Install WasmEdge with the GGML plugin. 
+
+```
+curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml
+```
+
+Download the cross-platform Wasm apps for inference. 
+
+```
+curl -LO https://github.com/second-state/llama-utils/raw/main/simple/llama-simple.wasm
+
+curl -LO https://github.com/second-state/llama-utils/raw/main/chat/llama-chat.wasm
+```
+
+## Use the f16 models
+
+
+The f16 version is an GGUF equivalent of the original llama2 models. It gives the best quality inference results but also consumes the most computing resources in both VRAM and computing time. The f16 models are also great as a basis for fine-tuning.
+
+Chat with the 7b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-chat-f16.gguf llama-chat.wasm
+```
+
+Generate text with the 7b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-f16.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```
+
+Chat with the 13b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-chat-f16.gguf llama-chat.wasm
+```
+
+Generate text with the 13b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-f16.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```
+
+
+## Use the quantized models
+
+
+The `q5_k_m` version is a quantized version of the llama2 models. They are only half of the size of the original models, and hence consumes half as much VRAM, but still gives high quality inference results. 
+
+Chat with the 7b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-chat-q5_k_m.gguf llama-chat.wasm
+```
+
+Generate text with the 7b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-q5_k_m.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```
+
+Chat with the 13b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-chat-q5_k_m.gguf llama-chat.wasm
+```
+
+Generate text with the 13b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-q5_k_m.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```
+
+## Resource constrained models
+
+
+The `q2_k` version is the smallest quantized version of the llama2 models. They can run on devices with only 4GB of RAM, but the inference quality is rather low.
+
+Chat with the 7b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-chat-q2_k.gguf llama-chat.wasm
+```
+
+Generate text with the 7b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-7b-q2_k.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```
+
+Chat with the 13b chat model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-chat-q2_k.gguf llama-chat.wasm
+```
+
+Generate text with the 13b base model
+
+```
+wasmedge --dir .:. --nn-preload default:GGML:CPU:llama-2-13b-q2_k.gguf llama-simple.wasm 'Robert Oppenheimer most important achievement is '
+```

llama-2-13b-chat-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:742b49f6a953717a511e60e480fd6c338e3acdab8b363034da247c268930b56a
+size 26033303264

llama-2-13b-chat-q2_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecfda81b21c88db496a48f76e758dd5c5289f52c85f8820896915248a11ef8a3
+size 5429348096

llama-2-13b-chat-q5_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69bcbe0985c12f01ebe56d837beaca6febd83a66cf3936b09e2bd1430035949d
+size 9229924096

llama-2-13b-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d021c71b94603bf4add6035912e1e78780c3b51fdafd1f0408d8438b20caff25
+size 26033303264

llama-2-13b-q2_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3db4d5339cb55368e972c64e782e0ed4592fc69b651089cab1fdfeaf0a3d9398
+size 5429348096

llama-2-13b-q5_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ffc205b67beb10be50fa588a8d0bb29d3538bdc5d202388338004c13d3c97af3
+size 9229924096

llama-2-7b-chat-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0c47e0b78d46d59bf9ffe986072e3206aabb9f4fe66d263ccedc689a3d1c269
+size 13478104576

llama-2-7b-chat-q2_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0da39ea6df19f861d46759aa1aa6e493c13cc92206047d911a00213b6eafa5cd
+size 2825940544

llama-2-7b-chat-q5_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34acb8c183aeb92be6fa4c63b3911ddcef6c1c52d4cdfd90a38dd7642546c1a9
+size 4783156800

llama-2-7b-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87dc1541426f2b23f57d3af817d156b4eeb8facc34e46254aa61602ab86cc7b0
+size 13478104576

llama-2-7b-q2_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:239452f5665e598afa74443935cf983674e9aad94c8a835393cd9b4ee92978cc
+size 2825940544

llama-2-7b-q5_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c61b678f868be0ca48fa4fa5a8f37e2638e1aae2618a271c67a3c2fb7be55aac
+size 4783156800