|
--- |
|
datasets: |
|
- togethercomputer/RedPajama-Data-1T-Sample |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
tags: |
|
- text-generation-inference |
|
--- |
|
This is [Llama2-22b](https://huggingface.co/chargoddard/llama2-22b) by [chargoddard](https://huggingface.co/chargoddard) in a couple of GGML formats. I have no idea what I'm doing so if something doesn't work as it should or not at all that's likely on me, not the models themselves.<br> |
|
A second model merge has been [released](https://huggingface.co/chargoddard/llama2-22b-blocktriangular) and the GGML conversions for that can be found [here](https://huggingface.co/IHaveNoClueAndIMustPost/llama2-22b-blocktriangular-GGML). |
|
|
|
While I haven't had any issues so far do note that the original repo states <i>"Not intended for use as-is - this model is meant to serve as a base for further tuning"</b>. |
|
|
|
Approximate VRAM requirements at 4K context: |
|
<table style='border: 2px #000000 solid; width: 50%' align='left' border='2'> |
|
<tbody> |
|
<tr> |
|
<td style='text-align: center'>MODEL</td> |
|
<td style='text-align: center'>SIZE</td> |
|
<td style='text-align: center'>VRAM</td> |
|
</tr> |
|
<tr> |
|
<td style='text-align: center'>q5_1</td> |
|
<td style='text-align: center'>16.4GB</td> |
|
<td style='text-align: center'>21.5GB</td> |
|
</tr> |
|
<tr> |
|
<td style='text-align: center'>q4_K_M</td> |
|
<td style='text-align: center'>13.2GB</td> |
|
<td style='text-align: center'>18.3GB</td> |
|
</tr> |
|
<tr> |
|
<td style='text-align: center'>q3_K_M</td> |
|
<td style='text-align: center'>10.6GB</td> |
|
<td style='text-align: center'>16.1GB</td> |
|
</tr> |
|
<tr> |
|
<td style='text-align: center'>q2_K</td> |
|
<td style='text-align: center'>9.2GB</td> |
|
<td style='text-align: center'>14.5GB</td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
|
