This is AIDC-ai-business/Marcoroni-70B-v1 quantized to 4bit AWQ with the following config:
python3 -m lmdeploy.lite.apis.auto_awq \
--model ./Marcoroni-70B-v1 \
--w_bits 4 \
--w_group_size 128 \
--work_dir ./quant
Original Model Card:
Marcoroni-70B
Model Details
- Trained by: trained by AIDC AI-Business.
- Model type: Marcoroni-70B is an auto-regressive language model based on the Llama 2 transformer architecture.
- Language(s): English
- License for Marcoroni-70B base weights: Non-Commercial Creative Commons license (CC BY-NC-4.0)
Prompting
Prompt Template for alpaca style
### Instruction:
<prompt> (without the <>)
### Response: