|
--- |
|
library_name: pytorch |
|
license: llama2 |
|
pipeline_tag: text-generation |
|
tags: |
|
- llm |
|
- generative_ai |
|
- quantized |
|
- android |
|
|
|
--- |
|
|
|
![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/llama_v2_7b_chat_quantized/web-assets/banner.png) |
|
|
|
# Llama-v2-7B-Chat: Optimized for Mobile Deployment |
|
## State-of-the-art large language model useful on a variety of language understanding and generation tasks |
|
|
|
Llama 2 is a family of LLMs. The "Chat" at the end indicates that the model is optimized for chatbot-like dialogue. The model is quantized to 4-bit weights and 16-bit activations making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-KVCache-Quantized's latency. |
|
|
|
This is based on the implementation of Llama-v2-7B-Chat found |
|
[here](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). More details on model performance |
|
accross various devices, can be found [here](https://aihub.qualcomm.com/models/llama_v2_7b_chat_quantized). |
|
|
|
### Model Details |
|
|
|
- **Model Type:** Text generation |
|
- **Model Stats:** |
|
- Number of parameters: 7B |
|
- Model size: 3.6GB |
|
- Model-1 (Prompt Processor): Llama-PromptProcessor-Quantized |
|
- Max context length: 1024 |
|
- Prompt processor input: 1024 tokens |
|
- Prompt processor output: 1 output token + KVCache for token generator |
|
- Model-2 (Token Generator): Llama-TokenGenerator-KVCache-Quantized |
|
- Token generator input: 1 input token + past KVCache |
|
- Token generator output: 1 output token + KVCache for next iteration |
|
- Decoding length: 1024 (1 output token + 1023 from KVCache) |
|
- Use: Initiate conversation with prompt-processor and then token generator for subsequent iterations. |
|
- QNN-SDK: 2.19 |
|
|
|
|
|
| Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model |
|
| ---|---|---|---|---|---|---|---| |
|
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Model Library | 117.812 ms | 66 - 238 MB | UINT16 | NPU | Llama-TokenGenerator-KVCache-Quantized |
|
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Model Library | 2578.521 ms | 12 - 17 MB | UINT16 | NPU | Llama-PromptProcessor-Quantized |
|
|
|
|
|
## License |
|
- The license for the original implementation of Llama-v2-7B-Chat can be found |
|
[here](https://github.com/facebookresearch/llama/blob/main/LICENSE). |
|
|
|
## References |
|
* [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) |
|
* [Source Model Implementation](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) |
|
|
|
## Community |
|
* Join [our AI Hub Slack community](https://join.slack.com/t/qualcomm-ai-hub/shared_invite/zt-2dgf95loi-CXHTDRR1rvPgQWPO~ZZZJg) to collaborate, post questions and learn more about on-device AI. |
|
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com). |
|
|
|
## Usage and Limitations |
|
|
|
Model may not be used for or in connection with any of the following applications: |
|
|
|
- Accessing essential private and public services and benefits; |
|
- Administration of justice and democratic processes; |
|
- Assessing or recognizing the emotional state of a person; |
|
- Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics; |
|
- Education and vocational training; |
|
- Employment and workers management; |
|
- Exploitation of the vulnerabilities of persons resulting in harmful behavior; |
|
- General purpose social scoring; |
|
- Law enforcement; |
|
- Management and operation of critical infrastructure; |
|
- Migration, asylum and border control management; |
|
- Predictive policing; |
|
- Real-time remote biometric identification in public spaces; |
|
- Recommender systems of social media platforms; |
|
- Scraping of facial images (from the internet or otherwise); and/or |
|
- Subliminal manipulation |
|
|