|
--- |
|
language: |
|
- en |
|
datasets: |
|
- Open-Orca/SlimOrca |
|
- Trelis/function_calling_v3 |
|
tags: |
|
- function calling |
|
- function-calling |
|
extra_gated_prompt: "Purchase access to this repo [HERE](https://buy.stripe.com/fZe28U09va9Z9IQ9BN)!" |
|
--- |
|
# Function Calling DeciLM 7B |
|
|
|
Purchase access to this model [here](https://buy.stripe.com/fZe28U09va9Z9IQ9BN). |
|
|
|
This model is fine-tuned for function calling. |
|
- The function metadata format is the same as used for OpenAI. |
|
- The model is suitable for commercial use. |
|
- GGUF, AWQ and GPTQ models are not yet available as Deci models are not broadly supported. |
|
|
|
Check out other fine-tuned function calling models [here](https://trelis.com/function-calling/). |
|
|
|
## Quick Server Setup |
|
Runpod one click template [here](https://runpod.io/gsc?template=hv8lwdsgiz&ref=jmfkcdio). You must add a HuggingFace Hub access token (HUGGING_FACE_HUB_TOKEN) to the environment variables as this is a gated model. |
|
|
|
Runpod Affiliate [Link](https://runpod.io?ref=jmfkcdio) (helps support the Trelis channel). |
|
|
|
## Inference Scripts |
|
See below for sample prompt format. |
|
|
|
Complete inference scripts are available for purchase [here](https://trelis.com/enterprise-server-api-and-inference-guide/): |
|
- Easily format prompts using tokenizer.apply_chat_format (starting from openai formatted functions and a list of messages) |
|
- Automate catching, handling and chaining of function calls. |
|
|
|
## Prompt Format |
|
``` |
|
B_FUNC, E_FUNC = "You have access to the following functions. Use them if required:\n\n", "\n\n" |
|
B_INST, E_INST = "\n### User:\n", "\n### Assistant:\n" #DeciLM |
|
prompt = f"{B_INST}{B_FUNC}{functionList.strip()}{E_FUNC}{user_prompt.strip()}{E_INST}\n\n" |
|
``` |
|
### Using tokenizer.apply_chat_template |
|
For an easier application of the prompt, you can set up as follows: |
|
|
|
Set up `messages`: |
|
``` |
|
[ |
|
{ |
|
"role": "function_metadata", |
|
"content": "FUNCTION_METADATA" |
|
}, |
|
{ |
|
"role": "user", |
|
"content": "What is the current weather in London?" |
|
}, |
|
{ |
|
"role": "function_call", |
|
"content": "{\n \"name\": \"get_current_weather\",\n \"arguments\": {\n \"city\": \"London\"\n }\n}" |
|
}, |
|
{ |
|
"role": "function_response", |
|
"content": "{\n \"temperature\": \"15 C\",\n \"condition\": \"Cloudy\"\n}" |
|
}, |
|
{ |
|
"role": "assistant", |
|
"content": "The current weather in London is Cloudy with a temperature of 15 Celsius" |
|
} |
|
] |
|
``` |
|
|
|
with `FUNCTION_METADATA` as: |
|
``` |
|
[ |
|
{ |
|
"type": "function", |
|
"function": { |
|
"name": "get_current_weather", |
|
"description": "This function gets the current weather in a given city", |
|
"parameters": { |
|
"type": "object", |
|
"properties": { |
|
"city": { |
|
"type": "string", |
|
"description": "The city, e.g., San Francisco" |
|
}, |
|
"format": { |
|
"type": "string", |
|
"enum": ["celsius", "fahrenheit"], |
|
"description": "The temperature unit to use." |
|
} |
|
}, |
|
"required": ["city"] |
|
} |
|
} |
|
}, |
|
{ |
|
"type": "function", |
|
"function": { |
|
"name": "get_clothes", |
|
"description": "This function provides a suggestion of clothes to wear based on the current weather", |
|
"parameters": { |
|
"type": "object", |
|
"properties": { |
|
"temperature": { |
|
"type": "string", |
|
"description": "The temperature, e.g., 15 C or 59 F" |
|
}, |
|
"condition": { |
|
"type": "string", |
|
"description": "The weather condition, e.g., 'Cloudy', 'Sunny', 'Rainy'" |
|
} |
|
}, |
|
"required": ["temperature", "condition"] |
|
} |
|
} |
|
} |
|
] |
|
``` |
|
and then apply the chat template to get a formatted prompt: |
|
``` |
|
tokenizer = AutoTokenizer.from_pretrained('Trelis/Mixtral-8x7B-Instruct-v0.1-function-calling-v3', trust_remote_code=True) |
|
|
|
prompt = tokenizer.apply_chat_template(prompt, tokenize=False) |
|
``` |
|
If you are using a gated model, you need to first run: |
|
``` |
|
pip install huggingface_hub |
|
huggingface-cli login |
|
``` |
|
|
|
### Manual Prompt: |
|
``` |
|
### User: |
|
You have access to the following functions. Use them if required: |
|
|
|
[ |
|
{ |
|
"type": "function", |
|
"function": { |
|
"name": "get_stock_price", |
|
"description": "Get the stock price of an array of stocks", |
|
"parameters": { |
|
"type": "object", |
|
"properties": { |
|
"names": { |
|
"type": "array", |
|
"items": { |
|
"type": "string" |
|
}, |
|
"description": "An array of stocks" |
|
} |
|
}, |
|
"required": [ |
|
"names" |
|
] |
|
} |
|
} |
|
}, |
|
{ |
|
"type": "function", |
|
"function": { |
|
"name": "get_big_stocks", |
|
"description": "Get the names of the largest N stocks by market cap", |
|
"parameters": { |
|
"type": "object", |
|
"properties": { |
|
"number": { |
|
"type": "integer", |
|
"description": "The number of largest stocks to get the names of, e.g. 25" |
|
}, |
|
"region": { |
|
"type": "string", |
|
"description": "The region to consider, can be \"US\" or \"World\"." |
|
} |
|
}, |
|
"required": [ |
|
"number" |
|
] |
|
} |
|
} |
|
} |
|
] |
|
|
|
Get the names of the five largest stocks by market cap |
|
### Assistant: |
|
|
|
{ |
|
"name": "get_big_stocks", |
|
"arguments": { |
|
"number": 5, |
|
"region": "US" |
|
} |
|
}</s> |
|
``` |
|
|
|
# Dataset |
|
See [Trelis/function_calling_v3](https://huggingface.co/datasets/Trelis/function_calling_v3). |
|
|
|
# License |
|
This model may be used commercially for inference, or for further fine-tuning and inference. Users may not re-publish or re-sell this model in the same or derivative form (including fine-tunes). |
|
|
|
~~~ |
|
The original repo card follows below. |
|
~~~ |
|
# DeciLM-7B-instruct |
|
|
|
DeciLM-7B-instruct is a model for short-form instruction following. It is built by LoRA fine-tuning on the [SlimOrca dataset](https://huggingface.co/datasets/Open-Orca/SlimOrca). |
|
|
|
### 🔥 Click [here](https://console.deci.ai/infery-llm-demo) for a live demo of DeciLM-7B + Infery! |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
DeciLM-7B-instruct is a derivative of the recently released [DeciLM-7B](https://huggingface.co/Deci/DeciLM-7B) language model, a pre-trained, high-efficiency generative text model with 7 billion parameters. DeciLM-7B-instruct is one the best 7B instruct models obtained using simple LoRA fine-tuning, without relying on preference optimization techniques such as RLHF and DPO. |
|
|
|
- **Developed by:** [Deci](https://deci.ai) |
|
- **Model type:** DeciLM is an auto-regressive language model using an optimized transformer decoder architecture that includes variable Grouped-Query Attention. |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
|
|
## Model Architecture |
|
|
|
| Parameters | Layers | Heads | Sequence Length | GQA num_key_value_heads* | |
|
|:----------|:----------|:----------|:----------|:----------| |
|
| 7.04 billion | 32 | 32 | 8192 | Variable | |
|
|
|
*AutoNAC was employed to optimize the selection of the GQA num_key_value_heads for each model layer. |
|
|
|
|
|
### Model Sources |
|
|
|
- **Blog:** [DeciLM-7B Technical Blog](https://deci.ai/blog/introducing-DeciLM-7B-the-fastest-and-most-accurate-7b-large-language-model-to-date) |
|
- **Demo:** [DeciLM-7B-instruct Demo](https://huggingface.co/spaces/Deci/DeciLM-7B-instruct) |
|
- **Finetuning Notebook:** [DeciLM-7B Finetuning Notebook](https://colab.research.google.com/drive/1kEV6i96AQ94xTCvSd11TxkEaksTb5o3U?usp=sharing) |
|
- **Text Generation Notebook:** [DeciLM-7B-instruct Text Generation Notebook](https://bit.ly/declm-7b-instruct) |
|
|
|
## Uses |
|
|
|
The model is intended for commercial and research use in English. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, pipeline |
|
|
|
model_name = "Deci/DeciLM-7B-instruct" |
|
|
|
device = "cuda" # for GPU usage or "cpu" for CPU usage |
|
|
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit = True, |
|
bnb_4bit_compute_dtype=torch.bfloat16 |
|
) |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
device_map="auto", |
|
trust_remote_code=True, |
|
quantization_config=bnb_config |
|
) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
tokenizer.pad_token = tokenizer.eos_token |
|
|
|
deci_generator = pipeline("text-generation", |
|
model=model, |
|
tokenizer=tokenizer, |
|
temperature=0.1, |
|
device_map="auto", |
|
max_length=4096, |
|
return_full_text=False |
|
) |
|
|
|
prompt = "How do I make the most delicious pancakes the world has ever tasted?" |
|
|
|
SYSTEM_PROMPT_TEMPLATE =""" |
|
### System: |
|
You are an AI assistant that follows instruction extremely well. Help as much as you can. |
|
### User: |
|
{instruction} |
|
### Assistant: |
|
""" |
|
|
|
# Function to construct the prompt using the new system prompt template |
|
def get_prompt_with_template(message: str) -> str: |
|
return SYSTEM_PROMPT_TEMPLATE.format(instruction=message) |
|
|
|
response = deci_generator(get_prompt_with_template(prompt))[0]['generated_text'] |
|
print(response) |
|
``` |
|
|
|
## Evaluation |
|
|
|
Below are DeciLM-7B and DeciLM-7B-instruct's evaluation results. |
|
|
|
| Model | Average | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K | |
|
|:----------|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:| |
|
| DecilLM-7B | 61.55 | 59.39 | 82.51 | 59.76 | 40.33 | 79.95 | 47.38 | |
|
| DecilLM-7B-instruct | 63.19 | 61.01 | 82.37 | 60.24 | 49.75 | 79.72 | 46.02 | |
|
|
|
|
|
|
|
### Runtime Benchmarks |
|
|
|
| Inference Tool | Hardware | Prompt length | Generation length | Generated tokens/sec | Batch Size | Number of Prompts | |
|
|:----------|:----------|:---------:|:---------:|:---------:|:---------:|:---------:| |
|
| HuggingFace (PyTorch) | A100 (SXM4-80GB-400W) | 512 | 512 | **1174** | 352 | 352 | |
|
| HuggingFace (PyTorch) | A100 (SXM4-80GB-400W) | 2048 | 2048 | **328** | 72 | 72 | |
|
| Infery-LLM | A100 (SXM4-80GB-400W)| 512 | 512 | **4559** | 1024 | 4096 | |
|
| Infery-LLM | A100 (SXM4-80GB-400W) | 2048 | 2048 | **3997** | 512 | 2048 | |
|
| Infery-LLM | A10 | 512 | 512 | **1345** | 128 | 512 | |
|
| Infery-LLM | A10 | 2048 | 2048 | **599** | 32 | 128 | |
|
|
|
- In order to replicate the results of the Hugging Face benchmarks, you can use this [code example](https://huggingface.co/Deci/DeciLM-7B/blob/main/benchmark_hf_model.py). |
|
- Infery-LLM, Deci's inference engine, features a suite of optimization algorithms, including selective quantization, optimized beam search, continuous batching, and custom CUDA kernels. To witness the full capabilities of Infery-LLM first-hand, we invite you to engage with our [interactive demo](https://console.deci.ai/infery-llm-demo). |
|
|
|
## Ethical Considerations and Limitations |
|
|
|
DeciLM-7B-instruct is a new technology that comes with inherent risks associated with its use. The testing conducted so far has been primarily in English and does not encompass all possible scenarios. Like those of all large language models, DeciLM-7B's outputs are unpredictable, and the model may generate responses that are inaccurate, biased, or otherwise objectionable. Consequently, developers planning to use DeciLM-7B should undertake thorough safety testing and tuning designed explicitly for their intended applications of the model before deployment. |
|
|
|
## How to Cite |
|
|
|
Please cite this model using this format. |
|
|
|
```bibtex |
|
@misc{DeciFoundationModels, |
|
title = {DeciLM-7B-instruct}, |
|
author = {DeciAI Research Team}, |
|
year = {2023} |
|
url={https://huggingface.co/Deci/DeciLM-7B-instruct}, |
|
} |
|
``` |
