nold commited on
Commit
6911bfe
1 Parent(s): 11d3650

2639ffe98b8c0bdc61fb91727e2dd3dd457036e7b3c853ee5f7cb3e45cf6f9a7

Browse files
Files changed (1) hide show
  1. README.md +352 -0
README.md ADDED
@@ -0,0 +1,352 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ language:
4
+ - en
5
+ - fr
6
+ - de
7
+ - es
8
+ - it
9
+ - pt
10
+ - ja
11
+ - ko
12
+ - zh
13
+ - ar
14
+ license: cc-by-nc-4.0
15
+ ---
16
+
17
+ # Model Card for C4AI Command-R
18
+
19
+ 🚨 **This model is non-quantized version of C4AI Command-R. You can find the quantized version of C4AI Command-R using bitsandbytes [here](https://huggingface.co/CohereForAI/c4ai-command-r-v01-4bit)**.
20
+
21
+ ## Model Summary
22
+
23
+ C4AI Command-R is a research release of a 35 billion parameter highly performant generative model. Command-R is a large language model with open weights optimized for a variety of use cases including reasoning, summarization, and question answering. Command-R has the capability for multilingual generation evaluated in 10 languages and highly performant RAG capabilities.
24
+
25
+ Developed by: Cohere and [Cohere For AI](https://cohere.for.ai)
26
+
27
+ - Point of Contact: Cohere For AI: [cohere.for.ai](https://cohere.for.ai/)
28
+ - License: [CC-BY-NC](https://cohere.com/c4ai-cc-by-nc-license), requires also adhering to [C4AI's Acceptable Use Policy](https://docs.cohere.com/docs/c4ai-acceptable-use-policy)
29
+ - Model: c4ai-command-r-v01
30
+ - Model Size: 35 billion parameters
31
+ - Context length: 128K
32
+
33
+ **Use**
34
+
35
+ ```python
36
+ # pip install transformers
37
+ from transformers import AutoTokenizer, AutoModelForCausalLM
38
+
39
+ model_id = "CohereForAI/c4ai-command-r-v01"
40
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
41
+ model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
42
+
43
+ # Format message with the command-r chat template
44
+ messages = [{"role": "user", "content": "Hello, how are you?"}]
45
+ input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
46
+ ## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
47
+
48
+ gen_tokens = model.generate(
49
+ input_ids,
50
+ max_new_tokens=100,
51
+ do_sample=True,
52
+ temperature=0.3,
53
+ )
54
+
55
+ gen_text = tokenizer.decode(gen_tokens[0])
56
+ print(gen_text)
57
+ ```
58
+
59
+ **Quantized model through bitsandbytes, 8-bit precision**
60
+
61
+ ```python
62
+ # pip install transformers bitsandbytes accelerate
63
+ from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
64
+
65
+ bnb_config = BitsAndBytesConfig(load_in_8bit=True)
66
+
67
+ model_id = "CohereForAI/c4ai-command-r-v01"
68
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
69
+ model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, quantization_config=bnb_config)
70
+
71
+ # Format message with the command-r chat template
72
+ messages = [{"role": "user", "content": "Hello, how are you?"}]
73
+ input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
74
+ ## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
75
+
76
+ gen_tokens = model.generate(
77
+ input_ids,
78
+ max_new_tokens=100,
79
+ do_sample=True,
80
+ temperature=0.3,
81
+ )
82
+
83
+ gen_text = tokenizer.decode(gen_tokens[0])
84
+ print(gen_text)
85
+ ```
86
+
87
+ **Quantized model through bitsandbytes, 4-bit precision**
88
+
89
+ ```python
90
+ # pip install transformers bitsandbytes accelerate
91
+ from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
92
+
93
+ bnb_config = BitsAndBytesConfig(load_in_4bit=True)
94
+
95
+ model_id = "CohereForAI/c4ai-command-r-v01"
96
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
97
+ model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, quantization_config=bnb_config)
98
+
99
+ # Format message with the command-r chat template
100
+ messages = [{"role": "user", "content": "Hello, how are you?"}]
101
+ input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
102
+ ## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
103
+
104
+ gen_tokens = model.generate(
105
+ input_ids,
106
+ max_new_tokens=100,
107
+ do_sample=True,
108
+ temperature=0.3,
109
+ )
110
+
111
+ gen_text = tokenizer.decode(gen_tokens[0])
112
+ print(gen_text)
113
+ ```
114
+
115
+ ## Model Details
116
+
117
+ **Input**: Models input text only.
118
+
119
+ **Output**: Models generate text only.
120
+
121
+ **Model Architecture**: This is an auto-regressive language model that uses an optimized transformer architecture. After pretraining, this model uses supervised fine-tuning (SFT) and preference training to align model behavior to human preferences for helpfulness and safety.
122
+
123
+ **Languages covered**: The model is optimized to perform well in the following languages: English, French, Spanish, Italian, German, Brazilian Portuguese, Japanese, Korean, Simplified Chinese, and Arabic.
124
+
125
+ Pre-training data additionally included the following 13 languages: Russian, Polish, Turkish, Vietnamese, Dutch, Czech, Indonesian, Ukrainian, Romanian, Greek, Hindi, Hebrew, Persian.
126
+
127
+ **Context length**: Command-R supports a context length of 128K.
128
+
129
+ ### Tool use capabilities:
130
+
131
+ Command-R has been specifically trained with conversational tool use capabilities. These have been trained into the model via a mixture of supervised fine-tuning and preference fine-tuning, using a specific prompt template. Deviating from this prompt template will likely reduce performance, but we encourage experimentation.
132
+
133
+ Command-R’s tool use functionality takes a conversation as input (with an optional user-system preamble), along with a list of available tools. The model will then generate a json-formatted list of actions to execute on a subset of those tools. Command-R may use one of its supplied tools more than once.
134
+
135
+ The model has been trained to recognise a special `directly_answer` tool, which it uses to indicate that it doesn’t want to use any of its other tools. We recommend including the `directly_answer` tool, but encourage experimentation.
136
+
137
+ Comprehensive documentation and guides on prompting strategies for tool use will be provided shortly.
138
+
139
+ <details>
140
+ <summary><b>Usage: Rendering Tool Use Prompts [CLICK TO EXPAND]</b> </summary>
141
+
142
+ ```python
143
+ from transformers import AutoTokenizer
144
+
145
+ model_id = "CohereForAI/c4ai-command-r-v01"
146
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
147
+
148
+ # define conversation input:
149
+ conversation = [
150
+ {"role": "user", "content": "Whats the biggest penguin in the world?"}
151
+ ]
152
+ # Define tools available for the model to use:
153
+ tools = [
154
+ {
155
+ "name": "internet_search",
156
+ "description": "Returns a list of relevant document snippets for a textual query retrieved from the internet",
157
+ "parameter_definitions": {
158
+ "query": {
159
+ "description": "Query to search the internet with",
160
+ "type": 'str',
161
+ "required": True
162
+ }
163
+ }
164
+ },
165
+ {
166
+ 'name': "directly_answer",
167
+ "description": "Calls a standard (un-augmented) AI chatbot to generate a response given the conversation history",
168
+ 'parameter_definitions': {}
169
+ }
170
+ ]
171
+
172
+ # render the tool use prompt as a string:
173
+ tool_use_prompt = tokenizer.apply_tool_use_template(
174
+ conversation,
175
+ tools=tools,
176
+ tokenize=False,
177
+ add_generation_prompt=True,
178
+ )
179
+ print(tool_use_prompt)
180
+ ```
181
+
182
+ </details>
183
+
184
+ <details>
185
+ <summary><b>Example Rendered Tool Use Prompt [CLICK TO EXPAND]</b></summary>
186
+
187
+ ````
188
+ <BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># Safety Preamble
189
+ The instructions in this section override those in the task description and style guide sections. Don't answer questions that are harmful or immoral.
190
+
191
+ # System Preamble
192
+ ## Basic Rules
193
+ You are a powerful conversational AI trained by Cohere to help people. You are augmented by a number of tools, and your job is to use and consume the output of these tools to best help the user. You will see a conversation history between yourself and a user, ending with an utterance from the user. You will then see a specific instruction instructing you what kind of response to generate. When you answer the user's requests, you cite your sources in your answers, according to those instructions.
194
+
195
+ # User Preamble
196
+ ## Task and Context
197
+ You help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user's needs as best you can, which will be wide-ranging.
198
+
199
+ ## Style Guide
200
+ Unless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.
201
+
202
+ ## Available Tools
203
+ Here is a list of tools that you have available to you:
204
+
205
+ ```python
206
+ def internet_search(query: str) -> List[Dict]:
207
+ """Returns a list of relevant document snippets for a textual query retrieved from the internet
208
+
209
+ Args:
210
+ query (str): Query to search the internet with
211
+ """
212
+ pass
213
+ ```
214
+
215
+ ```python
216
+ def directly_answer() -> List[Dict]:
217
+ """Calls a standard (un-augmented) AI chatbot to generate a response given the conversation history
218
+ """
219
+ pass
220
+ ```<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Whats the biggest penguin in the world?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>Write 'Action:' followed by a json-formatted list of actions that you want to perform in order to produce a good response to the user's last input. You can use any of the supplied tools any number of times, but you should aim to execute the minimum number of necessary actions for the input. You should use the `directly-answer` tool if calling the other tools is unnecessary. The list of actions you want to call should be formatted as a list of json objects, for example:
221
+ ```json
222
+ [
223
+ {
224
+ "tool_name": title of the tool in the specification,
225
+ "parameters": a dict of parameters to input into the tool as they are defined in the specs, or {} if it takes no parameters
226
+ }
227
+ ]```<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
228
+
229
+ ````
230
+
231
+ </details>
232
+
233
+ <details>
234
+ <summary><b>Example Rendered Tool Use Completion [CLICK TO EXPAND]</b></summary>
235
+
236
+ ````
237
+ Action: ```json
238
+ [
239
+ {
240
+ "tool_name": "internet_search",
241
+ "parameters": {
242
+ "query": "biggest penguin in the world"
243
+ }
244
+ }
245
+ ]
246
+ ```
247
+ ````
248
+ </details>
249
+
250
+ ### Grounded Generation and RAG Capabilities:
251
+
252
+ Command-R has been specifically trained with grounded generation capabilities. This means that it can generate responses based on a list of supplied document snippets, and it will include grounding spans (citations) in its response indicating the source of the information.
253
+ This can be used to enable behaviors such as grounded summarization and the final step of Retrieval Augmented Generation (RAG).This behavior has been trained into the model via a mixture of supervised fine-tuning and preference fine-tuning, using a specific prompt template.
254
+ Deviating from this prompt template may reduce performance, but we encourage experimentation.
255
+
256
+ Command-R’s grounded generation behavior takes a conversation as input (with an optional user-supplied system preamble), along with a list of retrieved document snippets.
257
+ The document snippets should be chunks, rather than long documents, typically around 100-400 words per chunk. Document snippets consist of key-value pairs. The keys should be short descriptive strings, the values can be text or semi-structured.
258
+
259
+ By default, Command-R will generate grounded responses by first predicting which documents are relevant, then predicting which ones it will cite, then generating an answer.
260
+ Finally, it will then insert grounding spans into the answer. See below for an example. This is referred to as `accurate` grounded generation.
261
+
262
+ The model is trained with a number of other answering modes, which can be selected by prompt changes . A `fast` citation mode is supported in the tokenizer, which will directly generate an answer with grounding spans in it, without first writing the answer out in full. This sacrifices some grounding accuracy in favor of generating fewer tokens.
263
+
264
+ The code snippet below shows a minimal working example on how to render a prompt, generate and parse a completion.
265
+
266
+ Comprehensive documentation and guides on prompting strategies on grounded generation will be provided in follow-ups at a later stage.
267
+
268
+ <details>
269
+ <summary> <b>Usage: Rendering Grounded Generation prompts [CLICK TO EXPAND]</b> </summary>
270
+
271
+ ````python
272
+ from transformers import AutoTokenizer
273
+
274
+ model_id = "CohereForAI/c4ai-command-r-v01"
275
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
276
+
277
+ # define conversation input:
278
+ conversation = [
279
+ {"role": "user", "content": "Whats the biggest penguin in the world?"}
280
+ ]
281
+ # define documents to ground on:
282
+ documents = [
283
+ { "title": "Tall penguins", "text": "Emperor penguins are the tallest growing up to 122 cm in height." },
284
+ { "title": "Penguin habitats", "text": "Emperor penguins only live in Antarctica."}
285
+ ]
286
+
287
+ # render the tool use prompt as a string:
288
+ grounded_generation_prompt = tokenizer.apply_grounded_generation_template(
289
+ conversation,
290
+ documents=documents,
291
+ citation_mode="accurate", # or "fast"
292
+ tokenize=False,
293
+ add_generation_prompt=True,
294
+ )
295
+ print(grounded_generation_prompt)
296
+ ````
297
+ </details>
298
+
299
+ <details>
300
+ <summary><b>Example Rendered Grounded Generation Prompt [CLICK TO EXPAND]</b></summary>
301
+
302
+ ````<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># Safety Preamble
303
+ The instructions in this section override those in the task description and style guide sections. Don't answer questions that are harmful or immoral.
304
+
305
+ # System Preamble
306
+ ## Basic Rules
307
+ You are a powerful conversational AI trained by Cohere to help people. You are augmented by a number of tools, and your job is to use and consume the output of these tools to best help the user. You will see a conversation history between yourself and a user, ending with an utterance from the user. You will then see a specific instruction instructing you what kind of response to generate. When you answer the user's requests, you cite your sources in your answers, according to those instructions.
308
+
309
+ # User Preamble
310
+ ## Task and Context
311
+ You help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user's needs as best you can, which will be wide-ranging.
312
+
313
+ ## Style Guide
314
+ Unless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Whats the biggest penguin in the world?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|><results>
315
+ Document: 0
316
+ title: Tall penguins
317
+ text: Emperor penguins are the tallest growing up to 122 cm in height.
318
+
319
+ Document: 1
320
+ title: Penguin habitats
321
+ text: Emperor penguins only live in Antarctica.
322
+ </results><|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>Carefully perform the following instructions, in order, starting each with a new line.
323
+ Firstly, Decide which of the retrieved documents are relevant to the user's last input by writing 'Relevant Documents:' followed by comma-separated list of document numbers. If none are relevant, you should instead write 'None'.
324
+ Secondly, Decide which of the retrieved documents contain facts that should be cited in a good answer to the user's last input by writing 'Cited Documents:' followed a comma-separated list of document numbers. If you dont want to cite any of them, you should instead write 'None'.
325
+ Thirdly, Write 'Answer:' followed by a response to the user's last input in high quality natural english. Use the retrieved documents to help you. Do not insert any citations or grounding markup.
326
+ Finally, Write 'Grounded answer:' followed by a response to the user's last input in high quality natural english. Use the symbols <co: doc> and </co: doc> to indicate when a fact comes from a document in the search result, e.g <co: 0>my fact</co: 0> for a fact from document 0.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
327
+ ````
328
+
329
+ </details>
330
+
331
+ <details>
332
+ <summary><b>Example Rendered Grounded Generation Completion [CLICK TO EXPAND]</b></summary>
333
+
334
+ ````
335
+ Relevant Documents: 0,1
336
+ Cited Documents: 0,1
337
+ Answer: The Emperor Penguin is the tallest or biggest penguin in the world. It is a bird that lives only in Antarctica and grows to a height of around 122 centimetres.
338
+ Grounded answer: The <co: 0>Emperor Penguin</co: 0> is the <co: 0>tallest</co: 0> or biggest penguin in the world. It is a bird that <co: 1>lives only in Antarctica</co: 1> and <co: 0>grows to a height of around 122 centimetres.</co: 0>
339
+ ````
340
+ </details>
341
+
342
+ ### Code Capabilities:
343
+ Command-R has been optimized to interact with your code, by requesting code snippets, code explanations, or code rewrites. It might not perform well out-of-the-box for pure code completion. For better performance, we also recommend using a low temperature (and even greedy decoding) for code-generation related instructions.
344
+
345
+ ### Model Card Contact
346
+ For errors or additional questions about details in this model card, contact [info@for.ai](mailto:info@for.ai).
347
+
348
+ ### Terms of Use:
349
+ We hope that the release of this model will make community-based research efforts more accessible, by releasing the weights of a highly performant 35 billion parameter model to researchers all over the world. This model is governed by a [CC-BY-NC](https://cohere.com/c4ai-cc-by-nc-license) License with an acceptable use addendum, and also requires adhering to [C4AI's Acceptable Use Policy](https://docs.cohere.com/docs/c4ai-acceptable-use-policy).
350
+
351
+ ### Try Chat:
352
+ You can try Command-R chat in the playground [here](https://dashboard.cohere.com/playground/chat).