Hugging Face's logo

Spaces:
gorilla-llm
/
berkeley-function-calling-leaderboard
Running

App Files Community
2
Huanzhi Mao commited on
Commit
e8e0a80
•
1 Parent(s): 6a966af

update data.csv

Browse files
Files changed (1) hide show
  1. data.csv +49 -47
data.csv CHANGED
@@ -1,48 +1,50 @@
1
  Rank,Overall Acc,Model,Model Link,Organization,License,AST Summary,Exec Summary,Simple Function AST,Python Simple Function AST,Java Simple Function AST,JavaScript Simple Function AST,Multiple Functions AST,Parallel Functions AST,Parallel Multiple AST,Simple Function Exec,Python Simple Function Exec,REST Simple Function Exec,Multiple Functions Exec,Parallel Functions Exec,Parallel Multiple Exec,Relevance Detection,Cost ($ Per 1k Function Calls),Latency Mean (s),Latency Standard Deviation (s),Latency 95th Percentile (s)
2
- 1,90.00%,Claude-3.5-Sonnet-20240620 (Prompt),https://www.anthropic.com/news/claude-3-5-sonnet,Anthropic,Proprietary,91.31%,89.50%,86.73%,94.00%,65.00%,72.00%,95.50%,92.00%,91.00%,100.00%,100.00%,100.00%,96.00%,82.00%,80.00%,85.42%,2.2,1.37,1.22,2.11
3
- 2,88.00%,GPT-4-0125-Preview (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,91.22%,88.10%,88.36%,94.75%,67.00%,80.00%,95.00%,90.50%,91.00%,99.41%,100.00%,98.57%,94.00%,84.00%,75.00%,70.42%,5.25,1.97,1.31,4.43
4
- 3,87.65%,Claude-3-Opus-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,88.93%,86.16%,86.73%,93.75%,66.00%,72.00%,94.00%,86.00%,89.00%,97.65%,98.00%,97.14%,92.00%,80.00%,75.00%,80.42%,10.84,4.61,1.53,7.32
5
- 4,86.35%,Gemini-1.5-Pro-Preview-0514 (FC),https://deepmind.google/technologies/gemini/pro/,Google,Proprietary,87.92%,83.32%,80.18%,92.00%,50.00%,46.00%,92.00%,91.00%,88.50%,91.76%,96.00%,85.71%,88.00%,76.00%,77.50%,89.58%,0.86,1.94,0.79,3.46
6
- 5,85.88%,Gemini-1.5-Pro-Preview-0409 (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,87.62%,82.88%,80.00%,91.75%,53.00%,40.00%,92.50%,90.00%,88.00%,90.00%,96.00%,81.43%,90.00%,74.00%,77.50%,88.75%,0.86,1.95,0.89,3.53
7
- 6,85.88%,GPT-4-1106-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,88.62%,81.73%,84.00%,93.00%,61.00%,58.00%,91.50%,92.00%,87.00%,89.41%,95.00%,81.43%,92.00%,78.00%,67.50%,80.42%,5.07,5.97,6.31,18.14
8
- 7,85.59%,GPT-4-turbo-2024-04-09 (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,90.01%,86.04%,86.55%,94.00%,60.00%,80.00%,95.00%,90.00%,88.50%,97.65%,97.00%,98.57%,94.00%,80.00%,72.50%,62.50%,5.25,2.57,2.2,5.72
9
- 8,84.71%,Gorilla-OpenFunctions-v2 (FC),https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html,Gorilla LLM,Apache 2.0,89.38%,81.55%,88.00%,94.75%,67.00%,76.00%,95.00%,87.50%,87.00%,94.71%,94.00%,95.71%,94.00%,70.00%,67.50%,61.25%,0.31,0.05,N/A,N/A
10
- 9,84.65%,GPT-4-0125-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,87.17%,84.76%,80.18%,90.50%,53.00%,52.00%,93.00%,90.50%,85.00%,83.53%,98.00%,62.86%,92.00%,86.00%,77.50%,82.92%,4.83,4.72,5.76,18.62
11
- 10,83.88%,Meta-Llama-3-70B-Instruct (Prompt),https://llama.meta.com/llama3,Meta,Meta Llama 3 Community,87.74%,85.32%,81.45%,91.00%,50.00%,68.00%,93.00%,91.50%,85.00%,91.76%,95.00%,87.14%,88.00%,84.00%,77.50%,69.17%,1.1,0.18,N/A,N/A
12
- 11,82.94%,GPT-4o-2024-05-13 (FC),https://openai.com/index/hello-gpt-4o/,OpenAI,Proprietary,85.23%,80.37%,78.91%,88.25%,56.00%,50.00%,90.00%,87.50%,84.50%,86.47%,94.00%,75.71%,78.00%,82.00%,75.00%,81.25%,2.33,2.09,2.52,6.93
13
- 12,82.88%,GPT-4-turbo-2024-04-09 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,85.56%,78.61%,74.73%,91.25%,33.00%,26.00%,90.00%,89.50%,88.00%,82.94%,93.00%,68.57%,88.00%,76.00%,67.50%,88.75%,4.78,5.48,6.37,18.51
14
- 13,81.88%,FireFunction-v2 (FC),https://huggingface.co/fireworks-ai/firefunction-v2,Fireworks,Apache 2.0,86.44%,80.26%,85.27%,94.50%,59.00%,64.00%,91.00%,89.50%,80.00%,93.53%,94.00%,92.86%,88.00%,72.00%,67.50%,56.67%,N/A,1.04,0.81,1.97
15
- 14,81.82%,Claude-3-Sonnet-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,87.40%,86.76%,83.09%,92.00%,53.00%,72.00%,89.00%,88.00%,89.50%,93.53%,96.00%,90.00%,92.00%,84.00%,77.50%,51.25%,2.13,1.95,1.16,3.08
16
- 15,81.35%,Mistral-Medium-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,83.76%,73.47%,80.55%,90.25%,54.00%,56.00%,92.00%,84.00%,78.50%,65.88%,96.00%,22.86%,76.00%,82.00%,70.00%,88.33%,1.76,2.75,2.15,6.38
17
- 16,80.53%,GPT-4o-2024-05-13 (Prompt),https://openai.com/index/hello-gpt-4o/,OpenAI,Proprietary,77.15%,77.62%,85.09%,90.75%,68.00%,74.00%,84.00%,78.50%,61.00%,90.00%,95.00%,82.86%,78.00%,70.00%,72.50%,82.50%,2.67,1.15,0.78,2.67
18
- 17,80.47%,Functionary-Medium-v2.4 (FC),https://huggingface.co/meetkai/functionary-medium-v2.4,MeetKai,MIT,85.61%,75.71%,79.45%,88.00%,55.00%,60.00%,90.50%,87.50%,85.00%,68.82%,85.00%,45.71%,84.00%,80.00%,70.00%,74.17%,N/A,2.49,2.69,7.45
19
- 18,80.35%,Gemini-1.5-Flash-Preview-0514 (FC),https://deepmind.google/technologies/gemini/flash/,Google,Proprietary,81.48%,74.57%,80.91%,91.00%,58.00%,46.00%,93.50%,78.00%,73.50%,81.76%,94.00%,64.29%,90.00%,54.00%,72.50%,79.58%,0.07,1.0,0.49,1.54
20
- 19,80.35%,Command-R-Plus (Prompt) (Optimized),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,83.60%,86.74%,82.91%,89.75%,64.00%,66.00%,88.50%,81.00%,82.00%,92.94%,97.00%,87.14%,90.00%,84.00%,80.00%,54.17%,1.9,1.27,0.93,3.24
21
- 20,80.29%,Command-R-Plus (Prompt) (Original),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,83.89%,86.24%,82.55%,89.75%,63.00%,64.00%,90.00%,80.00%,83.00%,92.94%,98.00%,85.71%,88.00%,84.00%,80.00%,53.75%,1.9,1.32,0.94,3.25
22
- 21,79.94%,Functionary-Small-v2.4 (FC),https://huggingface.co/meetkai/functionary-small-v2.4,MeetKai,MIT,83.55%,76.31%,82.18%,91.75%,56.00%,58.00%,88.50%,82.00%,81.50%,78.24%,96.00%,52.86%,82.00%,80.00%,65.00%,67.92%,N/A,2.43,2.55,7.18
23
- 22,79.76%,Command-R-Plus (FC) (Optimized),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,85.15%,77.17%,79.09%,89.75%,46.00%,60.00%,91.00%,88.00%,82.50%,81.18%,95.00%,61.43%,86.00%,74.00%,67.50%,63.75%,1.12,1.9,1.34,4.0
24
- 23,77.47%,Claude-3-Opus-20240229 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,73.06%,71.27%,82.73%,89.50%,61.00%,72.00%,91.50%,58.00%,60.00%,90.59%,97.00%,81.43%,94.00%,38.00%,62.50%,82.50%,30.87,12.92,3.95,20.48
25
- 24,76.47%,Claude-instant-1.2 (Prompt),https://www.anthropic.com/news/releasing-claude-instant-1-2,Anthropic,Proprietary,78.95%,77.93%,79.82%,87.00%,56.00%,70.00%,85.50%,83.00%,67.50%,84.71%,94.00%,71.43%,80.00%,82.00%,65.00%,57.50%,0.45,1.21,0.69,2.22
26
- 25,76.47%,Claude-3.5-Sonnet-20240620 (FC),https://www.anthropic.com/news/claude-3-5-sonnet,Anthropic,Proprietary,72.69%,59.51%,85.27%,91.50%,67.00%,72.00%,92.00%,59.00%,54.50%,97.06%,98.00%,95.71%,88.00%,18.00%,35.00%,78.33%,4.73,3.59,2.37,7.98
27
- 26,74.29%,Claude-3-Haiku-20240307 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,79.10%,70.49%,84.91%,93.50%,55.00%,76.00%,91.50%,84.50%,55.50%,92.94%,100.00%,82.86%,94.00%,70.00%,25.00%,34.58%,0.18,1.0,0.49,1.72
28
- 27,71.41%,Claude-2.1 (Prompt),https://www.anthropic.com/news/claude-2-1,Anthropic,Proprietary,66.05%,62.17%,80.18%,88.75%,54.00%,64.00%,76.00%,55.50%,52.50%,71.18%,90.00%,44.29%,84.00%,46.00%,47.50%,83.33%,4.81,3.27,2.13,7.38
29
- 28,70.94%,Command-R-Plus (FC) (Original),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,80.85%,73.19%,74.91%,84.50%,45.00%,58.00%,90.00%,82.00%,76.50%,81.76%,92.00%,67.14%,88.00%,68.00%,55.00%,24.17%,1.09,1.9,0.99,3.99
30
- 29,68.76%,Mistral-large-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,64.73%,60.01%,66.91%,89.50%,5.00%,10.00%,94.50%,25.50%,72.00%,83.53%,99.00%,61.43%,96.00%,8.00%,52.50%,84.17%,2.47,3.02,2.94,8.85
31
- 30,67.35%,Nexusflow-Raven-v2 (FC),https://huggingface.co/Nexusflow/NexusRaven-V2-13B,Nexusflow,Apache 2.0,65.19%,73.89%,75.27%,80.75%,56.00%,70.00%,86.00%,41.50%,58.00%,67.06%,95.00%,27.14%,92.00%,74.00%,62.50%,57.50%,N/A,2.07,1.23,4.48
32
- 31,67.00%,Gemini-1.0-Pro-001 (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,56.77%,56.74%,79.09%,93.00%,42.00%,42.00%,92.50%,30.00%,25.50%,86.47%,89.00%,82.86%,84.00%,44.00%,12.50%,80.00%,0.13,1.27,1.0,3.39
33
- 32,65.88%,DBRX-Instruct (Prompt),https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm,Databricks,Databricks Open Model,66.62%,74.92%,64.00%,75.75%,30.00%,38.00%,71.50%,72.00%,59.00%,71.18%,80.00%,58.57%,86.00%,80.00%,62.50%,55.83%,1.25,0.64,0.41,1.34
34
- 33,65.18%,Snowflake/snowflake-arctic-instruct (Prompt),https://huggingface.co/Snowflake/snowflake-arctic-instruct,Snowflake,apache-2.0,61.09%,80.04%,62.36%,67.50%,42.00%,62.00%,69.00%,59.00%,54.00%,87.65%,91.00%,82.86%,86.00%,74.00%,72.50%,59.58%,N/A,0.98,0.56,2.13
35
- 34,64.35%,Mistral-large-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,71.49%,64.93%,81.45%,89.50%,62.00%,56.00%,93.50%,31.50%,79.50%,94.71%,95.00%,94.29%,92.00%,8.00%,65.00%,0.00%,1.97,2.07,1.33,4.97
36
- 35,63.88%,GPT-3.5-Turbo-0125 (FC),https://platform.openai.com/docs/models/gpt-3-5-turbo,OpenAI,Proprietary,74.74%,81.38%,61.45%,63.50%,53.00%,62.00%,66.00%,90.50%,81.00%,93.53%,95.00%,91.43%,80.00%,82.00%,70.00%,2.08%,0.19,1.27,0.74,2.47
37
- 36,60.41%,Mistral-small-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,65.32%,52.62%,81.27%,90.50%,56.00%,58.00%,96.00%,39.00%,45.00%,96.47%,100.00%,91.43%,92.00%,12.00%,10.00%,0.00%,0.48,1.14,0.81,2.52
38
- 37,60.12%,Meta-Llama-3-8B-Instruct (Prompt),https://llama.meta.com/llama3,Meta,Meta Llama 3 Community,62.65%,69.95%,55.09%,58.50%,45.00%,48.00%,73.50%,58.00%,64.00%,75.29%,79.00%,70.00%,74.00%,68.00%,62.50%,43.33%,0.24,0.04,N/A,N/A
39
- 38,59.71%,Hermes-2-Pro-Mistral-7B (FC),https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B,NousResearch,apache-2.0,70.25%,55.62%,72.00%,81.75%,42.00%,54.00%,80.50%,67.00%,61.50%,56.47%,78.00%,25.71%,70.00%,56.00%,40.00%,10.83%,0.49,0.08,N/A,N/A
40
- 39,59.24%,Claude-3-Sonnet-20240229 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,44.18%,43.32%,76.73%,86.00%,49.00%,58.00%,88.00%,6.00%,6.00%,85.29%,96.00%,70.00%,88.00%,0.00%,0.00%,81.67%,3.44,3.25,1.46,6.85
41
- 40,53.82%,Claude-3-Haiku-20240307 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,45.05%,46.79%,86.18%,95.50%,60.00%,64.00%,93.50%,0.50%,0.00%,91.18%,96.00%,84.29%,94.00%,2.00%,0.00%,20.83%,0.29,1.49,0.61,2.4
42
- 41,53.65%,FireFunction-v1 (FC),https://huggingface.co/fireworks-ai/firefunction-v1,Fireworks,Apache 2.0,40.75%,39.79%,70.00%,90.25%,13.00%,22.00%,93.00%,0.00%,0.00%,71.18%,95.00%,37.14%,88.00%,0.00%,0.00%,73.33%,N/A,1.69,1.53,4.61
43
- 42,53.59%,GPT-4-0613 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,39.20%,38.53%,63.82%,86.50%,4.00%,2.00%,93.00%,0.00%,0.00%,64.12%,95.00%,20.00%,90.00%,0.00%,0.00%,91.67%,10.37,3.49,3.27,10.88
44
- 43,52.65%,Mistral-tiny-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,49.66%,36.16%,55.64%,70.00%,26.00%,0.00%,56.50%,47.50%,39.00%,27.65%,46.00%,1.43%,20.00%,62.00%,35.00%,83.75%,0.13,1.45,1.41,4.39
45
- 44,43.71%,Gemma-7b-it (Prompt),https://blog.google/technology/developers/gemma-open-models/,Google,gemma-terms-of-use,41.05%,31.75%,42.18%,47.75%,29.00%,24.00%,48.00%,30.00%,44.00%,30.00%,44.00%,10.00%,32.00%,40.00%,25.00%,70.83%,0.37,0.06,N/A,N/A
46
- 45,40.76%,Mistral-Small-2402 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,40.33%,38.03%,5.82%,6.00%,6.00%,4.00%,8.00%,79.00%,68.50%,34.12%,6.00%,74.29%,20.00%,68.00%,30.00%,98.33%,0.64,1.11,0.95,3.03
47
- 46,40.41%,Deepseek-v1.5 (Prompt),https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5,Deepseek,Deepseek License,38.44%,30.89%,39.27%,50.00%,4.00%,24.00%,49.00%,37.00%,28.50%,37.06%,38.00%,35.71%,38.00%,36.00%,12.50%,57.08%,3.24,0.53,N/A,N/A
48
- 47,23.71%,Mistral-small-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,2.62%,34.37%,2.00%,2.75%,0.00%,0.00%,2.50%,3.00%,3.00%,56.47%,79.00%,24.29%,70.00%,6.00%,5.00%,99.58%,0.97,3.06,1.8,6.23
 
 
 
1
  Rank,Overall Acc,Model,Model Link,Organization,License,AST Summary,Exec Summary,Simple Function AST,Python Simple Function AST,Java Simple Function AST,JavaScript Simple Function AST,Multiple Functions AST,Parallel Functions AST,Parallel Multiple AST,Simple Function Exec,Python Simple Function Exec,REST Simple Function Exec,Multiple Functions Exec,Parallel Functions Exec,Parallel Multiple Exec,Relevance Detection,Cost ($ Per 1k Function Calls),Latency Mean (s),Latency Standard Deviation (s),Latency 95th Percentile (s)
2
+ 1,90.18%,Claude-3.5-Sonnet-20240620 (Prompt),https://www.anthropic.com/news/claude-3-5-sonnet,Anthropic,Proprietary,91.68%,89.50%,86.73%,94.00%,65.00%,72.00%,95.50%,92.50%,92.00%,100.00%,100.00%,100.00%,96.00%,82.00%,80.00%,85.42%,2.2,1.37,1.22,2.11
3
+ 2,88.29%,GPT-4-0125-Preview (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,91.84%,88.10%,88.36%,94.75%,67.00%,80.00%,95.00%,92.00%,92.00%,99.41%,100.00%,98.57%,94.00%,84.00%,75.00%,70.42%,5.25,1.97,1.31,4.43
4
+ 3,87.71%,Claude-3-Opus-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,89.06%,86.16%,86.73%,93.75%,66.00%,72.00%,94.00%,86.50%,89.00%,97.65%,98.00%,97.14%,92.00%,80.00%,75.00%,80.42%,10.84,4.61,1.53,7.32
5
+ 4,86.53%,Nemotron-4-340b-instruct (Prompt),https://huggingface.co/nvidia/nemotron-4-340b-instruct,NVIDIA,nvidia-open-model-license,87.99%,88.43%,83.45%,90.75%,58.00%,76.00%,92.50%,90.50%,85.50%,98.24%,99.00%,97.14%,96.00%,82.00%,77.50%,78.33%,N/A,7.65,5.65,18.46
6
+ 5,86.35%,Gemini-1.5-Pro-Preview-0514 (FC),https://deepmind.google/technologies/gemini/pro/,Google,Proprietary,87.92%,83.32%,80.18%,92.00%,50.00%,46.00%,92.00%,91.50%,88.00%,91.76%,96.00%,85.71%,88.00%,76.00%,77.50%,89.58%,0.86,1.94,0.79,3.46
7
+ 6,85.88%,Gemini-1.5-Pro-Preview-0409 (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,87.62%,82.88%,80.00%,91.75%,53.00%,40.00%,92.50%,90.50%,87.50%,90.00%,96.00%,81.43%,90.00%,74.00%,77.50%,88.75%,0.86,1.95,0.89,3.53
8
+ 7,85.88%,GPT-4-1106-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,88.62%,81.73%,84.00%,93.00%,61.00%,58.00%,91.50%,92.50%,86.50%,89.41%,95.00%,81.43%,92.00%,78.00%,67.50%,80.42%,5.07,5.97,6.31,18.14
9
+ 8,85.88%,GPT-4-turbo-2024-04-09 (Prompt),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,90.64%,86.04%,86.55%,94.00%,60.00%,80.00%,95.00%,91.00%,90.00%,97.65%,97.00%,98.57%,94.00%,80.00%,72.50%,62.50%,5.25,2.57,2.2,5.72
10
+ 9,84.65%,Gorilla-OpenFunctions-v2 (FC),https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html,Gorilla LLM,Apache 2.0,89.25%,81.55%,88.00%,94.75%,67.00%,76.00%,95.00%,87.50%,86.50%,94.71%,94.00%,95.71%,94.00%,70.00%,67.50%,61.25%,0.31,0.05,N/A,N/A
11
+ 10,84.59%,GPT-4-0125-Preview (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,87.05%,84.76%,80.18%,90.50%,53.00%,52.00%,93.00%,90.50%,84.50%,83.53%,98.00%,62.86%,92.00%,86.00%,77.50%,82.92%,4.83,4.72,5.76,18.62
12
+ 11,84.00%,Meta-Llama-3-70B-Instruct (Prompt),https://llama.meta.com/llama3,Meta,Meta Llama 3 Community,87.99%,85.32%,81.45%,91.00%,50.00%,68.00%,93.00%,91.50%,86.00%,91.76%,95.00%,87.14%,88.00%,84.00%,77.50%,69.17%,1.1,0.18,N/A,N/A
13
+ 12,83.00%,GPT-4o-2024-05-13 (FC),https://openai.com/index/hello-gpt-4o/,OpenAI,Proprietary,85.35%,80.37%,78.91%,88.25%,56.00%,50.00%,90.00%,88.00%,84.50%,86.47%,94.00%,75.71%,78.00%,82.00%,75.00%,81.25%,2.33,2.09,2.52,6.93
14
+ 13,82.94%,GPT-4-turbo-2024-04-09 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,85.68%,78.61%,74.73%,91.25%,33.00%,26.00%,90.00%,90.00%,88.00%,82.94%,93.00%,68.57%,88.00%,76.00%,67.50%,88.75%,4.78,5.48,6.37,18.51
15
+ 14,82.06%,FireFunction-v2 (FC),https://huggingface.co/fireworks-ai/firefunction-v2,Fireworks,Apache 2.0,86.82%,80.26%,85.27%,94.50%,59.00%,64.00%,91.00%,89.50%,81.50%,93.53%,94.00%,92.86%,88.00%,72.00%,67.50%,56.67%,N/A,1.04,0.81,1.97
16
+ 15,81.88%,Claude-3-Sonnet-20240229 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,87.52%,86.76%,83.09%,92.00%,53.00%,72.00%,89.00%,88.50%,89.50%,93.53%,96.00%,90.00%,92.00%,84.00%,77.50%,51.25%,2.13,1.95,1.16,3.08
17
+ 16,81.65%,Mistral-Medium-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,84.39%,73.47%,80.55%,90.25%,54.00%,56.00%,92.00%,84.00%,81.00%,65.88%,96.00%,22.86%,76.00%,82.00%,70.00%,88.33%,1.76,2.75,2.15,6.38
18
+ 17,80.59%,Command-R-Plus (Prompt) (Optimized),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,84.10%,86.74%,82.91%,89.75%,64.00%,66.00%,88.50%,82.00%,83.00%,92.94%,97.00%,87.14%,90.00%,84.00%,80.00%,54.17%,1.9,1.27,0.93,3.24
19
+ 18,80.53%,GPT-4o-2024-05-13 (Prompt),https://openai.com/index/hello-gpt-4o/,OpenAI,Proprietary,77.15%,77.62%,85.09%,90.75%,68.00%,74.00%,84.00%,78.50%,61.00%,90.00%,95.00%,82.86%,78.00%,70.00%,72.50%,82.50%,2.67,1.15,0.78,2.67
20
+ 19,80.53%,Command-R-Plus (Prompt) (Original),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,84.39%,86.24%,82.55%,89.75%,63.00%,64.00%,90.00%,81.00%,84.00%,92.94%,98.00%,85.71%,88.00%,84.00%,80.00%,53.75%,1.9,1.32,0.94,3.25
21
+ 20,80.41%,Functionary-Medium-v2.4 (FC),https://huggingface.co/meetkai/functionary-medium-v2.4,MeetKai,MIT,85.49%,75.71%,79.45%,88.00%,55.00%,60.00%,90.50%,87.50%,84.50%,68.82%,85.00%,45.71%,84.00%,80.00%,70.00%,74.17%,N/A,2.49,2.69,7.45
22
+ 21,80.29%,Gemini-1.5-Flash-Preview-0514 (FC),https://deepmind.google/technologies/gemini/flash/,Google,Proprietary,81.35%,74.57%,80.91%,91.00%,58.00%,46.00%,93.50%,78.00%,73.00%,81.76%,94.00%,64.29%,90.00%,54.00%,72.50%,79.58%,0.07,1.0,0.49,1.54
23
+ 22,79.88%,Functionary-Small-v2.4 (FC),https://huggingface.co/meetkai/functionary-small-v2.4,MeetKai,MIT,83.42%,76.31%,82.18%,91.75%,56.00%,58.00%,88.50%,82.00%,81.00%,78.24%,96.00%,52.86%,82.00%,80.00%,65.00%,67.92%,N/A,2.43,2.55,7.18
24
+ 23,79.76%,Command-R-Plus (FC) (Optimized),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,85.15%,77.17%,79.09%,89.75%,46.00%,60.00%,91.00%,88.50%,82.00%,81.18%,95.00%,61.43%,86.00%,74.00%,67.50%,63.75%,1.12,1.9,1.34,4.0
25
+ 24,77.76%,Claude-3-Opus-20240229 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,73.68%,71.27%,82.73%,89.50%,61.00%,72.00%,91.50%,58.50%,62.00%,90.59%,97.00%,81.43%,94.00%,38.00%,62.50%,82.50%,30.87,12.92,3.95,20.48
26
+ 25,76.71%,Claude-instant-1.2 (Prompt),https://www.anthropic.com/news/releasing-claude-instant-1-2,Anthropic,Proprietary,79.45%,77.93%,79.82%,87.00%,56.00%,70.00%,85.50%,83.00%,69.50%,84.71%,94.00%,71.43%,80.00%,82.00%,65.00%,57.50%,0.45,1.21,0.69,2.22
27
+ 26,76.47%,Claude-3.5-Sonnet-20240620 (FC),https://www.anthropic.com/news/claude-3-5-sonnet,Anthropic,Proprietary,72.69%,59.51%,85.27%,91.50%,67.00%,72.00%,92.00%,59.00%,54.50%,97.06%,98.00%,95.71%,88.00%,18.00%,35.00%,78.33%,4.73,3.59,2.37,7.98
28
+ 27,74.35%,Claude-3-Haiku-20240307 (Prompt),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,79.23%,70.49%,84.91%,93.50%,55.00%,76.00%,91.50%,84.50%,56.00%,92.94%,100.00%,82.86%,94.00%,70.00%,25.00%,34.58%,0.18,1.0,0.49,1.72
29
+ 28,71.47%,Claude-2.1 (Prompt),https://www.anthropic.com/news/claude-2-1,Anthropic,Proprietary,66.17%,62.17%,80.18%,88.75%,54.00%,64.00%,76.00%,55.50%,53.00%,71.18%,90.00%,44.29%,84.00%,46.00%,47.50%,83.33%,4.81,3.27,2.13,7.38
30
+ 29,70.88%,Command-R-Plus (FC) (Original),https://txt.cohere.com/command-r-plus-microsoft-azure,Cohere For AI,cc-by-nc-4.0,80.73%,73.19%,74.91%,84.50%,45.00%,58.00%,90.00%,82.00%,76.00%,81.76%,92.00%,67.14%,88.00%,68.00%,55.00%,24.17%,1.09,1.9,0.99,3.99
31
+ 30,68.76%,Mistral-large-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,64.73%,60.01%,66.91%,89.50%,5.00%,10.00%,94.50%,25.50%,72.00%,83.53%,99.00%,61.43%,96.00%,8.00%,52.50%,84.17%,2.47,3.02,2.94,8.85
32
+ 31,68.06%,Nexusflow-Raven-v2 (FC),https://huggingface.co/Nexusflow/NexusRaven-V2-13B,Nexusflow,Apache 2.0,66.69%,73.89%,75.27%,80.75%,56.00%,70.00%,86.00%,44.50%,61.00%,67.06%,95.00%,27.14%,92.00%,74.00%,62.50%,57.50%,N/A,2.07,1.23,4.48
33
+ 32,67.06%,Gemini-1.0-Pro-001 (FC),https://deepmind.google/technologies/gemini/#introduction,Google,Proprietary,56.90%,56.74%,79.09%,93.00%,42.00%,42.00%,92.50%,30.50%,25.50%,86.47%,89.00%,82.86%,84.00%,44.00%,12.50%,80.00%,0.13,1.27,1.0,3.39
34
+ 33,66.06%,DBRX-Instruct (Prompt),https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm,Databricks,Databricks Open Model,67.00%,74.92%,64.00%,75.75%,30.00%,38.00%,71.50%,72.50%,60.00%,71.18%,80.00%,58.57%,86.00%,80.00%,62.50%,55.83%,1.25,0.64,0.41,1.34
35
+ 34,65.12%,Snowflake/snowflake-arctic-instruct (Prompt),https://huggingface.co/Snowflake/snowflake-arctic-instruct,Snowflake,apache-2.0,60.97%,80.04%,62.36%,67.50%,42.00%,62.00%,69.00%,59.00%,53.50%,87.65%,91.00%,82.86%,86.00%,74.00%,72.50%,59.58%,N/A,0.98,0.56,2.13
36
+ 35,64.29%,Mistral-large-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,71.36%,64.93%,81.45%,89.50%,62.00%,56.00%,93.50%,31.50%,79.00%,94.71%,95.00%,94.29%,92.00%,8.00%,65.00%,0.00%,1.97,2.07,1.33,4.97
37
+ 36,63.94%,GPT-3.5-Turbo-0125 (FC),https://platform.openai.com/docs/models/gpt-3-5-turbo,OpenAI,Proprietary,74.86%,81.38%,61.45%,63.50%,53.00%,62.00%,66.00%,91.00%,81.00%,93.53%,95.00%,91.43%,80.00%,82.00%,70.00%,2.08%,0.19,1.27,0.74,2.47
38
+ 37,60.53%,Mistral-small-2402 (FC Any),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,65.57%,52.62%,81.27%,90.50%,56.00%,58.00%,96.00%,39.00%,46.00%,96.47%,100.00%,91.43%,92.00%,12.00%,10.00%,0.00%,0.48,1.14,0.81,2.52
39
+ 38,60.29%,Meta-Llama-3-8B-Instruct (Prompt),https://llama.meta.com/llama3,Meta,Meta Llama 3 Community,63.02%,69.95%,55.09%,58.50%,45.00%,48.00%,73.50%,59.00%,64.50%,75.29%,79.00%,70.00%,74.00%,68.00%,62.50%,43.33%,0.24,0.04,N/A,N/A
40
+ 39,59.88%,Hermes-2-Pro-Mistral-7B (FC),https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B,NousResearch,apache-2.0,70.62%,55.62%,72.00%,81.75%,42.00%,54.00%,80.50%,67.00%,63.00%,56.47%,78.00%,25.71%,70.00%,56.00%,40.00%,10.83%,0.49,0.08,N/A,N/A
41
+ 40,59.24%,Claude-3-Sonnet-20240229 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,44.18%,43.32%,76.73%,86.00%,49.00%,58.00%,88.00%,6.00%,6.00%,85.29%,96.00%,70.00%,88.00%,0.00%,0.00%,81.67%,3.44,3.25,1.46,6.85
42
+ 41,54.88%,GLM-4-9b-Chat (FC),https://huggingface.co/THUDM/glm-4-9b-chat,THUDM,glm-4,38.69%,44.12%,63.27%,87.00%,0.00%,0.00%,91.50%,0.00%,0.00%,86.47%,90.00%,81.43%,90.00%,0.00%,0.00%,87.50%,N/A,0.13,N/A,N/A
43
+ 42,53.82%,Claude-3-Haiku-20240307 (FC tools-2024-04-04),https://www.anthropic.com/news/claude-3-family,Anthropic,Proprietary,45.05%,46.79%,86.18%,95.50%,60.00%,64.00%,93.50%,0.50%,0.00%,91.18%,96.00%,84.29%,94.00%,2.00%,0.00%,20.83%,0.29,1.49,0.61,2.4
44
+ 43,53.65%,FireFunction-v1 (FC),https://huggingface.co/fireworks-ai/firefunction-v1,Fireworks,Apache 2.0,40.75%,39.79%,70.00%,90.25%,13.00%,22.00%,93.00%,0.00%,0.00%,71.18%,95.00%,37.14%,88.00%,0.00%,0.00%,73.33%,N/A,1.69,1.53,4.61
45
+ 44,53.59%,GPT-4-0613 (FC),https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo,OpenAI,Proprietary,39.20%,38.53%,63.82%,86.50%,4.00%,2.00%,93.00%,0.00%,0.00%,64.12%,95.00%,20.00%,90.00%,0.00%,0.00%,91.67%,10.37,3.49,3.27,10.88
46
+ 45,52.76%,Mistral-tiny-2312 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,49.91%,36.16%,55.64%,70.00%,26.00%,0.00%,56.50%,47.50%,40.00%,27.65%,46.00%,1.43%,20.00%,62.00%,35.00%,83.75%,0.13,1.45,1.41,4.39
47
+ 46,43.82%,Gemma-7b-it (Prompt),https://blog.google/technology/developers/gemma-open-models/,Google,gemma-terms-of-use,41.30%,31.75%,42.18%,47.75%,29.00%,24.00%,48.00%,30.50%,44.50%,30.00%,44.00%,10.00%,32.00%,40.00%,25.00%,70.83%,0.37,0.06,N/A,N/A
48
+ 47,40.88%,Mistral-Small-2402 (Prompt),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,40.58%,38.03%,5.82%,6.00%,6.00%,4.00%,8.00%,79.50%,69.00%,34.12%,6.00%,74.29%,20.00%,68.00%,30.00%,98.33%,0.64,1.11,0.95,3.03
49
+ 48,40.41%,Deepseek-v1.5 (Prompt),https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5,Deepseek,Deepseek License,38.44%,30.89%,39.27%,50.00%,4.00%,24.00%,49.00%,37.00%,28.50%,37.06%,38.00%,35.71%,38.00%,36.00%,12.50%,57.08%,3.24,0.53,N/A,N/A
50
+ 49,23.82%,Mistral-small-2402 (FC Auto),https://docs.mistral.ai/guides/model-selection/,Mistral AI,Proprietary,2.88%,34.37%,2.00%,2.75%,0.00%,0.00%,2.50%,3.00%,4.00%,56.47%,79.00%,24.29%,70.00%,6.00%,5.00%,99.58%,0.97,3.06,1.8,6.23