jpacifico commited on
Commit
f12fd52
1 Parent(s): c403df6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +62 -9
README.md CHANGED
@@ -19,24 +19,77 @@ using the [jpacifico/french-orca-dpo-pairs-revised](https://huggingface.co/datas
19
  Chocolatine is a general model and can itself be finetuned to be specialized for specific use cases.
20
  Window context = 4k tokens
21
 
22
- ![image/jpeg](https://github.com/jpacifico/Chocolatine-LLM/blob/main/Assets/chocolatine_visuel_500x500.png?raw=true)
23
-
24
- ### Evaluation
25
 
26
  Submitted on [OpenLLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard), results in few days !
 
27
 
28
- ### Evaluation MT-Bench in French
29
 
30
  Chocolatine-3B-Instruct-DPO-Revised is outperforming GPT-3.5-Turbo on [MT-Bench-French](https://huggingface.co/datasets/bofenghuang/mt-bench-french) by Bofeng Huang,
31
- used with [multilingual-mt-bench](https://github.com/Peter-Devine/multilingual_mt_bench)
32
 
33
- ![image/jpeg](https://github.com/jpacifico/Chocolatine-LLM/blob/main/Assets/results_fr_mt_bench_400x900.png?raw=false)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
  ### Usage
36
 
37
  You can run this model using my [Colab notebook](https://github.com/jpacifico/Chocolatine-LLM/blob/main/Chocolatine_3B_inference_test_colab.ipynb)
38
 
39
- You can also run this model using the following code:
40
 
41
  ```python
42
  import transformers
@@ -71,8 +124,8 @@ print(sequences[0]['generated_text'])
71
 
72
  ### Limitations
73
 
74
- Chocolatine is a quick demonstration that a base 3B model can be easily fine-tuned to specialize in a particular language.
75
- It does not have any moderation mechanisms.
76
 
77
  - **Developed by:** Jonathan Pacifico, 2024
78
  - **Model type:** LLM
 
19
  Chocolatine is a general model and can itself be finetuned to be specialized for specific use cases.
20
  Window context = 4k tokens
21
 
22
+ ### Benchmarks
 
 
23
 
24
  Submitted on [OpenLLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard), results in few days !
25
+ First version Chocolatine-3B-Instruct-DPO-v1.0 is already one of the best-performing 3B models on the Open LLM Leaderboard
26
 
27
+ ### MT-Bench-French
28
 
29
  Chocolatine-3B-Instruct-DPO-Revised is outperforming GPT-3.5-Turbo on [MT-Bench-French](https://huggingface.co/datasets/bofenghuang/mt-bench-french) by Bofeng Huang,
30
+ used with [multilingual-mt-bench](https://github.com/Peter-Devine/multilingual_mt_bench)
31
 
32
+ ```
33
+ ########## First turn ##########
34
+ score
35
+ model turn
36
+ gpt-3.5-turbo 1 8.1375
37
+ Chocolatine-3B-Instruct-DPO-Revised 1 7.9875
38
+ Daredevil-8B 1 7.8875
39
+ Daredevil-8B-abliterated 1 7.8375
40
+ Chocolatine-3B-Instruct-DPO-v1.0 1 7.6875
41
+ NeuralDaredevil-8B-abliterated 1 7.6250
42
+ Phi-3-mini-4k-instruct 1 7.2125
43
+ Meta-Llama-3-8B-Instruct 1 7.1625
44
+ vigostral-7b-chat 1 6.7875
45
+ Mistral-7B-Instruct-v0.3 1 6.7500
46
+ Mistral-7B-Instruct-v0.2 1 6.2875
47
+ French-Alpaca-7B-Instruct_beta 1 5.6875
48
+ vigogne-2-7b-chat 1 5.6625
49
+ vigogne-2-7b-instruct 1 5.1375
50
+
51
+ ########## Second turn ##########
52
+ score
53
+ model turn
54
+ Chocolatine-3B-Instruct-DPO-Revised 2 7.937500
55
+ gpt-3.5-turbo 2 7.679167
56
+ Chocolatine-3B-Instruct-DPO-v1.0 2 7.612500
57
+ NeuralDaredevil-8B-abliterated 2 7.125000
58
+ Daredevil-8B 2 7.087500
59
+ Daredevil-8B-abliterated 2 6.873418
60
+ Meta-Llama-3-8B-Instruct 2 6.800000
61
+ Mistral-7B-Instruct-v0.2 2 6.512500
62
+ Mistral-7B-Instruct-v0.3 2 6.500000
63
+ Phi-3-mini-4k-instruct 2 6.487500
64
+ vigostral-7b-chat 2 6.162500
65
+ French-Alpaca-7B-Instruct_beta 2 5.487395
66
+ vigogne-2-7b-chat 2 2.775000
67
+ vigogne-2-7b-instruct 2 2.240506
68
+
69
+ ########## Average ##########
70
+ score
71
+ model
72
+ Chocolatine-3B-Instruct-DPO-Revised 7.962500
73
+ gpt-3.5-turbo 7.908333
74
+ Chocolatine-3B-Instruct-DPO-v1.0 7.650000
75
+ Daredevil-8B 7.487500
76
+ NeuralDaredevil-8B-abliterated 7.375000
77
+ Daredevil-8B-abliterated 7.358491
78
+ Meta-Llama-3-8B-Instruct 6.981250
79
+ Phi-3-mini-4k-instruct 6.850000
80
+ Mistral-7B-Instruct-v0.3 6.625000
81
+ vigostral-7b-chat 6.475000
82
+ Mistral-7B-Instruct-v0.2 6.400000
83
+ French-Alpaca-7B-Instruct_beta 5.587866
84
+ vigogne-2-7b-chat 4.218750
85
+ vigogne-2-7b-instruct 3.698113
86
+ ```
87
 
88
  ### Usage
89
 
90
  You can run this model using my [Colab notebook](https://github.com/jpacifico/Chocolatine-LLM/blob/main/Chocolatine_3B_inference_test_colab.ipynb)
91
 
92
+ You can also run Chocolatine using the following code:
93
 
94
  ```python
95
  import transformers
 
124
 
125
  ### Limitations
126
 
127
+ The Chocolatine model is a quick demonstration that a base model can be easily fine-tuned to achieve compelling performance.
128
+ It does not have any moderation mechanism.
129
 
130
  - **Developed by:** Jonathan Pacifico, 2024
131
  - **Model type:** LLM