appvoid commited on
Commit
3c17d2f
·
verified ·
1 Parent(s): 2206f39

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -4
README.md CHANGED
@@ -22,7 +22,7 @@ license: apache-2.0
22
 
23
  This is a passthrough of arco with an experimental model. It improved on arc challenge, only missing 1.2 points to get to the level of modern 3b baseline performance.
24
 
25
- If you prefer answering multilingual, general knowledge, trivially simple questions chose qwen. If you prefer solving trivially simple english tasks, chose arco.
26
 
27
  #### prompt
28
 
@@ -31,12 +31,15 @@ there is no prompt intentionally set.
31
 
32
  #### benchmarks
33
 
 
 
34
  | Parameters | Model | MMLU | ARC-C | HellaSwag | PIQA | Winogrande | Average |
35
  | -----------|--------------------------------|-------|-------|-----------|--------|------------|---------|
36
- | 0.5b | qwen2 |44.13| 28.92| 49.05 | 69.31 | 56.99 | 49.68 |
37
- | 0.5b | qwen2.5 |**47.29**|31.83|52.17|70.29|57.06|51.72|
38
  | 0.5b | arco |26.17|37.29|62.88|74.37|**62.27**|52.60|
39
- | 0.5b | arco 2 |25.51|**38.82**|**63.02**|**74.70**|61.25|**52.66**|
 
40
  #### supporters
41
 
42
  <a href="https://ko-fi.com/appvoid" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 34px !important; margin-top: -4px;width: 128px !important; filter: contrast(2) grayscale(100%) brightness(100%);" ></a>
 
22
 
23
  This is a passthrough of arco with an experimental model. It improved on arc challenge, only missing 1.2 points to get to the level of modern 3b baseline performance.
24
 
25
+ If you prefer answering multilingual, general knowledge, trivially simple questions chose qwen or llama. If you prefer solving trivially simple english tasks while being half the size, chose arco.
26
 
27
  #### prompt
28
 
 
31
 
32
  #### benchmarks
33
 
34
+ zero-shot results from state-of-the-art small language models
35
+
36
  | Parameters | Model | MMLU | ARC-C | HellaSwag | PIQA | Winogrande | Average |
37
  | -----------|--------------------------------|-------|-------|-----------|--------|------------|---------|
38
+ | 0.5b | qwen 2 |44.13| 28.92| 49.05 | 69.31 | 56.99 | 49.68 |
39
+ | 0.5b | qwen 2.5 |**47.29**|31.83|52.17|70.29|57.06|51.72|
40
  | 0.5b | arco |26.17|37.29|62.88|74.37|**62.27**|52.60|
41
+ | 0.5b | arco 2 |25.51|**38.82**|63.02|**74.70**|61.25|**52.66**|
42
+ | 1.24b | llama 3.2 |36.75|36.18|**63.70**|74.54 |60.54|**54.34**|
43
  #### supporters
44
 
45
  <a href="https://ko-fi.com/appvoid" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 34px !important; margin-top: -4px;width: 128px !important; filter: contrast(2) grayscale(100%) brightness(100%);" ></a>