Jan commited on
Commit
662c68e
1 Parent(s): 95db90d

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +71 -0
README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ ---
6
+ <!-- header start -->
7
+ <!-- 200823 -->
8
+ <div style="width: auto; margin-left: auto; margin-right: auto">
9
+ <img src="https://github.com/janhq/jan/assets/89722390/35daac7d-b895-487c-a6ac-6663daaad78e" alt="Jan banner" style="width: 100%; min-width: 400px; display: block; margin: auto;">
10
+ </div>
11
+
12
+ <p align="center">
13
+ <a href="https://jan.ai/">Jan</a>
14
+ - <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
15
+ </p>
16
+ <!-- header end -->
17
+
18
+ # Model Description
19
+ This model uses the `Slerp` merge method from 2 models:
20
+ 1. [Seraph-7B](https://huggingface.co/Weyaxi/Seraph-7B)
21
+ 2. [Marcoroni-7B-v3](https://huggingface.co/AIDC-ai-business/Marcoroni-7B-v3)
22
+
23
+ - base model: [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
24
+
25
+ The yaml config file for this model is here:
26
+
27
+ ```yaml
28
+ slices:
29
+ - sources:
30
+ - model: Weyaxi/Seraph-7B
31
+ layer_range: [0, 32]
32
+ - model: AIDC-ai-business/Marcoroni-7B-v3
33
+ layer_range: [0, 32]
34
+ merge_method: slerp
35
+ base_model: mistralai/Mistral-7B-v0.1
36
+ parameters:
37
+ t:
38
+ - filter: self_attn
39
+ value: [0, 0.5, 0.3, 0.7, 1]
40
+ - filter: mlp
41
+ value: [1, 0.5, 0.7, 0.3, 0]
42
+ - value: 0.5
43
+ dtype: bfloat16
44
+ ```
45
+
46
+ # About Jan
47
+ Jan believes in the need for an open-source AI ecosystem and is building the infra and tooling to allow open-source AIs to compete on a level playing field with proprietary ones.
48
+
49
+ Jan's long-term vision is to build a cognitive framework for future robots, who are practical, useful assistants for humans and businesses in everyday life.
50
+
51
+ # Jan Model Merger
52
+ This is a test project for merging models.
53
+
54
+ # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
55
+
56
+ Detailed results can be found here.
57
+
58
+ | Metric | Value |
59
+ |-----------------------|---------------------------|
60
+ | Avg. | ?|
61
+ | ARC (25-shot) | ? |
62
+ | HellaSwag (10-shot) | ? |
63
+ | MMLU (5-shot) | ?|
64
+ | TruthfulQA (0-shot) | ? |
65
+ | Winogrande (5-shot) | ? |
66
+ | GSM8K (5-shot) | ? |
67
+
68
+ # Acknowlegement
69
+ - [mergekit](https://github.com/cg123/mergekit)
70
+ - [DARE](https://github.com/yule-BUAA/MergeLM/blob/main/README.md)
71
+ - [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)