Steelskull commited on
Commit
122e8fe
1 Parent(s): 658a4dc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +128 -19
README.md CHANGED
@@ -6,39 +6,148 @@ base_model:
6
  - abacusai/Llama-3-Giraffe-70B
7
  library_name: transformers
8
  tags:
9
- - mergekit
10
  - merge
11
-
12
  ---
13
- # merge
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
- This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
 
 
16
 
17
- ## Merge Details
18
- ### Merge Method
 
 
19
 
20
- This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [failspy/llama-3-70B-Instruct-abliterated](https://huggingface.co/failspy/llama-3-70B-Instruct-abliterated) as a base.
 
 
 
 
21
 
22
- ### Models Merged
 
 
 
 
23
 
24
- The following models were included in the merge:
25
- * [migtissera/Tess-2.0-Llama-3-70B-v0.2](https://huggingface.co/migtissera/Tess-2.0-Llama-3-70B-v0.2)
26
- * [NeverSleep/Llama-3-Lumimaid-70B-v0.1-alt](https://huggingface.co/NeverSleep/Llama-3-Lumimaid-70B-v0.1-alt)
27
- * [abacusai/Llama-3-Giraffe-70B](https://huggingface.co/abacusai/Llama-3-Giraffe-70B)
28
 
29
- ### Configuration
 
 
30
 
31
- The following YAML configuration was used to produce this model:
 
 
 
 
 
 
 
 
32
 
33
- ```yaml
 
 
34
 
35
- base_model: failspy/llama-3-70B-Instruct-abliterated
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  merge_method: model_stock
37
  dtype: bfloat16
38
-
39
  models:
40
  - model: migtissera/Tess-2.0-Llama-3-70B-v0.2
41
  - model: abacusai/Llama-3-Giraffe-70B
42
  - model: NeverSleep/Llama-3-Lumimaid-70B-v0.1-alt
43
-
44
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  - abacusai/Llama-3-Giraffe-70B
7
  library_name: transformers
8
  tags:
 
9
  - merge
10
+ license: apache-2.0
11
  ---
12
+ <!DOCTYPE html>
13
+ <style>
14
+
15
+ body {
16
+ font-family: 'Quicksand', sans-serif;
17
+ background: linear-gradient(135deg, #2E3440 0%, #1A202C 100%);
18
+ color: #D8DEE9;
19
+ margin: 0;
20
+ padding: 0;
21
+ font-size: 16px;
22
+ }
23
+
24
+ .container {
25
+ width: 80% auto;
26
+ max-width: 1080px auto;
27
+ margin: 20px auto;
28
+ background-color: rgba(255, 255, 255, 0.02);
29
+ padding: 20px;
30
+ border-radius: 12px;
31
+ box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2);
32
+ backdrop-filter: blur(10px);
33
+ border: 1px solid rgba(255, 255, 255, 0.1);
34
+ }
35
+
36
+ .header h1 {
37
+ font-size: 28px;
38
+ color: #ECEFF4;
39
+ margin: 0 0 20px 0;
40
+ text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.3);
41
+ }
42
 
43
+ .update-section {
44
+ margin-top: 30px;
45
+ }
46
 
47
+ .update-section h2 {
48
+ font-size: 24px;
49
+ color: #88C0D0;
50
+ }
51
 
52
+ .update-section p {
53
+ font-size: 16px;
54
+ line-height: 1.6;
55
+ color: #ECEFF4;
56
+ }
57
 
58
+ .info img {
59
+ width: 100%;
60
+ border-radius: 10px;
61
+ margin-bottom: 15px;
62
+ }
63
 
64
+ a {
65
+ color: #88C0D0;
66
+ text-decoration: none;
67
+ }
68
 
69
+ a:hover {
70
+ color: #A3BE8C;
71
+ }
72
 
73
+ .button {
74
+ display: inline-block;
75
+ background-color: #5E81AC;
76
+ color: #E5E9F0;
77
+ padding: 10px 20px;
78
+ border-radius: 5px;
79
+ cursor: pointer;
80
+ text-decoration: none;
81
+ }
82
 
83
+ .button:hover {
84
+ background-color: #81A1C1;
85
+ }
86
 
87
+ pre {
88
+ background-color: #2E3440;
89
+ padding: 10px;
90
+ border-radius: 5px;
91
+ overflow-x: auto;
92
+ }
93
+
94
+ code {
95
+ font-family: 'Courier New', monospace;
96
+ color: #D8DEE9;
97
+ }
98
+
99
+ </style>
100
+ <html lang="en">
101
+ <head>
102
+ <meta charset="UTF-8">
103
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
104
+ <title>L3-MS-Astoria-70b Data Card</title>
105
+ <link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@400;500;600&display=swap" rel="stylesheet">
106
+ </head>
107
+ <body>
108
+ <div class="container">
109
+ <div class="header">
110
+ <h1>L3-MS-Astoria-70b</h1>
111
+ </div>
112
+ <div class="info">
113
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/nkW2r4nd9zJUlrjCxsE8A.png">
114
+ <p>Now that the cute anime girl has your attention.</p>
115
+ <p><strong>Creator:</strong> <a href="https://huggingface.co/Steelskull" target="_blank">SteelSkull</a></p>
116
+ <h1>About L3-MS-Astoria-70b:</h1>
117
+ <p>L3 = Llama-3 <p/>
118
+ <p>MS = Model Stock <p/>
119
+ <p>This is my first foray into 70b models, so this is more or less an experiment, please let me know your thoughts on the model and where their can be improvements.<br>
120
+ L3-MS-Astoria-70b combines the strengths of multiple models to deliver a well-rounded, capable assistant. It is aimed at performing general tasks, storytelling, roleplay, and more mature content.<br>
121
+ The model stock merging method attempts to make the model remain focused, tailored, and high-quality.
122
+ <li>Integrates models from notable sources for enhanced performance in diverse tasks.</p>
123
+ <h2>Config:</h2>
124
+ <pre><code>MODEL_NAME = "L3-MS-Astoria-70b"
125
+ yaml_config = """
126
+ base_model: failspy/llama-3-70B-Instruct-abliterated
127
  merge_method: model_stock
128
  dtype: bfloat16
 
129
  models:
130
  - model: migtissera/Tess-2.0-Llama-3-70B-v0.2
131
  - model: abacusai/Llama-3-Giraffe-70B
132
  - model: NeverSleep/Llama-3-Lumimaid-70B-v0.1-alt
133
+ """
134
+ </code></pre>
135
+ <h3>Source Model Details:</h3>
136
+ <p><strong>migtissera/Tess-2.0-Llama-3-70B-v0.2:</strong><br>
137
+ Tess, short for Tesoro (Treasure in Italian), is a general purpose Large Language Model series. Tess-2.0-Llama-3-70B-v0.2 was trained on the meta-llama/Meta-Llama-3-70B base. The change between v0.1 and this version, v0.2 is that v0.2 has undergone an additional step of uncensoring.
138
+ </p>
139
+ <p><strong>abacusai/Llama-3-Giraffe-70B:</strong><br>
140
+ General model trained on 1b tokens, up to 128k ctx
141
+ </p>
142
+ <p><strong>NeverSleep/Llama-3-Lumimaid-70B-v0.1-alt:</strong><br>
143
+ Llama3 trained on our RP datasets, NeverSleep tried to have a balance between the ERP and the RP, not too horny, but just enough.<br>
144
+ NeverSleep also added some non-RP dataset, making the model less dumb overall. It should look like a 40%/60% ratio for Non-RP/RP+ERP data.
145
+ </p>
146
+ <p><strong>Base model failspy/llama-3-70B-Instruct-abliterated:</strong><br>
147
+ This is meta-llama/Llama-3-70B-Instruct with orthogonalized bfloat16 safetensor weights, generated with the methodology that was described in the preview paper/blog post: 'Refusal in LLMs is mediated by a single direction' which I encourage you to read to understand more.<br>
148
+ TL;DR: this model has had certain weights manipulated to "inhibit" the model's ability to express refusal. It is not in anyway _guaranteed_ that it won't refuse you, understand your request, it may still lecture you about ethics/safety, etc. It is tuned in all other respects the same as the original 70B instruct model was, just with the strongest refusal direction orthogonalized out.
149
+ </p>
150
+ </div>
151
+ </div>
152
+ </body>
153
+ </html>