Update README.md

README.md

@@ -1,61 +1,53 @@
+
+
 ---
-base_model:
-- h2oai/h2o-danube3-500m-chat
-- appvoid/massive
-library_name: transformers
+language:
+- en
+license: apache-2.0
 tags:
-- mergekit
-- merge
-
+- text-generation-inference
+- transformers
+- unsloth
+- llama
+- trl
+- sft
 ---
-# mix-2
-
-This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
-
-## Merge Details
-### Merge Method
-
-This model was merged using the SLERP merge method.
-
-### Models Merged
-
-The following models were included in the merge:
-* [h2oai/h2o-danube3-500m-chat](https://huggingface.co/h2oai/h2o-danube3-500m-chat)
-* [appvoid/massive](https://huggingface.co/appvoid/massive)
-
-### Configuration
-
-The following YAML configuration was used to produce this model:
-
-```yaml
-slices:
-- sources:
-  - model: appvoid/massive
-    layer_range:
-    - 0
-    - 16
-  - model: h2oai/h2o-danube3-500m-chat
-    layer_range:
-    - 0
-    - 16
-merge_method: slerp
-base_model: appvoid/massive
-parameters:
-  t:
-  - filter: self_attn
-    value:
-    - 0
-    - 0.5
-    - 0.3
-    - 0.7
-    - 1
-  - filter: mlp
-    value:
-    - 1
-    - 0.5
-    - 0.7
-    - 0.3
-    - 0
-  - value: 0.5
-dtype: float16
-```
+
+<style>
+@import url('https://fonts.googleapis.com/css2?family=Vollkorn:ital,wght@0,400..900;1,400..900&display=swap');
+</style>
+
+<div style="background-color: #101010; border-radius: .5rem; padding: 2rem; font-family: monospace; font-size: .85rem; text-align: justify;">
+  
+![palmer-004](https://huggingface.co/appvoid/palmer-004-original/resolve/main/palmer-004.jpeg)
+
+  
+#### palmer turbo
+
+This model has a slightly different architecture and training style:
+
+1. The model was followed by a continual pretraining (lm_head + embedding layers were tuned).
+2. Base model was trained on 15k instruction/response pairs.
+3. Similar architecture than palmer series but smaller in context size (8192)
+
+In short, palmer is now half the size, twice the speed and same overall performance with a dramatical boost on arc challenge instead of winogrande.
+
+As all palmer models, the model is biased to respond to answers without using any specific prompt, feel free to further fine-tune it for your specific use case.
+
+| Model                          | MMLU  | ARC-C | HellaSwag | PIQA   | Winogrande | Average |
+|--------------------------------|-------|-------|-----------|--------|------------|---------|
+| tinyllama                      | 0.2577| 0.3029| 0.5935    | 0.7329 | 0.5959     | 0.4966  |
+| palmer-004-turbo               |**0.2736**|**0.3558**| 0.6031    |0.7367|**0.6117**|0.5191|
+| palmer-004                     | 0.2661  |0.3490 | 0.6173  | **0.7481**  | **0.6417**  | 0.5244|
+
+
+#### thanks to
+
+- h2oai: performant base model provider
+- teknium: openhermes dataset provider
+- unsloth: tooling for training software
+
+#### note
+
+Next versions of this model will be available through my upcoming app. Keep tuned to not miss the release date on my X account.
+</div>