Update README.md

README.md

@@ -1,7 +1,6 @@
 ---
 language:
 - en
-base_model: pszemraj/MiniLMv2-L6-H384_R-simplewiki
 tags:
 - generated_from_trainer
 metrics:
@@ -11,28 +10,23 @@ datasets:
 - BEE-spoke-data/fineweb-100k_en-med
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
 
-# MiniLMv2-L6-H384_R-simplewiki-fineweb-100k_en-med_512-vN
+# MiniLMv2-L6-H384_R-fineweb-100k
 
-This model is a fine-tuned version of [pszemraj/MiniLMv2-L6-H384_R-simplewiki](https://huggingface.co/pszemraj/MiniLMv2-L6-H384_R-simplewiki) on the BEE-spoke-data/fineweb-100k_en-med dataset.
-It achieves the following results on the evaluation set:
-- Loss: 4.0206
-- Accuracy: 0.3783
-- Num Input Tokens Seen: 162790400
-
-## Model description
+This is a MiniLMv2 model pretrained further on an MLM task with the goal improving downstream finetuning/performance:
 
-More information needed
+- activation updated to SiLU prior to further training
+- MLM @ 40% mask ratio
 
-## Intended uses & limitations
-
-More information needed
+- 
+## Model description
 
-## Training and evaluation data
+This model is a fine-tuned version of [nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large](https://huggingface.co/nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large) on the BEE-spoke-data/fineweb-100k_en-med dataset.
 
-More information needed
+It achieves the following results on the evaluation set:
+- Loss: 4.0206
+- Accuracy: 0.3783
+- Num Input Tokens Seen: 162790400
 
 ## Training procedure