Create README.md

README.md

@@ -0,0 +1,84 @@
+---
+library_name: transformers
+tags:
+- not-for-all-audiences
+- mergekit
+datasets:
+- crestf411/LimaRP-DS
+- Gryphe/Sonnet3.5-Charcard-Roleplay
+- anthracite-org/c2_logs_32k_mistral-v3_v1.2_no_system
+- anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system
+- anthracite-org/kalo-opus-instruct-3k-filtered-no-system
+- anthracite-org/nopm_claude_writing_fixed
+base_model:
+- Qwen/Qwen2.5-32B-Instruct
+---
+
+![slush.jpg](https://huggingface.co/crestf411/L3.1-8B-Slush/resolve/main/slush.jpg?)
+
+([GGUFs](https://huggingface.co/mradermacher/MN-Slush-i1-GGUF))
+
+**Slush** is a two-stage model trained with high LoRA dropout, where stage 1 is a pretraining continuation on the base model, aimed at boosting the model's creativity and writing capabilities. This is then merged into the instruction tune model, and stage 2 is a fine tuning step on top of this to further enhance its roleplaying capabilities and/or to repair any damage caused in the stage 1 merge.
+
+This is still early stage. As always, feedback is welcome, and begone if you demand perfection.
+
+The second stage, like the *Sunfall* series, follows the Silly Tavern preset (ChatML), so ymmv in particular if you use some other tool and/or preset.
+
+**Parameter suggestions:**
+
+I did all my testing with temp 1, min-p 0.1, DRY 0.8, but enabled XTC as context grew and/or the model started saying "the same stuff".
+
+**Training details:**
+
+* Stage 1 (continued pretraining)
+  *  Target: Qwen/Qwen2.5-32B (resulting LoRA merged into Qwen/Qwen2.5-32B-Instruct)
+  *  LoRA dropout 0.5 ([motivation](https://arxiv.org/abs/2403.00946))
+  *  LoRA rank 32, alpha 64 ([motivation](https://arxiv.org/abs/2410.21228))
+  *  LR cosine 4e-6
+  *  [LoRA+](https://arxiv.org/abs/2402.12354) with LR Ratio: 15
+  *  Context size: 8192
+  *  Gradient accumulation steps: 4
+  *  Epochs: 1
+* Stage 2 (fine tune)
+  *  Target: Stage 1 model
+  *  LoRA dropout 0.5
+  *  LoRA rank 32, alpha 64
+  *  LR cosine 5e-6 (min 5e-7)
+  *  [LoRA+](https://arxiv.org/abs/2402.12354) with LR Ratio: 15
+  *  Context size: 16384
+  *  Gradient accumulation steps: 4
+  *  Epochs: 1
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [TIES](https://arxiv.org/abs/2306.01708) merge method.
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+models:
+  - model: stage1-model
+    parameters:
+      weight: 1
+      density: 1
+  - model: stage2-model
+    parameters:
+      weight: 1
+      density: 1
+  - model: Qwen/Qwen2.5-32B-Instruct
+    parameters:
+      weight: 0.9
+      density: 0.9
+merge_method: ties
+base_model: Qwen/Qwen2.5-32B
+parameters:
+  weight: 0.9
+  density: 0.9
+  normalize: true
+  int8_mask: true
+tokenizer_source: Qwen/Qwen2.5-32B-Instruct
+dtype: bfloat16
+```