Hugging Face's logo

DavidAU
/
D_AU-Orac-13B-Tiefighter-slerp

Text Generation
Transformers
Safetensors
llama
mergekit
Merge
text-generation-inference
Inference Endpoints
Model card Files Community
D_AU-Orac-13B-Tiefighter-slerp
File size: 1,943 Bytes 
f589e01
 
 
 
 
 
 
 
 
 
d6de84b
c78d770
bdb1e7c
 
 
 
 
 
e69657a
bdb1e7c
 
 
 
e69657a
bdb1e7c
 
e69657a
 
 
9405d1a
 
 
 
 
f589e01
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
---
base_model:
- microsoft/Orca-2-13b
- KoboldAI/LLaMA2-13B-Tiefighter
library_name: transformers
tags:
- mergekit
- merge

---

28/04/2024- UPDATE: 

Fixed tokenizer / vocab issues. 
Verified operation, conversion to GGUF now works too.
GGUF uploaded, with Imatrix Plus GGUFs to follow shortly.

Imatrix Plus GGUFs are [here](https://huggingface.co/DavidAU/D_AU-Orac-13B-Tiefighter-slerp-imat-plus-GGUF)

This includes all Imatrix compressions as well as regular "Qs" which have also been "Imatrixed" too.

"Imatrix Plus" is an upgraded form of Imatrix which using full precision for specific parts of the compression.

This results in a higher quality model, especially at lower compressions.
This method is applied across all compressions from IQ1 to Q8.

This merge was an experiment to test already established Roleplay, Fiction and Story 
generation of "Tiefighter" with a some of "Orca 2"'s qualities.

A blank or standard Alpaca Template for text generation will work.
Currently "CHATML" is untested.

Context length: 4096.

# merge

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

## Merge Details
### Merge Method

This model was merged using the SLERP merge method.

### Models Merged

The following models were included in the merge:
* [microsoft/Orca-2-13b](https://huggingface.co/microsoft/Orca-2-13b)
* [KoboldAI/LLaMA2-13B-Tiefighter](https://huggingface.co/KoboldAI/LLaMA2-13B-Tiefighter)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
slices:
  - sources:
      - model: KoboldAI/LLaMA2-13B-Tiefighter
        layer_range: [0, 40]
      - model: microsoft/Orca-2-13b
        layer_range: [0, 40]
merge_method: slerp
base_model: microsoft/Orca-2-13b
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

```