Imatrix compressions of FP Merge of "D_AU-Mistral-7B-Instruct-v0.2-Bagel-DarkSapling-DPO-7B-v2.0".
"Imatrix Plus" is an upgraded form of Imatrix which using full precision for specific parts of the compression. As a result all compressions will be slightly larger in size than standard 7B compressions.
This method results in a higher quality model, especially at lower compressions. This method is applied across all compressions from IQ1 to Q8.
Even IQ1_S - the most compressed verison - works well, however IQ4/Q4 are suggested as minimums for quality. Highest quality will be Q6/Q8.
Q8 Imatrix Plus quality will exceed standard Q8 and Regular Imatrix Q8.
This merge was an experiment to test already established Roleplay, Fiction and Story generation of "DarkSapling" with a some of "Bagel"'s qualities with a Mistral Instruct Base.
For Imatrix plus this was a test of high precision in specific areas of the model leading to a slightly larger compressed file. In addition the Imatrix process itself used a larger "calibration" file than standard was used to further enhance quality.
The process added appoximately 250 MB to each compressed file. An additional enhancement added another 250 mb to each compressed file.
A blank or standard Alpaca Template for text generation will work.
Context length: 32768.
Please see the orginal model card for specific details of use, additional credits and tips under "Models Merged" below.
merge
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the SLERP merge method.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: MaziyarPanahi/bagel-dpo-7b-v0.1-Mistral-7B-Instruct-v0.2-slerp
layer_range: [0, 32]
- model: TeeZee/DarkSapling-7B-v2.0
layer_range: [0, 32]
merge_method: slerp
base_model: MaziyarPanahi/bagel-dpo-7b-v0.1-Mistral-7B-Instruct-v0.2-slerp
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: bfloat16
- Downloads last month
- 30