Hugging Face's logo

legraphista
/
DeepSeek-V2.5-IMat-GGUF

Text Generation
GGUF
quantized
GGUF
quantization
imat
imatrix
static
8bit
6bit
5bit
4bit
3bit
2bit
1bit
conversational
Model card Files Community
1
legraphista commited on
Commit
bd514b7
•
1 Parent(s): 422f02b

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +174 -0
imatrix.log ADDED
@@ -0,0 +1,174 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ llama_model_loader: loaded meta data with 46 key-value pairs and 959 tensors from DeepSeek-V2.5-IMat-GGUF/DeepSeek-V2.5.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
2
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
3
+ llama_model_loader: - kv 0: general.architecture str = deepseek2
4
+ llama_model_loader: - kv 1: general.type str = model
5
+ llama_model_loader: - kv 2: general.name str = DeepSeek V2.5
6
+ llama_model_loader: - kv 3: general.version str = V2.5
7
+ llama_model_loader: - kv 4: general.basename str = DeepSeek
8
+ llama_model_loader: - kv 5: general.size_label str = 160x14B
9
+ llama_model_loader: - kv 6: general.license str = other
10
+ llama_model_loader: - kv 7: general.license.name str = deepseek
11
+ llama_model_loader: - kv 8: general.license.link str = https://github.com/deepseek-ai/DeepSe...
12
+ llama_model_loader: - kv 9: deepseek2.block_count u32 = 60
13
+ llama_model_loader: - kv 10: deepseek2.context_length u32 = 163840
14
+ llama_model_loader: - kv 11: deepseek2.embedding_length u32 = 5120
15
+ llama_model_loader: - kv 12: deepseek2.feed_forward_length u32 = 12288
16
+ llama_model_loader: - kv 13: deepseek2.attention.head_count u32 = 128
17
+ llama_model_loader: - kv 14: deepseek2.attention.head_count_kv u32 = 128
18
+ llama_model_loader: - kv 15: deepseek2.rope.freq_base f32 = 10000.000000
19
+ llama_model_loader: - kv 16: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
20
+ llama_model_loader: - kv 17: deepseek2.expert_used_count u32 = 6
21
+ llama_model_loader: - kv 18: general.file_type u32 = 7
22
+ llama_model_loader: - kv 19: deepseek2.leading_dense_block_count u32 = 1
23
+ llama_model_loader: - kv 20: deepseek2.vocab_size u32 = 102400
24
+ llama_model_loader: - kv 21: deepseek2.attention.q_lora_rank u32 = 1536
25
+ llama_model_loader: - kv 22: deepseek2.attention.kv_lora_rank u32 = 512
26
+ llama_model_loader: - kv 23: deepseek2.attention.key_length u32 = 192
27
+ llama_model_loader: - kv 24: deepseek2.attention.value_length u32 = 128
28
+ llama_model_loader: - kv 25: deepseek2.expert_feed_forward_length u32 = 1536
29
+ llama_model_loader: - kv 26: deepseek2.expert_count u32 = 160
30
+ llama_model_loader: - kv 27: deepseek2.expert_shared_count u32 = 2
31
+ llama_model_loader: - kv 28: deepseek2.expert_weights_scale f32 = 16.000000
32
+ llama_model_loader: - kv 29: deepseek2.rope.dimension_count u32 = 64
33
+ llama_model_loader: - kv 30: deepseek2.rope.scaling.type str = yarn
34
+ llama_model_loader: - kv 31: deepseek2.rope.scaling.factor f32 = 40.000000
35
+ llama_model_loader: - kv 32: deepseek2.rope.scaling.original_context_length u32 = 4096
36
+ llama_model_loader: - kv 33: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
37
+ llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2
38
+ llama_model_loader: - kv 35: tokenizer.ggml.pre str = deepseek-llm
39
+ llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,102400] = ["!", "\"", "#", "$", "%", "&", "'", ...
40
+ llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,102400] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
41
+ llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,99757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
42
+ llama_model_loader: - kv 39: tokenizer.ggml.bos_token_id u32 = 100000
43
+ llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 100001
44
+ llama_model_loader: - kv 41: tokenizer.ggml.padding_token_id u32 = 100001
45
+ llama_model_loader: - kv 42: tokenizer.ggml.add_bos_token bool = true
46
+ llama_model_loader: - kv 43: tokenizer.ggml.add_eos_token bool = false
47
+ llama_model_loader: - kv 44: tokenizer.chat_template str = {% if not add_generation_prompt is de...
48
+ llama_model_loader: - kv 45: general.quantization_version u32 = 2
49
+ llama_model_loader: - type f32: 300 tensors
50
+ llama_model_loader: - type q8_0: 659 tensors
51
+ llm_load_vocab: special tokens cache size = 18
52
+ llm_load_vocab: token to piece cache size = 0.6411 MB
53
+ llm_load_print_meta: format = GGUF V3 (latest)
54
+ llm_load_print_meta: arch = deepseek2
55
+ llm_load_print_meta: vocab type = BPE
56
+ llm_load_print_meta: n_vocab = 102400
57
+ llm_load_print_meta: n_merges = 99757
58
+ llm_load_print_meta: vocab_only = 0
59
+ llm_load_print_meta: n_ctx_train = 163840
60
+ llm_load_print_meta: n_embd = 5120
61
+ llm_load_print_meta: n_layer = 60
62
+ llm_load_print_meta: n_head = 128
63
+ llm_load_print_meta: n_head_kv = 128
64
+ llm_load_print_meta: n_rot = 64
65
+ llm_load_print_meta: n_swa = 0
66
+ llm_load_print_meta: n_embd_head_k = 192
67
+ llm_load_print_meta: n_embd_head_v = 128
68
+ llm_load_print_meta: n_gqa = 1
69
+ llm_load_print_meta: n_embd_k_gqa = 24576
70
+ llm_load_print_meta: n_embd_v_gqa = 16384
71
+ llm_load_print_meta: f_norm_eps = 0.0e+00
72
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-06
73
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
74
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
75
+ llm_load_print_meta: f_logit_scale = 0.0e+00
76
+ llm_load_print_meta: n_ff = 12288
77
+ llm_load_print_meta: n_expert = 160
78
+ llm_load_print_meta: n_expert_used = 6
79
+ llm_load_print_meta: causal attn = 1
80
+ llm_load_print_meta: pooling type = 0
81
+ llm_load_print_meta: rope type = 0
82
+ llm_load_print_meta: rope scaling = yarn
83
+ llm_load_print_meta: freq_base_train = 10000.0
84
+ llm_load_print_meta: freq_scale_train = 0.025
85
+ llm_load_print_meta: n_ctx_orig_yarn = 4096
86
+ llm_load_print_meta: rope_finetuned = unknown
87
+ llm_load_print_meta: ssm_d_conv = 0
88
+ llm_load_print_meta: ssm_d_inner = 0
89
+ llm_load_print_meta: ssm_d_state = 0
90
+ llm_load_print_meta: ssm_dt_rank = 0
91
+ llm_load_print_meta: ssm_dt_b_c_rms = 0
92
+ llm_load_print_meta: model type = 236B
93
+ llm_load_print_meta: model ftype = Q8_0
94
+ llm_load_print_meta: model params = 235.74 B
95
+ llm_load_print_meta: model size = 233.41 GiB (8.50 BPW)
96
+ llm_load_print_meta: general.name = DeepSeek V2.5
97
+ llm_load_print_meta: BOS token = 100000 '<｜begin▁of▁sentence｜>'
98
+ llm_load_print_meta: EOS token = 100001 '<｜end▁of▁sentence｜>'
99
+ llm_load_print_meta: PAD token = 100001 '<｜end▁of▁sentence｜>'
100
+ llm_load_print_meta: LF token = 126 'Ä'
101
+ llm_load_print_meta: max token length = 256
102
+ llm_load_print_meta: n_layer_dense_lead = 1
103
+ llm_load_print_meta: n_lora_q = 1536
104
+ llm_load_print_meta: n_lora_kv = 512
105
+ llm_load_print_meta: n_ff_exp = 1536
106
+ llm_load_print_meta: n_expert_shared = 2
107
+ llm_load_print_meta: expert_weights_scale = 16.0
108
+ llm_load_print_meta: rope_yarn_log_mul = 0.1000
109
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
110
+ ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
111
+ ggml_cuda_init: found 1 CUDA devices:
112
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
113
+ llm_load_tensors: ggml ctx size = 0.80 MiB
114
+ llm_load_tensors: offloading 5 repeating layers to GPU
115
+ llm_load_tensors: offloaded 5/61 layers to GPU
116
+ llm_load_tensors: CPU buffer size = 218873.36 MiB
117
+ llm_load_tensors: CUDA0 buffer size = 20135.96 MiB
118
+ ....................................................................................................
119
+ llama_new_context_with_model: n_ctx = 512
120
+ llama_new_context_with_model: n_batch = 512
121
+ llama_new_context_with_model: n_ubatch = 512
122
+ llama_new_context_with_model: flash_attn = 0
123
+ llama_new_context_with_model: freq_base = 10000.0
124
+ llama_new_context_with_model: freq_scale = 0.025
125
+ llama_kv_cache_init: CUDA_Host KV buffer size = 2200.00 MiB
126
+ llama_kv_cache_init: CUDA0 KV buffer size = 200.00 MiB
127
+ llama_new_context_with_model: KV self size = 2400.00 MiB, K (f16): 1440.00 MiB, V (f16): 960.00 MiB
128
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.39 MiB
129
+ llama_new_context_with_model: CUDA0 compute buffer size = 1422.00 MiB
130
+ llama_new_context_with_model: CUDA_Host compute buffer size = 81.01 MiB
131
+ llama_new_context_with_model: graph nodes = 4480
132
+ llama_new_context_with_model: graph splits = 990
133
+
134
+ system_info: n_threads = 25 (n_threads_batch = 25) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
135
+ compute_imatrix: tokenizing the input ..
136
+ compute_imatrix: tokenization took 227.203 ms
137
+ compute_imatrix: computing over 139 chunks with batch_size 512
138
+ compute_imatrix: 75.87 seconds per pass - ETA 2 hours 55.77 minutes
139
+ [1]5.5015,[2]3.7649,[3]3.6729,[4]4.0519,[5]4.0107,[6]3.8092,[7]4.0137,[8]3.8730,[9]4.3245,
140
+ save_imatrix: stored collected data after 10 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
141
+ [10]4.5113,[11]4.2002,[12]4.4440,[13]4.7697,[14]5.0262,[15]5.1554,[16]5.3737,[17]5.5523,[18]5.6777,[19]5.7885,
142
+ save_imatrix: stored collected data after 20 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
143
+ [20]5.5213,[21]5.5350,[22]5.4706,[23]5.5090,[24]5.4301,[25]5.5961,[26]5.5111,[27]5.6447,[28]5.4996,[29]5.2319,
144
+ save_imatrix: stored collected data after 30 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
145
+ [30]5.0140,[31]4.9498,[32]4.8967,[33]4.7701,[34]4.5632,[35]4.3906,[36]4.3358,[37]4.2842,[38]4.2968,[39]4.2397,
146
+ save_imatrix: stored collected data after 40 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
147
+ [40]4.1962,[41]4.1293,[42]4.0756,[43]4.1196,[44]4.1897,[45]4.2780,[46]4.2952,[47]4.1724,[48]4.0553,[49]3.9527,
148
+ save_imatrix: stored collected data after 50 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
149
+ [50]3.8533,[51]3.8519,[52]3.8249,[53]3.9112,[54]3.9904,[55]4.0774,[56]4.0375,[57]4.0570,[58]4.0796,[59]4.1426,
150
+ save_imatrix: stored collected data after 60 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
151
+ [60]4.2123,[61]4.2756,[62]4.3190,[63]4.3468,[64]4.3812,[65]4.4107,[66]4.4078,[67]4.4155,[68]4.4210,[69]4.4617,
152
+ save_imatrix: stored collected data after 70 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
153
+ [70]4.4805,[71]4.4887,[72]4.5195,[73]4.5275,[74]4.5322,[75]4.5374,[76]4.5511,[77]4.5618,[78]4.5849,[79]4.5736,
154
+ save_imatrix: stored collected data after 80 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
155
+ [80]4.5966,[81]4.5972,[82]4.6062,[83]4.6034,[84]4.6068,[85]4.6044,[86]4.6032,[87]4.5966,[88]4.6224,[89]4.6473,
156
+ save_imatrix: stored collected data after 90 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
157
+ [90]4.6531,[91]4.6850,[92]4.7205,[93]4.6958,[94]4.7014,[95]4.6854,[96]4.7096,[97]4.7276,[98]4.7255,[99]4.6876,
158
+ save_imatrix: stored collected data after 100 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
159
+ [100]4.6500,[101]4.6125,[102]4.5737,[103]4.5371,[104]4.5050,[105]4.4702,[106]4.4360,[107]4.4042,[108]4.3785,[109]4.3936,
160
+ save_imatrix: stored collected data after 110 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
161
+ [110]4.4266,[111]4.4703,[112]4.5140,[113]4.5552,[114]4.6251,[115]4.6639,[116]4.6870,[117]4.7021,[118]4.7271,[119]4.7273,
162
+ save_imatrix: stored collected data after 120 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
163
+ [120]4.7050,[121]4.6528,[122]4.6047,[123]4.6342,[124]4.6665,[125]4.6711,[126]4.6768,[127]4.6893,[128]4.7165,[129]4.7215,
164
+ save_imatrix: stored collected data after 130 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
165
+ [130]4.7351,[131]4.7572,[132]4.7526,[133]4.7442,[134]4.6946,[135]4.6445,[136]4.6437,[137]4.5964,[138]4.5544,[139]4.5095,
166
+ save_imatrix: stored collected data after 139 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat
167
+
168
+ llama_print_timings: load time = 166881.68 ms
169
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
170
+ llama_print_timings: prompt eval time = 15552712.87 ms / 71168 tokens ( 218.54 ms per token, 4.58 tokens per second)
171
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
172
+ llama_print_timings: total time = 15670205.49 ms / 71169 tokens
173
+
174
+ Final estimate: PPL = 4.5095 +/- 0.05621