llama_model_loader: loaded meta data with 35 key-value pairs and 291 tensors from unsloth-Phi-3.5-mini-instruct-IMat-GGUF/unsloth-Phi-3.5-mini-instruct.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Unsloth Phi 3.5 Mini Instruct llama_model_loader: - kv 3: general.finetune str = instruct llama_model_loader: - kv 4: general.basename str = unsloth-Phi-3.5 llama_model_loader: - kv 5: general.size_label str = mini llama_model_loader: - kv 6: general.license str = mit llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-... llama_model_loader: - kv 8: general.tags arr[str,4] = ["unsloth", "transformers", "phi3", "... llama_model_loader: - kv 9: general.languages arr[str,1] = ["multilingual"] llama_model_loader: - kv 10: llama.block_count u32 = 32 llama_model_loader: - kv 11: llama.context_length u32 = 131072 llama_model_loader: - kv 12: llama.embedding_length u32 = 3072 llama_model_loader: - kv 13: llama.feed_forward_length u32 = 8192 llama_model_loader: - kv 14: llama.attention.head_count u32 = 32 llama_model_loader: - kv 15: llama.attention.head_count_kv u32 = 32 llama_model_loader: - kv 16: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 17: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 18: general.file_type u32 = 7 llama_model_loader: - kv 19: llama.vocab_size u32 = 32064 llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 96 llama_model_loader: - kv 21: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 22: tokenizer.ggml.model str = llama llama_model_loader: - kv 23: tokenizer.ggml.pre str = default llama_model_loader: - kv 24: tokenizer.ggml.tokens arr[str,32064] = ["", "~~", "~~", "<0x00>", "<... llama_model_loader: - kv 25: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 27: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 29: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 30: tokenizer.ggml.padding_token_id u32 = 32009 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: tokenizer.chat_template str = {% for message in messages %}{% if me... llama_model_loader: - kv 34: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q8_0: 226 tensors llm_load_vocab: special tokens cache size = 14 llm_load_vocab: token to piece cache size = 0.1685 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32064 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 3072 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_rot = 96 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 96 llm_load_print_meta: n_embd_head_v = 96 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 3072 llm_load_print_meta: n_embd_v_gqa = 3072 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 8192 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 3.82 B llm_load_print_meta: model size = 3.78 GiB (8.50 BPW) llm_load_print_meta: general.name = Unsloth Phi 3.5 Mini Instruct llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 32000 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: PAD token = 32009 '<|placeholder6|>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: EOT token = 32007 '<|end|>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.27 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 99.81 MiB llm_load_tensors: CUDA0 buffer size = 3772.57 MiB ................................................................................................. llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB llama_new_context_with_model: CUDA0 compute buffer size = 68.62 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB llama_new_context_with_model: graph nodes = 1030 llama_new_context_with_model: graph splits = 2 system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | compute_imatrix: tokenizing the input .. compute_imatrix: tokenization took 143.583 ms compute_imatrix: computing over 151 chunks with batch_size 512 compute_imatrix: 0.42 seconds per pass - ETA 1.05 minutes [1]5.6455,[2]4.2496,[3]4.2246,[4]4.7837,[5]5.2227,[6]5.3658,[7]4.8447,[8]5.3686,[9]5.5685, save_imatrix: stored collected data after 10 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [10]5.9472,[11]5.9353,[12]5.4576,[13]5.3625,[14]5.6370,[15]6.0624,[16]6.1669,[17]6.4863,[18]6.6621,[19]6.8056, save_imatrix: stored collected data after 20 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [20]6.9293,[21]7.1482,[22]6.8526,[23]6.5285,[24]6.5929,[25]6.6662,[26]6.5827,[27]6.4753,[28]6.5536,[29]6.7352, save_imatrix: stored collected data after 30 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [30]6.8579,[31]6.8264,[32]6.9704,[33]7.0894,[34]7.2737,[35]7.2872,[36]7.2740,[37]6.9656,[38]6.7646,[39]6.6571, save_imatrix: stored collected data after 40 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [40]6.5504,[41]6.4638,[42]6.3819,[43]6.2510,[44]6.1741,[45]6.0896,[46]6.0434,[47]6.0518,[48]6.1210,[49]6.2249, save_imatrix: stored collected data after 50 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [50]6.2443,[51]6.4326,[52]6.6097,[53]6.8114,[54]7.0049,[55]7.1141,[56]7.0518,[57]6.9795,[58]6.9947,[59]7.0529, save_imatrix: stored collected data after 60 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [60]7.1530,[61]7.0435,[62]7.0612,[63]7.1083,[64]7.1889,[65]7.2617,[66]7.2995,[67]7.3482,[68]7.4029,[69]7.3956, save_imatrix: stored collected data after 70 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [70]7.4210,[71]7.4304,[72]7.4420,[73]7.3864,[74]7.3039,[75]7.2846,[76]7.3311,[77]7.3356,[78]7.2983,[79]7.2867, save_imatrix: stored collected data after 80 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [80]7.2890,[81]7.2540,[82]7.2330,[83]7.2053,[84]7.2096,[85]7.2169,[86]7.2051,[87]7.2006,[88]7.1903,[89]7.1772, save_imatrix: stored collected data after 90 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [90]7.1623,[91]7.1720,[92]7.1281,[93]7.1245,[94]7.0971,[95]7.0535,[96]7.0674,[97]7.0479,[98]7.0504,[99]7.0249, save_imatrix: stored collected data after 100 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [100]7.0132,[101]7.0250,[102]6.9877,[103]6.9490,[104]6.9416,[105]6.9636,[106]6.9712,[107]6.9971,[108]7.0300,[109]6.9885, save_imatrix: stored collected data after 110 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [110]6.9476,[111]6.9077,[112]6.8681,[113]6.8215,[114]6.7730,[115]6.7377,[116]6.6973,[117]6.6657,[118]6.6775,[119]6.6842, save_imatrix: stored collected data after 120 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [120]6.7383,[121]6.7896,[122]6.8503,[123]6.9010,[124]6.9832,[125]7.0567,[126]7.0667,[127]7.0719,[128]7.0068,[129]7.0001, save_imatrix: stored collected data after 130 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [130]6.9700,[131]6.9500,[132]6.9093,[133]6.8680,[134]6.8811,[135]6.9009,[136]6.8981,[137]6.8963,[138]6.9030,[139]6.9162, save_imatrix: stored collected data after 140 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [140]6.9331,[141]6.9349,[142]6.9355,[143]6.9334,[144]6.9116,[145]6.9291,[146]6.9632,[147]7.0090,[148]7.0513,[149]7.0970, save_imatrix: stored collected data after 150 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [150]7.1404,[151]7.1870, save_imatrix: stored collected data after 151 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat llama_print_timings: load time = 1420.67 ms llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: prompt eval time = 55358.52 ms / 77312 tokens ( 0.72 ms per token, 1396.57 tokens per second) llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: total time = 56810.47 ms / 77313 tokens Final estimate: PPL = 7.1870 +/- 0.09220