Hugging Face's logo

GreenBitAI
/
Mistral-7B-v0.1-channel-mix-bpw-3.0

Text Generation
Transformers
Safetensors
mistral
Inference Endpoints
text-generation-inference
Model card Files Community
NicoNico6 commited on
Commit
c0a4789
•
1 Parent(s): 4dfec28

update

Browse files
Files changed (7) hide show
  1. config.json +26 -0
  2. generation_config.json +6 -0
  3. model.safetensors +3 -0
  4. quant_strategy.json +3276 -0
  5. special_tokens_map.json +23 -0
  6. tokenizer.model +3 -0
  7. tokenizer_config.json +43 -0
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/hpi/fs00/share/fg/meinel/nianhui.guo/mistral-hf/models--mistralai--Mistral-7B-v0.1/snapshots/26bca36bde8333b5d7f72e9ed20ccda6a618af24/",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 32,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 10000.0,
20
+ "sliding_window": 4096,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "float16",
23
+ "transformers_version": "4.39.2",
24
+ "use_cache": true,
25
+ "vocab_size": 32000
26
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.39.2"
6
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9bbcee4c07f81b643579ae976441eb65d4beaae73e715d2dee105fbb15c3b23e
3
+ size 3330222928
quant_strategy.json ADDED
@@ -0,0 +1,3276 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "measurement": {
3
+ "model.layers.0": {
4
+ "accuracy": 0.9371232986450195,
5
+ "total_bits": 646461696,
6
+ "q_proj": {
7
+ "group_size": {
8
+ "4": 128,
9
+ "2": 128
10
+ },
11
+ "bits": [
12
+ 4,
13
+ 2
14
+ ],
15
+ "bits_prop": [
16
+ 0.25,
17
+ 0.75
18
+ ],
19
+ "scale_bits": 4
20
+ },
21
+ "k_proj": {
22
+ "group_size": {
23
+ "4": 128,
24
+ "2": 128
25
+ },
26
+ "bits": [
27
+ 4,
28
+ 2
29
+ ],
30
+ "bits_prop": [
31
+ 0.25,
32
+ 0.75
33
+ ],
34
+ "scale_bits": 4
35
+ },
36
+ "v_proj": {
37
+ "group_size": {
38
+ "4": 128
39
+ },
40
+ "bits": [
41
+ 4
42
+ ],
43
+ "bits_prop": [
44
+ 1.0
45
+ ],
46
+ "scale_bits": 4
47
+ },
48
+ "o_proj": {
49
+ "group_size": {
50
+ "4": 128,
51
+ "2": 128
52
+ },
53
+ "bits": [
54
+ 4,
55
+ 2
56
+ ],
57
+ "bits_prop": [
58
+ 0.25,
59
+ 0.75
60
+ ],
61
+ "scale_bits": 4
62
+ },
63
+ "up_proj": {
64
+ "group_size": {
65
+ "4": 128,
66
+ "2": 128
67
+ },
68
+ "bits": [
69
+ 4,
70
+ 2
71
+ ],
72
+ "bits_prop": [
73
+ 0.25,
74
+ 0.75
75
+ ],
76
+ "scale_bits": 4
77
+ },
78
+ "gate_proj": {
79
+ "group_size": {
80
+ "4": 128,
81
+ "2": 128
82
+ },
83
+ "bits": [
84
+ 4,
85
+ 2
86
+ ],
87
+ "bits_prop": [
88
+ 0.25,
89
+ 0.75
90
+ ],
91
+ "scale_bits": 4
92
+ },
93
+ "down_proj": {
94
+ "group_size": {
95
+ "4": 128
96
+ },
97
+ "bits": [
98
+ 4
99
+ ],
100
+ "bits_prop": [
101
+ 1
102
+ ],
103
+ "scale_bits": 4
104
+ }
105
+ },
106
+ "model.layers.1": {
107
+ "accuracy": 0.9321787357330322,
108
+ "total_bits": 646461696,
109
+ "q_proj": {
110
+ "group_size": {
111
+ "4": 128,
112
+ "2": 128
113
+ },
114
+ "bits": [
115
+ 4,
116
+ 2
117
+ ],
118
+ "bits_prop": [
119
+ 0.25,
120
+ 0.75
121
+ ],
122
+ "scale_bits": 4
123
+ },
124
+ "k_proj": {
125
+ "group_size": {
126
+ "4": 128,
127
+ "2": 128
128
+ },
129
+ "bits": [
130
+ 4,
131
+ 2
132
+ ],
133
+ "bits_prop": [
134
+ 0.25,
135
+ 0.75
136
+ ],
137
+ "scale_bits": 4
138
+ },
139
+ "v_proj": {
140
+ "group_size": {
141
+ "4": 128
142
+ },
143
+ "bits": [
144
+ 4
145
+ ],
146
+ "bits_prop": [
147
+ 1.0
148
+ ],
149
+ "scale_bits": 4
150
+ },
151
+ "o_proj": {
152
+ "group_size": {
153
+ "4": 128,
154
+ "2": 128
155
+ },
156
+ "bits": [
157
+ 4,
158
+ 2
159
+ ],
160
+ "bits_prop": [
161
+ 0.25,
162
+ 0.75
163
+ ],
164
+ "scale_bits": 4
165
+ },
166
+ "up_proj": {
167
+ "group_size": {
168
+ "4": 128,
169
+ "2": 128
170
+ },
171
+ "bits": [
172
+ 4,
173
+ 2
174
+ ],
175
+ "bits_prop": [
176
+ 0.25,
177
+ 0.75
178
+ ],
179
+ "scale_bits": 4
180
+ },
181
+ "gate_proj": {
182
+ "group_size": {
183
+ "4": 128,
184
+ "2": 128
185
+ },
186
+ "bits": [
187
+ 4,
188
+ 2
189
+ ],
190
+ "bits_prop": [
191
+ 0.25,
192
+ 0.75
193
+ ],
194
+ "scale_bits": 4
195
+ },
196
+ "down_proj": {
197
+ "group_size": {
198
+ "4": 128
199
+ },
200
+ "bits": [
201
+ 4
202
+ ],
203
+ "bits_prop": [
204
+ 1
205
+ ],
206
+ "scale_bits": 4
207
+ }
208
+ },
209
+ "model.layers.2": {
210
+ "accuracy": 0.9608855843544006,
211
+ "total_bits": 456407296,
212
+ "q_proj": {
213
+ "group_size": {
214
+ "4": 128,
215
+ "2": 128
216
+ },
217
+ "bits": [
218
+ 4,
219
+ 2
220
+ ],
221
+ "bits_prop": [
222
+ 0.01,
223
+ 0.99
224
+ ],
225
+ "scale_bits": 4
226
+ },
227
+ "k_proj": {
228
+ "group_size": {
229
+ "4": 128,
230
+ "2": 128
231
+ },
232
+ "bits": [
233
+ 4,
234
+ 2
235
+ ],
236
+ "bits_prop": [
237
+ 0.01,
238
+ 0.99
239
+ ],
240
+ "scale_bits": 4
241
+ },
242
+ "v_proj": {
243
+ "group_size": {
244
+ "4": 128,
245
+ "2": 128
246
+ },
247
+ "bits": [
248
+ 4,
249
+ 2
250
+ ],
251
+ "bits_prop": [
252
+ 0.05,
253
+ 0.95
254
+ ],
255
+ "scale_bits": 4
256
+ },
257
+ "o_proj": {
258
+ "group_size": {
259
+ "4": 128,
260
+ "2": 128
261
+ },
262
+ "bits": [
263
+ 4,
264
+ 2
265
+ ],
266
+ "bits_prop": [
267
+ 0.02,
268
+ 0.98
269
+ ],
270
+ "scale_bits": 4
271
+ },
272
+ "up_proj": {
273
+ "group_size": {
274
+ "4": 128,
275
+ "2": 128
276
+ },
277
+ "bits": [
278
+ 4,
279
+ 2
280
+ ],
281
+ "bits_prop": [
282
+ 0.01,
283
+ 0.99
284
+ ],
285
+ "scale_bits": 4
286
+ },
287
+ "gate_proj": {
288
+ "group_size": {
289
+ "4": 128,
290
+ "2": 128
291
+ },
292
+ "bits": [
293
+ 4,
294
+ 2
295
+ ],
296
+ "bits_prop": [
297
+ 0.01,
298
+ 0.99
299
+ ],
300
+ "scale_bits": 4
301
+ },
302
+ "down_proj": {
303
+ "group_size": {
304
+ "4": 128,
305
+ "2": 128
306
+ },
307
+ "bits": [
308
+ 4,
309
+ 2
310
+ ],
311
+ "bits_prop": [
312
+ 0.02,
313
+ 0.98
314
+ ],
315
+ "scale_bits": 4
316
+ }
317
+ },
318
+ "model.layers.3": {
319
+ "accuracy": 0.9555550813674927,
320
+ "total_bits": 456407296,
321
+ "q_proj": {
322
+ "group_size": {
323
+ "4": 128,
324
+ "2": 128
325
+ },
326
+ "bits": [
327
+ 4,
328
+ 2
329
+ ],
330
+ "bits_prop": [
331
+ 0.01,
332
+ 0.99
333
+ ],
334
+ "scale_bits": 4
335
+ },
336
+ "k_proj": {
337
+ "group_size": {
338
+ "4": 128,
339
+ "2": 128
340
+ },
341
+ "bits": [
342
+ 4,
343
+ 2
344
+ ],
345
+ "bits_prop": [
346
+ 0.01,
347
+ 0.99
348
+ ],
349
+ "scale_bits": 4
350
+ },
351
+ "v_proj": {
352
+ "group_size": {
353
+ "4": 128,
354
+ "2": 128
355
+ },
356
+ "bits": [
357
+ 4,
358
+ 2
359
+ ],
360
+ "bits_prop": [
361
+ 0.05,
362
+ 0.95
363
+ ],
364
+ "scale_bits": 4
365
+ },
366
+ "o_proj": {
367
+ "group_size": {
368
+ "4": 128,
369
+ "2": 128
370
+ },
371
+ "bits": [
372
+ 4,
373
+ 2
374
+ ],
375
+ "bits_prop": [
376
+ 0.02,
377
+ 0.98
378
+ ],
379
+ "scale_bits": 4
380
+ },
381
+ "up_proj": {
382
+ "group_size": {
383
+ "4": 128,
384
+ "2": 128
385
+ },
386
+ "bits": [
387
+ 4,
388
+ 2
389
+ ],
390
+ "bits_prop": [
391
+ 0.01,
392
+ 0.99
393
+ ],
394
+ "scale_bits": 4
395
+ },
396
+ "gate_proj": {
397
+ "group_size": {
398
+ "4": 128,
399
+ "2": 128
400
+ },
401
+ "bits": [
402
+ 4,
403
+ 2
404
+ ],
405
+ "bits_prop": [
406
+ 0.01,
407
+ 0.99
408
+ ],
409
+ "scale_bits": 4
410
+ },
411
+ "down_proj": {
412
+ "group_size": {
413
+ "4": 128,
414
+ "2": 128
415
+ },
416
+ "bits": [
417
+ 4,
418
+ 2
419
+ ],
420
+ "bits_prop": [
421
+ 0.02,
422
+ 0.98
423
+ ],
424
+ "scale_bits": 4
425
+ }
426
+ },
427
+ "model.layers.4": {
428
+ "accuracy": 0.9511148929595947,
429
+ "total_bits": 469252352,
430
+ "q_proj": {
431
+ "group_size": {
432
+ "4": 128,
433
+ "2": 128
434
+ },
435
+ "bits": [
436
+ 4,
437
+ 2
438
+ ],
439
+ "bits_prop": [
440
+ 0.05,
441
+ 0.95
442
+ ],
443
+ "scale_bits": 4
444
+ },
445
+ "k_proj": {
446
+ "group_size": {
447
+ "4": 128,
448
+ "2": 128
449
+ },
450
+ "bits": [
451
+ 4,
452
+ 2
453
+ ],
454
+ "bits_prop": [
455
+ 0.05,
456
+ 0.95
457
+ ],
458
+ "scale_bits": 4
459
+ },
460
+ "v_proj": {
461
+ "group_size": {
462
+ "4": 128,
463
+ "2": 128
464
+ },
465
+ "bits": [
466
+ 4,
467
+ 2
468
+ ],
469
+ "bits_prop": [
470
+ 0.05,
471
+ 0.95
472
+ ],
473
+ "scale_bits": 4
474
+ },
475
+ "o_proj": {
476
+ "group_size": {
477
+ "4": 128,
478
+ "2": 128
479
+ },
480
+ "bits": [
481
+ 4,
482
+ 2
483
+ ],
484
+ "bits_prop": [
485
+ 0.05,
486
+ 0.95
487
+ ],
488
+ "scale_bits": 4
489
+ },
490
+ "up_proj": {
491
+ "group_size": {
492
+ "4": 128,
493
+ "2": 128
494
+ },
495
+ "bits": [
496
+ 4,
497
+ 2
498
+ ],
499
+ "bits_prop": [
500
+ 0.05,
501
+ 0.95
502
+ ],
503
+ "scale_bits": 4
504
+ },
505
+ "gate_proj": {
506
+ "group_size": {
507
+ "4": 128,
508
+ "2": 128
509
+ },
510
+ "bits": [
511
+ 4,
512
+ 2
513
+ ],
514
+ "bits_prop": [
515
+ 0.05,
516
+ 0.95
517
+ ],
518
+ "scale_bits": 4
519
+ },
520
+ "down_proj": {
521
+ "group_size": {
522
+ "4": 128,
523
+ "2": 128
524
+ },
525
+ "bits": [
526
+ 4,
527
+ 2
528
+ ],
529
+ "bits_prop": [
530
+ 0.05,
531
+ 0.95
532
+ ],
533
+ "scale_bits": 4
534
+ }
535
+ },
536
+ "model.layers.5": {
537
+ "accuracy": 0.9530089497566223,
538
+ "total_bits": 513030400,
539
+ "q_proj": {
540
+ "group_size": {
541
+ "4": 128,
542
+ "2": 128
543
+ },
544
+ "bits": [
545
+ 4,
546
+ 2
547
+ ],
548
+ "bits_prop": [
549
+ 0.05,
550
+ 0.95
551
+ ],
552
+ "scale_bits": 4
553
+ },
554
+ "k_proj": {
555
+ "group_size": {
556
+ "4": 128,
557
+ "2": 128
558
+ },
559
+ "bits": [
560
+ 4,
561
+ 2
562
+ ],
563
+ "bits_prop": [
564
+ 0.05,
565
+ 0.95
566
+ ],
567
+ "scale_bits": 4
568
+ },
569
+ "v_proj": {
570
+ "group_size": {
571
+ "4": 128,
572
+ "2": 128
573
+ },
574
+ "bits": [
575
+ 4,
576
+ 2
577
+ ],
578
+ "bits_prop": [
579
+ 0.4,
580
+ 0.6
581
+ ],
582
+ "scale_bits": 4
583
+ },
584
+ "o_proj": {
585
+ "group_size": {
586
+ "4": 128,
587
+ "2": 128
588
+ },
589
+ "bits": [
590
+ 4,
591
+ 2
592
+ ],
593
+ "bits_prop": [
594
+ 0.05,
595
+ 0.95
596
+ ],
597
+ "scale_bits": 4
598
+ },
599
+ "up_proj": {
600
+ "group_size": {
601
+ "4": 128,
602
+ "2": 128
603
+ },
604
+ "bits": [
605
+ 4,
606
+ 2
607
+ ],
608
+ "bits_prop": [
609
+ 0.05,
610
+ 0.95
611
+ ],
612
+ "scale_bits": 4
613
+ },
614
+ "gate_proj": {
615
+ "group_size": {
616
+ "4": 128,
617
+ "2": 128
618
+ },
619
+ "bits": [
620
+ 4,
621
+ 2
622
+ ],
623
+ "bits_prop": [
624
+ 0.05,
625
+ 0.95
626
+ ],
627
+ "scale_bits": 4
628
+ },
629
+ "down_proj": {
630
+ "group_size": {
631
+ "4": 128,
632
+ "2": 128
633
+ },
634
+ "bits": [
635
+ 4,
636
+ 2
637
+ ],
638
+ "bits_prop": [
639
+ 0.4,
640
+ 0.6
641
+ ],
642
+ "scale_bits": 4
643
+ }
644
+ },
645
+ "model.layers.6": {
646
+ "accuracy": 0.9506196975708008,
647
+ "total_bits": 513030400,
648
+ "q_proj": {
649
+ "group_size": {
650
+ "4": 128,
651
+ "2": 128
652
+ },
653
+ "bits": [
654
+ 4,
655
+ 2
656
+ ],
657
+ "bits_prop": [
658
+ 0.05,
659
+ 0.95
660
+ ],
661
+ "scale_bits": 4
662
+ },
663
+ "k_proj": {
664
+ "group_size": {
665
+ "4": 128,
666
+ "2": 128
667
+ },
668
+ "bits": [
669
+ 4,
670
+ 2
671
+ ],
672
+ "bits_prop": [
673
+ 0.05,
674
+ 0.95
675
+ ],
676
+ "scale_bits": 4
677
+ },
678
+ "v_proj": {
679
+ "group_size": {
680
+ "4": 128,
681
+ "2": 128
682
+ },
683
+ "bits": [
684
+ 4,
685
+ 2
686
+ ],
687
+ "bits_prop": [
688
+ 0.4,
689
+ 0.6
690
+ ],
691
+ "scale_bits": 4
692
+ },
693
+ "o_proj": {
694
+ "group_size": {
695
+ "4": 128,
696
+ "2": 128
697
+ },
698
+ "bits": [
699
+ 4,
700
+ 2
701
+ ],
702
+ "bits_prop": [
703
+ 0.05,
704
+ 0.95
705
+ ],
706
+ "scale_bits": 4
707
+ },
708
+ "up_proj": {
709
+ "group_size": {
710
+ "4": 128,
711
+ "2": 128
712
+ },
713
+ "bits": [
714
+ 4,
715
+ 2
716
+ ],
717
+ "bits_prop": [
718
+ 0.05,
719
+ 0.95
720
+ ],
721
+ "scale_bits": 4
722
+ },
723
+ "gate_proj": {
724
+ "group_size": {
725
+ "4": 128,
726
+ "2": 128
727
+ },
728
+ "bits": [
729
+ 4,
730
+ 2
731
+ ],
732
+ "bits_prop": [
733
+ 0.05,
734
+ 0.95
735
+ ],
736
+ "scale_bits": 4
737
+ },
738
+ "down_proj": {
739
+ "group_size": {
740
+ "4": 128,
741
+ "2": 128
742
+ },
743
+ "bits": [
744
+ 4,
745
+ 2
746
+ ],
747
+ "bits_prop": [
748
+ 0.4,
749
+ 0.6
750
+ ],
751
+ "scale_bits": 4
752
+ }
753
+ },
754
+ "model.layers.7": {
755
+ "accuracy": 0.9419316053390503,
756
+ "total_bits": 513030400,
757
+ "q_proj": {
758
+ "group_size": {
759
+ "4": 128,
760
+ "2": 128
761
+ },
762
+ "bits": [
763
+ 4,
764
+ 2
765
+ ],
766
+ "bits_prop": [
767
+ 0.05,
768
+ 0.95
769
+ ],
770
+ "scale_bits": 4
771
+ },
772
+ "k_proj": {
773
+ "group_size": {
774
+ "4": 128,
775
+ "2": 128
776
+ },
777
+ "bits": [
778
+ 4,
779
+ 2
780
+ ],
781
+ "bits_prop": [
782
+ 0.05,
783
+ 0.95
784
+ ],
785
+ "scale_bits": 4
786
+ },
787
+ "v_proj": {
788
+ "group_size": {
789
+ "4": 128,
790
+ "2": 128
791
+ },
792
+ "bits": [
793
+ 4,
794
+ 2
795
+ ],
796
+ "bits_prop": [
797
+ 0.4,
798
+ 0.6
799
+ ],
800
+ "scale_bits": 4
801
+ },
802
+ "o_proj": {
803
+ "group_size": {
804
+ "4": 128,
805
+ "2": 128
806
+ },
807
+ "bits": [
808
+ 4,
809
+ 2
810
+ ],
811
+ "bits_prop": [
812
+ 0.05,
813
+ 0.95
814
+ ],
815
+ "scale_bits": 4
816
+ },
817
+ "up_proj": {
818
+ "group_size": {
819
+ "4": 128,
820
+ "2": 128
821
+ },
822
+ "bits": [
823
+ 4,
824
+ 2
825
+ ],
826
+ "bits_prop": [
827
+ 0.05,
828
+ 0.95
829
+ ],
830
+ "scale_bits": 4
831
+ },
832
+ "gate_proj": {
833
+ "group_size": {
834
+ "4": 128,
835
+ "2": 128
836
+ },
837
+ "bits": [
838
+ 4,
839
+ 2
840
+ ],
841
+ "bits_prop": [
842
+ 0.05,
843
+ 0.95
844
+ ],
845
+ "scale_bits": 4
846
+ },
847
+ "down_proj": {
848
+ "group_size": {
849
+ "4": 128,
850
+ "2": 128
851
+ },
852
+ "bits": [
853
+ 4,
854
+ 2
855
+ ],
856
+ "bits_prop": [
857
+ 0.4,
858
+ 0.6
859
+ ],
860
+ "scale_bits": 4
861
+ }
862
+ },
863
+ "model.layers.8": {
864
+ "accuracy": 0.9573614597320557,
865
+ "total_bits": 607664384,
866
+ "q_proj": {
867
+ "group_size": {
868
+ "4": 128,
869
+ "2": 128
870
+ },
871
+ "bits": [
872
+ 4,
873
+ 2
874
+ ],
875
+ "bits_prop": [
876
+ 0.1,
877
+ 0.9
878
+ ],
879
+ "scale_bits": 4
880
+ },
881
+ "k_proj": {
882
+ "group_size": {
883
+ "4": 128,
884
+ "2": 128
885
+ },
886
+ "bits": [
887
+ 4,
888
+ 2
889
+ ],
890
+ "bits_prop": [
891
+ 0.1,
892
+ 0.9
893
+ ],
894
+ "scale_bits": 4
895
+ },
896
+ "v_proj": {
897
+ "group_size": {
898
+ "4": 128
899
+ },
900
+ "bits": [
901
+ 4
902
+ ],
903
+ "bits_prop": [
904
+ 1.0
905
+ ],
906
+ "scale_bits": 4
907
+ },
908
+ "o_proj": {
909
+ "group_size": {
910
+ "4": 128,
911
+ "2": 128
912
+ },
913
+ "bits": [
914
+ 4,
915
+ 2
916
+ ],
917
+ "bits_prop": [
918
+ 0.1,
919
+ 0.9
920
+ ],
921
+ "scale_bits": 4
922
+ },
923
+ "up_proj": {
924
+ "group_size": {
925
+ "4": 128,
926
+ "2": 128
927
+ },
928
+ "bits": [
929
+ 4,
930
+ 2
931
+ ],
932
+ "bits_prop": [
933
+ 0.1,
934
+ 0.9
935
+ ],
936
+ "scale_bits": 4
937
+ },
938
+ "gate_proj": {
939
+ "group_size": {
940
+ "4": 128,
941
+ "2": 128
942
+ },
943
+ "bits": [
944
+ 4,
945
+ 2
946
+ ],
947
+ "bits_prop": [
948
+ 0.1,
949
+ 0.9
950
+ ],
951
+ "scale_bits": 4
952
+ },
953
+ "down_proj": {
954
+ "group_size": {
955
+ "4": 128
956
+ },
957
+ "bits": [
958
+ 4
959
+ ],
960
+ "bits_prop": [
961
+ 1.0
962
+ ],
963
+ "scale_bits": 4
964
+ }
965
+ },
966
+ "model.layers.9": {
967
+ "accuracy": 0.9613507986068726,
968
+ "total_bits": 646461696,
969
+ "q_proj": {
970
+ "group_size": {
971
+ "4": 128,
972
+ "2": 128
973
+ },
974
+ "bits": [
975
+ 4,
976
+ 2
977
+ ],
978
+ "bits_prop": [
979
+ 0.25,
980
+ 0.75
981
+ ],
982
+ "scale_bits": 4
983
+ },
984
+ "k_proj": {
985
+ "group_size": {
986
+ "4": 128,
987
+ "2": 128
988
+ },
989
+ "bits": [
990
+ 4,
991
+ 2
992
+ ],
993
+ "bits_prop": [
994
+ 0.25,
995
+ 0.75
996
+ ],
997
+ "scale_bits": 4
998
+ },
999
+ "v_proj": {
1000
+ "group_size": {
1001
+ "4": 128
1002
+ },
1003
+ "bits": [
1004
+ 4
1005
+ ],
1006
+ "bits_prop": [
1007
+ 1.0
1008
+ ],
1009
+ "scale_bits": 4
1010
+ },
1011
+ "o_proj": {
1012
+ "group_size": {
1013
+ "4": 128,
1014
+ "2": 128
1015
+ },
1016
+ "bits": [
1017
+ 4,
1018
+ 2
1019
+ ],
1020
+ "bits_prop": [
1021
+ 0.25,
1022
+ 0.75
1023
+ ],
1024
+ "scale_bits": 4
1025
+ },
1026
+ "up_proj": {
1027
+ "group_size": {
1028
+ "4": 128,
1029
+ "2": 128
1030
+ },
1031
+ "bits": [
1032
+ 4,
1033
+ 2
1034
+ ],
1035
+ "bits_prop": [
1036
+ 0.25,
1037
+ 0.75
1038
+ ],
1039
+ "scale_bits": 4
1040
+ },
1041
+ "gate_proj": {
1042
+ "group_size": {
1043
+ "4": 128,
1044
+ "2": 128
1045
+ },
1046
+ "bits": [
1047
+ 4,
1048
+ 2
1049
+ ],
1050
+ "bits_prop": [
1051
+ 0.25,
1052
+ 0.75
1053
+ ],
1054
+ "scale_bits": 4
1055
+ },
1056
+ "down_proj": {
1057
+ "group_size": {
1058
+ "4": 128
1059
+ },
1060
+ "bits": [
1061
+ 4
1062
+ ],
1063
+ "bits_prop": [
1064
+ 1
1065
+ ],
1066
+ "scale_bits": 4
1067
+ }
1068
+ },
1069
+ "model.layers.10": {
1070
+ "accuracy": 0.96047043800354,
1071
+ "total_bits": 646461696,
1072
+ "q_proj": {
1073
+ "group_size": {
1074
+ "4": 128,
1075
+ "2": 128
1076
+ },
1077
+ "bits": [
1078
+ 4,
1079
+ 2
1080
+ ],
1081
+ "bits_prop": [
1082
+ 0.25,
1083
+ 0.75
1084
+ ],
1085
+ "scale_bits": 4
1086
+ },
1087
+ "k_proj": {
1088
+ "group_size": {
1089
+ "4": 128,
1090
+ "2": 128
1091
+ },
1092
+ "bits": [
1093
+ 4,
1094
+ 2
1095
+ ],
1096
+ "bits_prop": [
1097
+ 0.25,
1098
+ 0.75
1099
+ ],
1100
+ "scale_bits": 4
1101
+ },
1102
+ "v_proj": {
1103
+ "group_size": {
1104
+ "4": 128
1105
+ },
1106
+ "bits": [
1107
+ 4
1108
+ ],
1109
+ "bits_prop": [
1110
+ 1.0
1111
+ ],
1112
+ "scale_bits": 4
1113
+ },
1114
+ "o_proj": {
1115
+ "group_size": {
1116
+ "4": 128,
1117
+ "2": 128
1118
+ },
1119
+ "bits": [
1120
+ 4,
1121
+ 2
1122
+ ],
1123
+ "bits_prop": [
1124
+ 0.25,
1125
+ 0.75
1126
+ ],
1127
+ "scale_bits": 4
1128
+ },
1129
+ "up_proj": {
1130
+ "group_size": {
1131
+ "4": 128,
1132
+ "2": 128
1133
+ },
1134
+ "bits": [
1135
+ 4,
1136
+ 2
1137
+ ],
1138
+ "bits_prop": [
1139
+ 0.25,
1140
+ 0.75
1141
+ ],
1142
+ "scale_bits": 4
1143
+ },
1144
+ "gate_proj": {
1145
+ "group_size": {
1146
+ "4": 128,
1147
+ "2": 128
1148
+ },
1149
+ "bits": [
1150
+ 4,
1151
+ 2
1152
+ ],
1153
+ "bits_prop": [
1154
+ 0.25,
1155
+ 0.75
1156
+ ],
1157
+ "scale_bits": 4
1158
+ },
1159
+ "down_proj": {
1160
+ "group_size": {
1161
+ "4": 128
1162
+ },
1163
+ "bits": [
1164
+ 4
1165
+ ],
1166
+ "bits_prop": [
1167
+ 1
1168
+ ],
1169
+ "scale_bits": 4
1170
+ }
1171
+ },
1172
+ "model.layers.11": {
1173
+ "accuracy": 0.960319995880127,
1174
+ "total_bits": 646461696,
1175
+ "q_proj": {
1176
+ "group_size": {
1177
+ "4": 128,
1178
+ "2": 128
1179
+ },
1180
+ "bits": [
1181
+ 4,
1182
+ 2
1183
+ ],
1184
+ "bits_prop": [
1185
+ 0.25,
1186
+ 0.75
1187
+ ],
1188
+ "scale_bits": 4
1189
+ },
1190
+ "k_proj": {
1191
+ "group_size": {
1192
+ "4": 128,
1193
+ "2": 128
1194
+ },
1195
+ "bits": [
1196
+ 4,
1197
+ 2
1198
+ ],
1199
+ "bits_prop": [
1200
+ 0.25,
1201
+ 0.75
1202
+ ],
1203
+ "scale_bits": 4
1204
+ },
1205
+ "v_proj": {
1206
+ "group_size": {
1207
+ "4": 128
1208
+ },
1209
+ "bits": [
1210
+ 4
1211
+ ],
1212
+ "bits_prop": [
1213
+ 1.0
1214
+ ],
1215
+ "scale_bits": 4
1216
+ },
1217
+ "o_proj": {
1218
+ "group_size": {
1219
+ "4": 128,
1220
+ "2": 128
1221
+ },
1222
+ "bits": [
1223
+ 4,
1224
+ 2
1225
+ ],
1226
+ "bits_prop": [
1227
+ 0.25,
1228
+ 0.75
1229
+ ],
1230
+ "scale_bits": 4
1231
+ },
1232
+ "up_proj": {
1233
+ "group_size": {
1234
+ "4": 128,
1235
+ "2": 128
1236
+ },
1237
+ "bits": [
1238
+ 4,
1239
+ 2
1240
+ ],
1241
+ "bits_prop": [
1242
+ 0.25,
1243
+ 0.75
1244
+ ],
1245
+ "scale_bits": 4
1246
+ },
1247
+ "gate_proj": {
1248
+ "group_size": {
1249
+ "4": 128,
1250
+ "2": 128
1251
+ },
1252
+ "bits": [
1253
+ 4,
1254
+ 2
1255
+ ],
1256
+ "bits_prop": [
1257
+ 0.25,
1258
+ 0.75
1259
+ ],
1260
+ "scale_bits": 4
1261
+ },
1262
+ "down_proj": {
1263
+ "group_size": {
1264
+ "4": 128
1265
+ },
1266
+ "bits": [
1267
+ 4
1268
+ ],
1269
+ "bits_prop": [
1270
+ 1
1271
+ ],
1272
+ "scale_bits": 4
1273
+ }
1274
+ },
1275
+ "model.layers.12": {
1276
+ "accuracy": 0.9560093879699707,
1277
+ "total_bits": 646461696,
1278
+ "q_proj": {
1279
+ "group_size": {
1280
+ "4": 128,
1281
+ "2": 128
1282
+ },
1283
+ "bits": [
1284
+ 4,
1285
+ 2
1286
+ ],
1287
+ "bits_prop": [
1288
+ 0.25,
1289
+ 0.75
1290
+ ],
1291
+ "scale_bits": 4
1292
+ },
1293
+ "k_proj": {
1294
+ "group_size": {
1295
+ "4": 128,
1296
+ "2": 128
1297
+ },
1298
+ "bits": [
1299
+ 4,
1300
+ 2
1301
+ ],
1302
+ "bits_prop": [
1303
+ 0.25,
1304
+ 0.75
1305
+ ],
1306
+ "scale_bits": 4
1307
+ },
1308
+ "v_proj": {
1309
+ "group_size": {
1310
+ "4": 128
1311
+ },
1312
+ "bits": [
1313
+ 4
1314
+ ],
1315
+ "bits_prop": [
1316
+ 1.0
1317
+ ],
1318
+ "scale_bits": 4
1319
+ },
1320
+ "o_proj": {
1321
+ "group_size": {
1322
+ "4": 128,
1323
+ "2": 128
1324
+ },
1325
+ "bits": [
1326
+ 4,
1327
+ 2
1328
+ ],
1329
+ "bits_prop": [
1330
+ 0.25,
1331
+ 0.75
1332
+ ],
1333
+ "scale_bits": 4
1334
+ },
1335
+ "up_proj": {
1336
+ "group_size": {
1337
+ "4": 128,
1338
+ "2": 128
1339
+ },
1340
+ "bits": [
1341
+ 4,
1342
+ 2
1343
+ ],
1344
+ "bits_prop": [
1345
+ 0.25,
1346
+ 0.75
1347
+ ],
1348
+ "scale_bits": 4
1349
+ },
1350
+ "gate_proj": {
1351
+ "group_size": {
1352
+ "4": 128,
1353
+ "2": 128
1354
+ },
1355
+ "bits": [
1356
+ 4,
1357
+ 2
1358
+ ],
1359
+ "bits_prop": [
1360
+ 0.25,
1361
+ 0.75
1362
+ ],
1363
+ "scale_bits": 4
1364
+ },
1365
+ "down_proj": {
1366
+ "group_size": {
1367
+ "4": 128
1368
+ },
1369
+ "bits": [
1370
+ 4
1371
+ ],
1372
+ "bits_prop": [
1373
+ 1
1374
+ ],
1375
+ "scale_bits": 4
1376
+ }
1377
+ },
1378
+ "model.layers.13": {
1379
+ "accuracy": 0.9528203010559082,
1380
+ "total_bits": 646461696,
1381
+ "q_proj": {
1382
+ "group_size": {
1383
+ "4": 128,
1384
+ "2": 128
1385
+ },
1386
+ "bits": [
1387
+ 4,
1388
+ 2
1389
+ ],
1390
+ "bits_prop": [
1391
+ 0.25,
1392
+ 0.75
1393
+ ],
1394
+ "scale_bits": 4
1395
+ },
1396
+ "k_proj": {
1397
+ "group_size": {
1398
+ "4": 128,
1399
+ "2": 128
1400
+ },
1401
+ "bits": [
1402
+ 4,
1403
+ 2
1404
+ ],
1405
+ "bits_prop": [
1406
+ 0.25,
1407
+ 0.75
1408
+ ],
1409
+ "scale_bits": 4
1410
+ },
1411
+ "v_proj": {
1412
+ "group_size": {
1413
+ "4": 128
1414
+ },
1415
+ "bits": [
1416
+ 4
1417
+ ],
1418
+ "bits_prop": [
1419
+ 1.0
1420
+ ],
1421
+ "scale_bits": 4
1422
+ },
1423
+ "o_proj": {
1424
+ "group_size": {
1425
+ "4": 128,
1426
+ "2": 128
1427
+ },
1428
+ "bits": [
1429
+ 4,
1430
+ 2
1431
+ ],
1432
+ "bits_prop": [
1433
+ 0.25,
1434
+ 0.75
1435
+ ],
1436
+ "scale_bits": 4
1437
+ },
1438
+ "up_proj": {
1439
+ "group_size": {
1440
+ "4": 128,
1441
+ "2": 128
1442
+ },
1443
+ "bits": [
1444
+ 4,
1445
+ 2
1446
+ ],
1447
+ "bits_prop": [
1448
+ 0.25,
1449
+ 0.75
1450
+ ],
1451
+ "scale_bits": 4
1452
+ },
1453
+ "gate_proj": {
1454
+ "group_size": {
1455
+ "4": 128,
1456
+ "2": 128
1457
+ },
1458
+ "bits": [
1459
+ 4,
1460
+ 2
1461
+ ],
1462
+ "bits_prop": [
1463
+ 0.25,
1464
+ 0.75
1465
+ ],
1466
+ "scale_bits": 4
1467
+ },
1468
+ "down_proj": {
1469
+ "group_size": {
1470
+ "4": 128
1471
+ },
1472
+ "bits": [
1473
+ 4
1474
+ ],
1475
+ "bits_prop": [
1476
+ 1
1477
+ ],
1478
+ "scale_bits": 4
1479
+ }
1480
+ },
1481
+ "model.layers.14": {
1482
+ "accuracy": 0.9506630897521973,
1483
+ "total_bits": 646461696,
1484
+ "q_proj": {
1485
+ "group_size": {
1486
+ "4": 128,
1487
+ "2": 128
1488
+ },
1489
+ "bits": [
1490
+ 4,
1491
+ 2
1492
+ ],
1493
+ "bits_prop": [
1494
+ 0.25,
1495
+ 0.75
1496
+ ],
1497
+ "scale_bits": 4
1498
+ },
1499
+ "k_proj": {
1500
+ "group_size": {
1501
+ "4": 128,
1502
+ "2": 128
1503
+ },
1504
+ "bits": [
1505
+ 4,
1506
+ 2
1507
+ ],
1508
+ "bits_prop": [
1509
+ 0.25,
1510
+ 0.75
1511
+ ],
1512
+ "scale_bits": 4
1513
+ },
1514
+ "v_proj": {
1515
+ "group_size": {
1516
+ "4": 128
1517
+ },
1518
+ "bits": [
1519
+ 4
1520
+ ],
1521
+ "bits_prop": [
1522
+ 1.0
1523
+ ],
1524
+ "scale_bits": 4
1525
+ },
1526
+ "o_proj": {
1527
+ "group_size": {
1528
+ "4": 128,
1529
+ "2": 128
1530
+ },
1531
+ "bits": [
1532
+ 4,
1533
+ 2
1534
+ ],
1535
+ "bits_prop": [
1536
+ 0.25,
1537
+ 0.75
1538
+ ],
1539
+ "scale_bits": 4
1540
+ },
1541
+ "up_proj": {
1542
+ "group_size": {
1543
+ "4": 128,
1544
+ "2": 128
1545
+ },
1546
+ "bits": [
1547
+ 4,
1548
+ 2
1549
+ ],
1550
+ "bits_prop": [
1551
+ 0.25,
1552
+ 0.75
1553
+ ],
1554
+ "scale_bits": 4
1555
+ },
1556
+ "gate_proj": {
1557
+ "group_size": {
1558
+ "4": 128,
1559
+ "2": 128
1560
+ },
1561
+ "bits": [
1562
+ 4,
1563
+ 2
1564
+ ],
1565
+ "bits_prop": [
1566
+ 0.25,
1567
+ 0.75
1568
+ ],
1569
+ "scale_bits": 4
1570
+ },
1571
+ "down_proj": {
1572
+ "group_size": {
1573
+ "4": 128
1574
+ },
1575
+ "bits": [
1576
+ 4
1577
+ ],
1578
+ "bits_prop": [
1579
+ 1
1580
+ ],
1581
+ "scale_bits": 4
1582
+ }
1583
+ },
1584
+ "model.layers.15": {
1585
+ "accuracy": 0.9465978145599365,
1586
+ "total_bits": 646461696,
1587
+ "q_proj": {
1588
+ "group_size": {
1589
+ "4": 128,
1590
+ "2": 128
1591
+ },
1592
+ "bits": [
1593
+ 4,
1594
+ 2
1595
+ ],
1596
+ "bits_prop": [
1597
+ 0.25,
1598
+ 0.75
1599
+ ],
1600
+ "scale_bits": 4
1601
+ },
1602
+ "k_proj": {
1603
+ "group_size": {
1604
+ "4": 128,
1605
+ "2": 128
1606
+ },
1607
+ "bits": [
1608
+ 4,
1609
+ 2
1610
+ ],
1611
+ "bits_prop": [
1612
+ 0.25,
1613
+ 0.75
1614
+ ],
1615
+ "scale_bits": 4
1616
+ },
1617
+ "v_proj": {
1618
+ "group_size": {
1619
+ "4": 128
1620
+ },
1621
+ "bits": [
1622
+ 4
1623
+ ],
1624
+ "bits_prop": [
1625
+ 1.0
1626
+ ],
1627
+ "scale_bits": 4
1628
+ },
1629
+ "o_proj": {
1630
+ "group_size": {
1631
+ "4": 128,
1632
+ "2": 128
1633
+ },
1634
+ "bits": [
1635
+ 4,
1636
+ 2
1637
+ ],
1638
+ "bits_prop": [
1639
+ 0.25,
1640
+ 0.75
1641
+ ],
1642
+ "scale_bits": 4
1643
+ },
1644
+ "up_proj": {
1645
+ "group_size": {
1646
+ "4": 128,
1647
+ "2": 128
1648
+ },
1649
+ "bits": [
1650
+ 4,
1651
+ 2
1652
+ ],
1653
+ "bits_prop": [
1654
+ 0.25,
1655
+ 0.75
1656
+ ],
1657
+ "scale_bits": 4
1658
+ },
1659
+ "gate_proj": {
1660
+ "group_size": {
1661
+ "4": 128,
1662
+ "2": 128
1663
+ },
1664
+ "bits": [
1665
+ 4,
1666
+ 2
1667
+ ],
1668
+ "bits_prop": [
1669
+ 0.25,
1670
+ 0.75
1671
+ ],
1672
+ "scale_bits": 4
1673
+ },
1674
+ "down_proj": {
1675
+ "group_size": {
1676
+ "4": 128
1677
+ },
1678
+ "bits": [
1679
+ 4
1680
+ ],
1681
+ "bits_prop": [
1682
+ 1
1683
+ ],
1684
+ "scale_bits": 4
1685
+ }
1686
+ },
1687
+ "model.layers.16": {
1688
+ "accuracy": 0.9447896480560303,
1689
+ "total_bits": 646461696,
1690
+ "q_proj": {
1691
+ "group_size": {
1692
+ "4": 128,
1693
+ "2": 128
1694
+ },
1695
+ "bits": [
1696
+ 4,
1697
+ 2
1698
+ ],
1699
+ "bits_prop": [
1700
+ 0.25,
1701
+ 0.75
1702
+ ],
1703
+ "scale_bits": 4
1704
+ },
1705
+ "k_proj": {
1706
+ "group_size": {
1707
+ "4": 128,
1708
+ "2": 128
1709
+ },
1710
+ "bits": [
1711
+ 4,
1712
+ 2
1713
+ ],
1714
+ "bits_prop": [
1715
+ 0.25,
1716
+ 0.75
1717
+ ],
1718
+ "scale_bits": 4
1719
+ },
1720
+ "v_proj": {
1721
+ "group_size": {
1722
+ "4": 128
1723
+ },
1724
+ "bits": [
1725
+ 4
1726
+ ],
1727
+ "bits_prop": [
1728
+ 1.0
1729
+ ],
1730
+ "scale_bits": 4
1731
+ },
1732
+ "o_proj": {
1733
+ "group_size": {
1734
+ "4": 128,
1735
+ "2": 128
1736
+ },
1737
+ "bits": [
1738
+ 4,
1739
+ 2
1740
+ ],
1741
+ "bits_prop": [
1742
+ 0.25,
1743
+ 0.75
1744
+ ],
1745
+ "scale_bits": 4
1746
+ },
1747
+ "up_proj": {
1748
+ "group_size": {
1749
+ "4": 128,
1750
+ "2": 128
1751
+ },
1752
+ "bits": [
1753
+ 4,
1754
+ 2
1755
+ ],
1756
+ "bits_prop": [
1757
+ 0.25,
1758
+ 0.75
1759
+ ],
1760
+ "scale_bits": 4
1761
+ },
1762
+ "gate_proj": {
1763
+ "group_size": {
1764
+ "4": 128,
1765
+ "2": 128
1766
+ },
1767
+ "bits": [
1768
+ 4,
1769
+ 2
1770
+ ],
1771
+ "bits_prop": [
1772
+ 0.25,
1773
+ 0.75
1774
+ ],
1775
+ "scale_bits": 4
1776
+ },
1777
+ "down_proj": {
1778
+ "group_size": {
1779
+ "4": 128
1780
+ },
1781
+ "bits": [
1782
+ 4
1783
+ ],
1784
+ "bits_prop": [
1785
+ 1
1786
+ ],
1787
+ "scale_bits": 4
1788
+ }
1789
+ },
1790
+ "model.layers.17": {
1791
+ "accuracy": 0.9538173675537109,
1792
+ "total_bits": 724056320,
1793
+ "q_proj": {
1794
+ "group_size": {
1795
+ "4": 128,
1796
+ "2": 128
1797
+ },
1798
+ "bits": [
1799
+ 4,
1800
+ 2
1801
+ ],
1802
+ "bits_prop": [
1803
+ 0.5,
1804
+ 0.5
1805
+ ],
1806
+ "scale_bits": 4
1807
+ },
1808
+ "k_proj": {
1809
+ "group_size": {
1810
+ "4": 128,
1811
+ "2": 128
1812
+ },
1813
+ "bits": [
1814
+ 4,
1815
+ 2
1816
+ ],
1817
+ "bits_prop": [
1818
+ 0.5,
1819
+ 0.5
1820
+ ],
1821
+ "scale_bits": 4
1822
+ },
1823
+ "v_proj": {
1824
+ "group_size": {
1825
+ "4": 128
1826
+ },
1827
+ "bits": [
1828
+ 4
1829
+ ],
1830
+ "bits_prop": [
1831
+ 1.0
1832
+ ],
1833
+ "scale_bits": 4
1834
+ },
1835
+ "o_proj": {
1836
+ "group_size": {
1837
+ "4": 128,
1838
+ "2": 128
1839
+ },
1840
+ "bits": [
1841
+ 4,
1842
+ 2
1843
+ ],
1844
+ "bits_prop": [
1845
+ 0.5,
1846
+ 0.5
1847
+ ],
1848
+ "scale_bits": 4
1849
+ },
1850
+ "up_proj": {
1851
+ "group_size": {
1852
+ "4": 128,
1853
+ "2": 128
1854
+ },
1855
+ "bits": [
1856
+ 4,
1857
+ 2
1858
+ ],
1859
+ "bits_prop": [
1860
+ 0.5,
1861
+ 0.5
1862
+ ],
1863
+ "scale_bits": 4
1864
+ },
1865
+ "gate_proj": {
1866
+ "group_size": {
1867
+ "4": 128,
1868
+ "2": 128
1869
+ },
1870
+ "bits": [
1871
+ 4,
1872
+ 2
1873
+ ],
1874
+ "bits_prop": [
1875
+ 0.5,
1876
+ 0.5
1877
+ ],
1878
+ "scale_bits": 4
1879
+ },
1880
+ "down_proj": {
1881
+ "group_size": {
1882
+ "4": 128
1883
+ },
1884
+ "bits": [
1885
+ 4
1886
+ ],
1887
+ "bits_prop": [
1888
+ 1.0
1889
+ ],
1890
+ "scale_bits": 4
1891
+ }
1892
+ },
1893
+ "model.layers.18": {
1894
+ "accuracy": 0.953197717666626,
1895
+ "total_bits": 724056320,
1896
+ "q_proj": {
1897
+ "group_size": {
1898
+ "4": 128,
1899
+ "2": 128
1900
+ },
1901
+ "bits": [
1902
+ 4,
1903
+ 2
1904
+ ],
1905
+ "bits_prop": [
1906
+ 0.5,
1907
+ 0.5
1908
+ ],
1909
+ "scale_bits": 4
1910
+ },
1911
+ "k_proj": {
1912
+ "group_size": {
1913
+ "4": 128,
1914
+ "2": 128
1915
+ },
1916
+ "bits": [
1917
+ 4,
1918
+ 2
1919
+ ],
1920
+ "bits_prop": [
1921
+ 0.5,
1922
+ 0.5
1923
+ ],
1924
+ "scale_bits": 4
1925
+ },
1926
+ "v_proj": {
1927
+ "group_size": {
1928
+ "4": 128
1929
+ },
1930
+ "bits": [
1931
+ 4
1932
+ ],
1933
+ "bits_prop": [
1934
+ 1.0
1935
+ ],
1936
+ "scale_bits": 4
1937
+ },
1938
+ "o_proj": {
1939
+ "group_size": {
1940
+ "4": 128,
1941
+ "2": 128
1942
+ },
1943
+ "bits": [
1944
+ 4,
1945
+ 2
1946
+ ],
1947
+ "bits_prop": [
1948
+ 0.5,
1949
+ 0.5
1950
+ ],
1951
+ "scale_bits": 4
1952
+ },
1953
+ "up_proj": {
1954
+ "group_size": {
1955
+ "4": 128,
1956
+ "2": 128
1957
+ },
1958
+ "bits": [
1959
+ 4,
1960
+ 2
1961
+ ],
1962
+ "bits_prop": [
1963
+ 0.5,
1964
+ 0.5
1965
+ ],
1966
+ "scale_bits": 4
1967
+ },
1968
+ "gate_proj": {
1969
+ "group_size": {
1970
+ "4": 128,
1971
+ "2": 128
1972
+ },
1973
+ "bits": [
1974
+ 4,
1975
+ 2
1976
+ ],
1977
+ "bits_prop": [
1978
+ 0.5,
1979
+ 0.5
1980
+ ],
1981
+ "scale_bits": 4
1982
+ },
1983
+ "down_proj": {
1984
+ "group_size": {
1985
+ "4": 128
1986
+ },
1987
+ "bits": [
1988
+ 4
1989
+ ],
1990
+ "bits_prop": [
1991
+ 1.0
1992
+ ],
1993
+ "scale_bits": 4
1994
+ }
1995
+ },
1996
+ "model.layers.19": {
1997
+ "accuracy": 0.9511487483978271,
1998
+ "total_bits": 724056320,
1999
+ "q_proj": {
2000
+ "group_size": {
2001
+ "4": 128,
2002
+ "2": 128
2003
+ },
2004
+ "bits": [
2005
+ 4,
2006
+ 2
2007
+ ],
2008
+ "bits_prop": [
2009
+ 0.5,
2010
+ 0.5
2011
+ ],
2012
+ "scale_bits": 4
2013
+ },
2014
+ "k_proj": {
2015
+ "group_size": {
2016
+ "4": 128,
2017
+ "2": 128
2018
+ },
2019
+ "bits": [
2020
+ 4,
2021
+ 2
2022
+ ],
2023
+ "bits_prop": [
2024
+ 0.5,
2025
+ 0.5
2026
+ ],
2027
+ "scale_bits": 4
2028
+ },
2029
+ "v_proj": {
2030
+ "group_size": {
2031
+ "4": 128
2032
+ },
2033
+ "bits": [
2034
+ 4
2035
+ ],
2036
+ "bits_prop": [
2037
+ 1.0
2038
+ ],
2039
+ "scale_bits": 4
2040
+ },
2041
+ "o_proj": {
2042
+ "group_size": {
2043
+ "4": 128,
2044
+ "2": 128
2045
+ },
2046
+ "bits": [
2047
+ 4,
2048
+ 2
2049
+ ],
2050
+ "bits_prop": [
2051
+ 0.5,
2052
+ 0.5
2053
+ ],
2054
+ "scale_bits": 4
2055
+ },
2056
+ "up_proj": {
2057
+ "group_size": {
2058
+ "4": 128,
2059
+ "2": 128
2060
+ },
2061
+ "bits": [
2062
+ 4,
2063
+ 2
2064
+ ],
2065
+ "bits_prop": [
2066
+ 0.5,
2067
+ 0.5
2068
+ ],
2069
+ "scale_bits": 4
2070
+ },
2071
+ "gate_proj": {
2072
+ "group_size": {
2073
+ "4": 128,
2074
+ "2": 128
2075
+ },
2076
+ "bits": [
2077
+ 4,
2078
+ 2
2079
+ ],
2080
+ "bits_prop": [
2081
+ 0.5,
2082
+ 0.5
2083
+ ],
2084
+ "scale_bits": 4
2085
+ },
2086
+ "down_proj": {
2087
+ "group_size": {
2088
+ "4": 128
2089
+ },
2090
+ "bits": [
2091
+ 4
2092
+ ],
2093
+ "bits_prop": [
2094
+ 1.0
2095
+ ],
2096
+ "scale_bits": 4
2097
+ }
2098
+ },
2099
+ "model.layers.20": {
2100
+ "accuracy": 0.9419491291046143,
2101
+ "total_bits": 646461696,
2102
+ "q_proj": {
2103
+ "group_size": {
2104
+ "4": 128,
2105
+ "2": 128
2106
+ },
2107
+ "bits": [
2108
+ 4,
2109
+ 2
2110
+ ],
2111
+ "bits_prop": [
2112
+ 0.25,
2113
+ 0.75
2114
+ ],
2115
+ "scale_bits": 4
2116
+ },
2117
+ "k_proj": {
2118
+ "group_size": {
2119
+ "4": 128,
2120
+ "2": 128
2121
+ },
2122
+ "bits": [
2123
+ 4,
2124
+ 2
2125
+ ],
2126
+ "bits_prop": [
2127
+ 0.25,
2128
+ 0.75
2129
+ ],
2130
+ "scale_bits": 4
2131
+ },
2132
+ "v_proj": {
2133
+ "group_size": {
2134
+ "4": 128
2135
+ },
2136
+ "bits": [
2137
+ 4
2138
+ ],
2139
+ "bits_prop": [
2140
+ 1.0
2141
+ ],
2142
+ "scale_bits": 4
2143
+ },
2144
+ "o_proj": {
2145
+ "group_size": {
2146
+ "4": 128,
2147
+ "2": 128
2148
+ },
2149
+ "bits": [
2150
+ 4,
2151
+ 2
2152
+ ],
2153
+ "bits_prop": [
2154
+ 0.25,
2155
+ 0.75
2156
+ ],
2157
+ "scale_bits": 4
2158
+ },
2159
+ "up_proj": {
2160
+ "group_size": {
2161
+ "4": 128,
2162
+ "2": 128
2163
+ },
2164
+ "bits": [
2165
+ 4,
2166
+ 2
2167
+ ],
2168
+ "bits_prop": [
2169
+ 0.25,
2170
+ 0.75
2171
+ ],
2172
+ "scale_bits": 4
2173
+ },
2174
+ "gate_proj": {
2175
+ "group_size": {
2176
+ "4": 128,
2177
+ "2": 128
2178
+ },
2179
+ "bits": [
2180
+ 4,
2181
+ 2
2182
+ ],
2183
+ "bits_prop": [
2184
+ 0.25,
2185
+ 0.75
2186
+ ],
2187
+ "scale_bits": 4
2188
+ },
2189
+ "down_proj": {
2190
+ "group_size": {
2191
+ "4": 128
2192
+ },
2193
+ "bits": [
2194
+ 4
2195
+ ],
2196
+ "bits_prop": [
2197
+ 1
2198
+ ],
2199
+ "scale_bits": 4
2200
+ }
2201
+ },
2202
+ "model.layers.21": {
2203
+ "accuracy": 0.9437572956085205,
2204
+ "total_bits": 646461696,
2205
+ "q_proj": {
2206
+ "group_size": {
2207
+ "4": 128,
2208
+ "2": 128
2209
+ },
2210
+ "bits": [
2211
+ 4,
2212
+ 2
2213
+ ],
2214
+ "bits_prop": [
2215
+ 0.25,
2216
+ 0.75
2217
+ ],
2218
+ "scale_bits": 4
2219
+ },
2220
+ "k_proj": {
2221
+ "group_size": {
2222
+ "4": 128,
2223
+ "2": 128
2224
+ },
2225
+ "bits": [
2226
+ 4,
2227
+ 2
2228
+ ],
2229
+ "bits_prop": [
2230
+ 0.25,
2231
+ 0.75
2232
+ ],
2233
+ "scale_bits": 4
2234
+ },
2235
+ "v_proj": {
2236
+ "group_size": {
2237
+ "4": 128
2238
+ },
2239
+ "bits": [
2240
+ 4
2241
+ ],
2242
+ "bits_prop": [
2243
+ 1.0
2244
+ ],
2245
+ "scale_bits": 4
2246
+ },
2247
+ "o_proj": {
2248
+ "group_size": {
2249
+ "4": 128,
2250
+ "2": 128
2251
+ },
2252
+ "bits": [
2253
+ 4,
2254
+ 2
2255
+ ],
2256
+ "bits_prop": [
2257
+ 0.25,
2258
+ 0.75
2259
+ ],
2260
+ "scale_bits": 4
2261
+ },
2262
+ "up_proj": {
2263
+ "group_size": {
2264
+ "4": 128,
2265
+ "2": 128
2266
+ },
2267
+ "bits": [
2268
+ 4,
2269
+ 2
2270
+ ],
2271
+ "bits_prop": [
2272
+ 0.25,
2273
+ 0.75
2274
+ ],
2275
+ "scale_bits": 4
2276
+ },
2277
+ "gate_proj": {
2278
+ "group_size": {
2279
+ "4": 128,
2280
+ "2": 128
2281
+ },
2282
+ "bits": [
2283
+ 4,
2284
+ 2
2285
+ ],
2286
+ "bits_prop": [
2287
+ 0.25,
2288
+ 0.75
2289
+ ],
2290
+ "scale_bits": 4
2291
+ },
2292
+ "down_proj": {
2293
+ "group_size": {
2294
+ "4": 128
2295
+ },
2296
+ "bits": [
2297
+ 4
2298
+ ],
2299
+ "bits_prop": [
2300
+ 1
2301
+ ],
2302
+ "scale_bits": 4
2303
+ }
2304
+ },
2305
+ "model.layers.22": {
2306
+ "accuracy": 0.9443583488464355,
2307
+ "total_bits": 646461696,
2308
+ "q_proj": {
2309
+ "group_size": {
2310
+ "4": 128,
2311
+ "2": 128
2312
+ },
2313
+ "bits": [
2314
+ 4,
2315
+ 2
2316
+ ],
2317
+ "bits_prop": [
2318
+ 0.25,
2319
+ 0.75
2320
+ ],
2321
+ "scale_bits": 4
2322
+ },
2323
+ "k_proj": {
2324
+ "group_size": {
2325
+ "4": 128,
2326
+ "2": 128
2327
+ },
2328
+ "bits": [
2329
+ 4,
2330
+ 2
2331
+ ],
2332
+ "bits_prop": [
2333
+ 0.25,
2334
+ 0.75
2335
+ ],
2336
+ "scale_bits": 4
2337
+ },
2338
+ "v_proj": {
2339
+ "group_size": {
2340
+ "4": 128
2341
+ },
2342
+ "bits": [
2343
+ 4
2344
+ ],
2345
+ "bits_prop": [
2346
+ 1.0
2347
+ ],
2348
+ "scale_bits": 4
2349
+ },
2350
+ "o_proj": {
2351
+ "group_size": {
2352
+ "4": 128,
2353
+ "2": 128
2354
+ },
2355
+ "bits": [
2356
+ 4,
2357
+ 2
2358
+ ],
2359
+ "bits_prop": [
2360
+ 0.25,
2361
+ 0.75
2362
+ ],
2363
+ "scale_bits": 4
2364
+ },
2365
+ "up_proj": {
2366
+ "group_size": {
2367
+ "4": 128,
2368
+ "2": 128
2369
+ },
2370
+ "bits": [
2371
+ 4,
2372
+ 2
2373
+ ],
2374
+ "bits_prop": [
2375
+ 0.25,
2376
+ 0.75
2377
+ ],
2378
+ "scale_bits": 4
2379
+ },
2380
+ "gate_proj": {
2381
+ "group_size": {
2382
+ "4": 128,
2383
+ "2": 128
2384
+ },
2385
+ "bits": [
2386
+ 4,
2387
+ 2
2388
+ ],
2389
+ "bits_prop": [
2390
+ 0.25,
2391
+ 0.75
2392
+ ],
2393
+ "scale_bits": 4
2394
+ },
2395
+ "down_proj": {
2396
+ "group_size": {
2397
+ "4": 128
2398
+ },
2399
+ "bits": [
2400
+ 4
2401
+ ],
2402
+ "bits_prop": [
2403
+ 1
2404
+ ],
2405
+ "scale_bits": 4
2406
+ }
2407
+ },
2408
+ "model.layers.23": {
2409
+ "accuracy": 0.9421937465667725,
2410
+ "total_bits": 646461696,
2411
+ "q_proj": {
2412
+ "group_size": {
2413
+ "4": 128,
2414
+ "2": 128
2415
+ },
2416
+ "bits": [
2417
+ 4,
2418
+ 2
2419
+ ],
2420
+ "bits_prop": [
2421
+ 0.25,
2422
+ 0.75
2423
+ ],
2424
+ "scale_bits": 4
2425
+ },
2426
+ "k_proj": {
2427
+ "group_size": {
2428
+ "4": 128,
2429
+ "2": 128
2430
+ },
2431
+ "bits": [
2432
+ 4,
2433
+ 2
2434
+ ],
2435
+ "bits_prop": [
2436
+ 0.25,
2437
+ 0.75
2438
+ ],
2439
+ "scale_bits": 4
2440
+ },
2441
+ "v_proj": {
2442
+ "group_size": {
2443
+ "4": 128
2444
+ },
2445
+ "bits": [
2446
+ 4
2447
+ ],
2448
+ "bits_prop": [
2449
+ 1.0
2450
+ ],
2451
+ "scale_bits": 4
2452
+ },
2453
+ "o_proj": {
2454
+ "group_size": {
2455
+ "4": 128,
2456
+ "2": 128
2457
+ },
2458
+ "bits": [
2459
+ 4,
2460
+ 2
2461
+ ],
2462
+ "bits_prop": [
2463
+ 0.25,
2464
+ 0.75
2465
+ ],
2466
+ "scale_bits": 4
2467
+ },
2468
+ "up_proj": {
2469
+ "group_size": {
2470
+ "4": 128,
2471
+ "2": 128
2472
+ },
2473
+ "bits": [
2474
+ 4,
2475
+ 2
2476
+ ],
2477
+ "bits_prop": [
2478
+ 0.25,
2479
+ 0.75
2480
+ ],
2481
+ "scale_bits": 4
2482
+ },
2483
+ "gate_proj": {
2484
+ "group_size": {
2485
+ "4": 128,
2486
+ "2": 128
2487
+ },
2488
+ "bits": [
2489
+ 4,
2490
+ 2
2491
+ ],
2492
+ "bits_prop": [
2493
+ 0.25,
2494
+ 0.75
2495
+ ],
2496
+ "scale_bits": 4
2497
+ },
2498
+ "down_proj": {
2499
+ "group_size": {
2500
+ "4": 128
2501
+ },
2502
+ "bits": [
2503
+ 4
2504
+ ],
2505
+ "bits_prop": [
2506
+ 1
2507
+ ],
2508
+ "scale_bits": 4
2509
+ }
2510
+ },
2511
+ "model.layers.24": {
2512
+ "accuracy": 0.9423227310180664,
2513
+ "total_bits": 646461696,
2514
+ "q_proj": {
2515
+ "group_size": {
2516
+ "4": 128,
2517
+ "2": 128
2518
+ },
2519
+ "bits": [
2520
+ 4,
2521
+ 2
2522
+ ],
2523
+ "bits_prop": [
2524
+ 0.25,
2525
+ 0.75
2526
+ ],
2527
+ "scale_bits": 4
2528
+ },
2529
+ "k_proj": {
2530
+ "group_size": {
2531
+ "4": 128,
2532
+ "2": 128
2533
+ },
2534
+ "bits": [
2535
+ 4,
2536
+ 2
2537
+ ],
2538
+ "bits_prop": [
2539
+ 0.25,
2540
+ 0.75
2541
+ ],
2542
+ "scale_bits": 4
2543
+ },
2544
+ "v_proj": {
2545
+ "group_size": {
2546
+ "4": 128
2547
+ },
2548
+ "bits": [
2549
+ 4
2550
+ ],
2551
+ "bits_prop": [
2552
+ 1.0
2553
+ ],
2554
+ "scale_bits": 4
2555
+ },
2556
+ "o_proj": {
2557
+ "group_size": {
2558
+ "4": 128,
2559
+ "2": 128
2560
+ },
2561
+ "bits": [
2562
+ 4,
2563
+ 2
2564
+ ],
2565
+ "bits_prop": [
2566
+ 0.25,
2567
+ 0.75
2568
+ ],
2569
+ "scale_bits": 4
2570
+ },
2571
+ "up_proj": {
2572
+ "group_size": {
2573
+ "4": 128,
2574
+ "2": 128
2575
+ },
2576
+ "bits": [
2577
+ 4,
2578
+ 2
2579
+ ],
2580
+ "bits_prop": [
2581
+ 0.25,
2582
+ 0.75
2583
+ ],
2584
+ "scale_bits": 4
2585
+ },
2586
+ "gate_proj": {
2587
+ "group_size": {
2588
+ "4": 128,
2589
+ "2": 128
2590
+ },
2591
+ "bits": [
2592
+ 4,
2593
+ 2
2594
+ ],
2595
+ "bits_prop": [
2596
+ 0.25,
2597
+ 0.75
2598
+ ],
2599
+ "scale_bits": 4
2600
+ },
2601
+ "down_proj": {
2602
+ "group_size": {
2603
+ "4": 128
2604
+ },
2605
+ "bits": [
2606
+ 4
2607
+ ],
2608
+ "bits_prop": [
2609
+ 1
2610
+ ],
2611
+ "scale_bits": 4
2612
+ }
2613
+ },
2614
+ "model.layers.25": {
2615
+ "accuracy": 0.9409959316253662,
2616
+ "total_bits": 646461696,
2617
+ "q_proj": {
2618
+ "group_size": {
2619
+ "4": 128,
2620
+ "2": 128
2621
+ },
2622
+ "bits": [
2623
+ 4,
2624
+ 2
2625
+ ],
2626
+ "bits_prop": [
2627
+ 0.25,
2628
+ 0.75
2629
+ ],
2630
+ "scale_bits": 4
2631
+ },
2632
+ "k_proj": {
2633
+ "group_size": {
2634
+ "4": 128,
2635
+ "2": 128
2636
+ },
2637
+ "bits": [
2638
+ 4,
2639
+ 2
2640
+ ],
2641
+ "bits_prop": [
2642
+ 0.25,
2643
+ 0.75
2644
+ ],
2645
+ "scale_bits": 4
2646
+ },
2647
+ "v_proj": {
2648
+ "group_size": {
2649
+ "4": 128
2650
+ },
2651
+ "bits": [
2652
+ 4
2653
+ ],
2654
+ "bits_prop": [
2655
+ 1.0
2656
+ ],
2657
+ "scale_bits": 4
2658
+ },
2659
+ "o_proj": {
2660
+ "group_size": {
2661
+ "4": 128,
2662
+ "2": 128
2663
+ },
2664
+ "bits": [
2665
+ 4,
2666
+ 2
2667
+ ],
2668
+ "bits_prop": [
2669
+ 0.25,
2670
+ 0.75
2671
+ ],
2672
+ "scale_bits": 4
2673
+ },
2674
+ "up_proj": {
2675
+ "group_size": {
2676
+ "4": 128,
2677
+ "2": 128
2678
+ },
2679
+ "bits": [
2680
+ 4,
2681
+ 2
2682
+ ],
2683
+ "bits_prop": [
2684
+ 0.25,
2685
+ 0.75
2686
+ ],
2687
+ "scale_bits": 4
2688
+ },
2689
+ "gate_proj": {
2690
+ "group_size": {
2691
+ "4": 128,
2692
+ "2": 128
2693
+ },
2694
+ "bits": [
2695
+ 4,
2696
+ 2
2697
+ ],
2698
+ "bits_prop": [
2699
+ 0.25,
2700
+ 0.75
2701
+ ],
2702
+ "scale_bits": 4
2703
+ },
2704
+ "down_proj": {
2705
+ "group_size": {
2706
+ "4": 128
2707
+ },
2708
+ "bits": [
2709
+ 4
2710
+ ],
2711
+ "bits_prop": [
2712
+ 1
2713
+ ],
2714
+ "scale_bits": 4
2715
+ }
2716
+ },
2717
+ "model.layers.26": {
2718
+ "accuracy": 0.951786994934082,
2719
+ "total_bits": 724056320,
2720
+ "q_proj": {
2721
+ "group_size": {
2722
+ "4": 128,
2723
+ "2": 128
2724
+ },
2725
+ "bits": [
2726
+ 4,
2727
+ 2
2728
+ ],
2729
+ "bits_prop": [
2730
+ 0.5,
2731
+ 0.5
2732
+ ],
2733
+ "scale_bits": 4
2734
+ },
2735
+ "k_proj": {
2736
+ "group_size": {
2737
+ "4": 128,
2738
+ "2": 128
2739
+ },
2740
+ "bits": [
2741
+ 4,
2742
+ 2
2743
+ ],
2744
+ "bits_prop": [
2745
+ 0.5,
2746
+ 0.5
2747
+ ],
2748
+ "scale_bits": 4
2749
+ },
2750
+ "v_proj": {
2751
+ "group_size": {
2752
+ "4": 128
2753
+ },
2754
+ "bits": [
2755
+ 4
2756
+ ],
2757
+ "bits_prop": [
2758
+ 1.0
2759
+ ],
2760
+ "scale_bits": 4
2761
+ },
2762
+ "o_proj": {
2763
+ "group_size": {
2764
+ "4": 128,
2765
+ "2": 128
2766
+ },
2767
+ "bits": [
2768
+ 4,
2769
+ 2
2770
+ ],
2771
+ "bits_prop": [
2772
+ 0.5,
2773
+ 0.5
2774
+ ],
2775
+ "scale_bits": 4
2776
+ },
2777
+ "up_proj": {
2778
+ "group_size": {
2779
+ "4": 128,
2780
+ "2": 128
2781
+ },
2782
+ "bits": [
2783
+ 4,
2784
+ 2
2785
+ ],
2786
+ "bits_prop": [
2787
+ 0.5,
2788
+ 0.5
2789
+ ],
2790
+ "scale_bits": 4
2791
+ },
2792
+ "gate_proj": {
2793
+ "group_size": {
2794
+ "4": 128,
2795
+ "2": 128
2796
+ },
2797
+ "bits": [
2798
+ 4,
2799
+ 2
2800
+ ],
2801
+ "bits_prop": [
2802
+ 0.5,
2803
+ 0.5
2804
+ ],
2805
+ "scale_bits": 4
2806
+ },
2807
+ "down_proj": {
2808
+ "group_size": {
2809
+ "4": 128
2810
+ },
2811
+ "bits": [
2812
+ 4
2813
+ ],
2814
+ "bits_prop": [
2815
+ 1.0
2816
+ ],
2817
+ "scale_bits": 4
2818
+ }
2819
+ },
2820
+ "model.layers.27": {
2821
+ "accuracy": 0.9403626918792725,
2822
+ "total_bits": 646461696,
2823
+ "q_proj": {
2824
+ "group_size": {
2825
+ "4": 128,
2826
+ "2": 128
2827
+ },
2828
+ "bits": [
2829
+ 4,
2830
+ 2
2831
+ ],
2832
+ "bits_prop": [
2833
+ 0.25,
2834
+ 0.75
2835
+ ],
2836
+ "scale_bits": 4
2837
+ },
2838
+ "k_proj": {
2839
+ "group_size": {
2840
+ "4": 128,
2841
+ "2": 128
2842
+ },
2843
+ "bits": [
2844
+ 4,
2845
+ 2
2846
+ ],
2847
+ "bits_prop": [
2848
+ 0.25,
2849
+ 0.75
2850
+ ],
2851
+ "scale_bits": 4
2852
+ },
2853
+ "v_proj": {
2854
+ "group_size": {
2855
+ "4": 128
2856
+ },
2857
+ "bits": [
2858
+ 4
2859
+ ],
2860
+ "bits_prop": [
2861
+ 1.0
2862
+ ],
2863
+ "scale_bits": 4
2864
+ },
2865
+ "o_proj": {
2866
+ "group_size": {
2867
+ "4": 128,
2868
+ "2": 128
2869
+ },
2870
+ "bits": [
2871
+ 4,
2872
+ 2
2873
+ ],
2874
+ "bits_prop": [
2875
+ 0.25,
2876
+ 0.75
2877
+ ],
2878
+ "scale_bits": 4
2879
+ },
2880
+ "up_proj": {
2881
+ "group_size": {
2882
+ "4": 128,
2883
+ "2": 128
2884
+ },
2885
+ "bits": [
2886
+ 4,
2887
+ 2
2888
+ ],
2889
+ "bits_prop": [
2890
+ 0.25,
2891
+ 0.75
2892
+ ],
2893
+ "scale_bits": 4
2894
+ },
2895
+ "gate_proj": {
2896
+ "group_size": {
2897
+ "4": 128,
2898
+ "2": 128
2899
+ },
2900
+ "bits": [
2901
+ 4,
2902
+ 2
2903
+ ],
2904
+ "bits_prop": [
2905
+ 0.25,
2906
+ 0.75
2907
+ ],
2908
+ "scale_bits": 4
2909
+ },
2910
+ "down_proj": {
2911
+ "group_size": {
2912
+ "4": 128
2913
+ },
2914
+ "bits": [
2915
+ 4
2916
+ ],
2917
+ "bits_prop": [
2918
+ 1
2919
+ ],
2920
+ "scale_bits": 4
2921
+ }
2922
+ },
2923
+ "model.layers.28": {
2924
+ "accuracy": 0.9786742925643921,
2925
+ "total_bits": 879245568,
2926
+ "q_proj": {
2927
+ "group_size": {
2928
+ "4": 128
2929
+ },
2930
+ "bits": [
2931
+ 4
2932
+ ],
2933
+ "bits_prop": [
2934
+ 1
2935
+ ],
2936
+ "scale_bits": 4
2937
+ },
2938
+ "k_proj": {
2939
+ "group_size": {
2940
+ "4": 128
2941
+ },
2942
+ "bits": [
2943
+ 4
2944
+ ],
2945
+ "bits_prop": [
2946
+ 1
2947
+ ],
2948
+ "scale_bits": 4
2949
+ },
2950
+ "v_proj": {
2951
+ "group_size": {
2952
+ "4": 128
2953
+ },
2954
+ "bits": [
2955
+ 4
2956
+ ],
2957
+ "bits_prop": [
2958
+ 1
2959
+ ],
2960
+ "scale_bits": 4
2961
+ },
2962
+ "o_proj": {
2963
+ "group_size": {
2964
+ "4": 128
2965
+ },
2966
+ "bits": [
2967
+ 4
2968
+ ],
2969
+ "bits_prop": [
2970
+ 1
2971
+ ],
2972
+ "scale_bits": 4
2973
+ },
2974
+ "up_proj": {
2975
+ "group_size": {
2976
+ "4": 128
2977
+ },
2978
+ "bits": [
2979
+ 4
2980
+ ],
2981
+ "bits_prop": [
2982
+ 1
2983
+ ],
2984
+ "scale_bits": 4
2985
+ },
2986
+ "gate_proj": {
2987
+ "group_size": {
2988
+ "4": 128
2989
+ },
2990
+ "bits": [
2991
+ 4
2992
+ ],
2993
+ "bits_prop": [
2994
+ 1
2995
+ ],
2996
+ "scale_bits": 4
2997
+ },
2998
+ "down_proj": {
2999
+ "group_size": {
3000
+ "4": 128
3001
+ },
3002
+ "bits": [
3003
+ 4
3004
+ ],
3005
+ "bits_prop": [
3006
+ 1
3007
+ ],
3008
+ "scale_bits": 4
3009
+ }
3010
+ },
3011
+ "model.layers.29": {
3012
+ "accuracy": 0.9775421619415283,
3013
+ "total_bits": 879245568,
3014
+ "q_proj": {
3015
+ "group_size": {
3016
+ "4": 128
3017
+ },
3018
+ "bits": [
3019
+ 4
3020
+ ],
3021
+ "bits_prop": [
3022
+ 1
3023
+ ],
3024
+ "scale_bits": 4
3025
+ },
3026
+ "k_proj": {
3027
+ "group_size": {
3028
+ "4": 128
3029
+ },
3030
+ "bits": [
3031
+ 4
3032
+ ],
3033
+ "bits_prop": [
3034
+ 1
3035
+ ],
3036
+ "scale_bits": 4
3037
+ },
3038
+ "v_proj": {
3039
+ "group_size": {
3040
+ "4": 128
3041
+ },
3042
+ "bits": [
3043
+ 4
3044
+ ],
3045
+ "bits_prop": [
3046
+ 1
3047
+ ],
3048
+ "scale_bits": 4
3049
+ },
3050
+ "o_proj": {
3051
+ "group_size": {
3052
+ "4": 128
3053
+ },
3054
+ "bits": [
3055
+ 4
3056
+ ],
3057
+ "bits_prop": [
3058
+ 1
3059
+ ],
3060
+ "scale_bits": 4
3061
+ },
3062
+ "up_proj": {
3063
+ "group_size": {
3064
+ "4": 128
3065
+ },
3066
+ "bits": [
3067
+ 4
3068
+ ],
3069
+ "bits_prop": [
3070
+ 1
3071
+ ],
3072
+ "scale_bits": 4
3073
+ },
3074
+ "gate_proj": {
3075
+ "group_size": {
3076
+ "4": 128
3077
+ },
3078
+ "bits": [
3079
+ 4
3080
+ ],
3081
+ "bits_prop": [
3082
+ 1
3083
+ ],
3084
+ "scale_bits": 4
3085
+ },
3086
+ "down_proj": {
3087
+ "group_size": {
3088
+ "4": 128
3089
+ },
3090
+ "bits": [
3091
+ 4
3092
+ ],
3093
+ "bits_prop": [
3094
+ 1
3095
+ ],
3096
+ "scale_bits": 4
3097
+ }
3098
+ },
3099
+ "model.layers.30": {
3100
+ "accuracy": 0.9752198457717896,
3101
+ "total_bits": 879245568,
3102
+ "q_proj": {
3103
+ "group_size": {
3104
+ "4": 128
3105
+ },
3106
+ "bits": [
3107
+ 4
3108
+ ],
3109
+ "bits_prop": [
3110
+ 1
3111
+ ],
3112
+ "scale_bits": 4
3113
+ },
3114
+ "k_proj": {
3115
+ "group_size": {
3116
+ "4": 128
3117
+ },
3118
+ "bits": [
3119
+ 4
3120
+ ],
3121
+ "bits_prop": [
3122
+ 1
3123
+ ],
3124
+ "scale_bits": 4
3125
+ },
3126
+ "v_proj": {
3127
+ "group_size": {
3128
+ "4": 128
3129
+ },
3130
+ "bits": [
3131
+ 4
3132
+ ],
3133
+ "bits_prop": [
3134
+ 1
3135
+ ],
3136
+ "scale_bits": 4
3137
+ },
3138
+ "o_proj": {
3139
+ "group_size": {
3140
+ "4": 128
3141
+ },
3142
+ "bits": [
3143
+ 4
3144
+ ],
3145
+ "bits_prop": [
3146
+ 1
3147
+ ],
3148
+ "scale_bits": 4
3149
+ },
3150
+ "up_proj": {
3151
+ "group_size": {
3152
+ "4": 128
3153
+ },
3154
+ "bits": [
3155
+ 4
3156
+ ],
3157
+ "bits_prop": [
3158
+ 1
3159
+ ],
3160
+ "scale_bits": 4
3161
+ },
3162
+ "gate_proj": {
3163
+ "group_size": {
3164
+ "4": 128
3165
+ },
3166
+ "bits": [
3167
+ 4
3168
+ ],
3169
+ "bits_prop": [
3170
+ 1
3171
+ ],
3172
+ "scale_bits": 4
3173
+ },
3174
+ "down_proj": {
3175
+ "group_size": {
3176
+ "4": 128
3177
+ },
3178
+ "bits": [
3179
+ 4
3180
+ ],
3181
+ "bits_prop": [
3182
+ 1
3183
+ ],
3184
+ "scale_bits": 4
3185
+ }
3186
+ },
3187
+ "model.layers.31": {
3188
+ "accuracy": 0.9711686372756958,
3189
+ "total_bits": 879245568,
3190
+ "q_proj": {
3191
+ "group_size": {
3192
+ "4": 128
3193
+ },
3194
+ "bits": [
3195
+ 4
3196
+ ],
3197
+ "bits_prop": [
3198
+ 1
3199
+ ],
3200
+ "scale_bits": 4
3201
+ },
3202
+ "k_proj": {
3203
+ "group_size": {
3204
+ "4": 128
3205
+ },
3206
+ "bits": [
3207
+ 4
3208
+ ],
3209
+ "bits_prop": [
3210
+ 1
3211
+ ],
3212
+ "scale_bits": 4
3213
+ },
3214
+ "v_proj": {
3215
+ "group_size": {
3216
+ "4": 128
3217
+ },
3218
+ "bits": [
3219
+ 4
3220
+ ],
3221
+ "bits_prop": [
3222
+ 1
3223
+ ],
3224
+ "scale_bits": 4
3225
+ },
3226
+ "o_proj": {
3227
+ "group_size": {
3228
+ "4": 128
3229
+ },
3230
+ "bits": [
3231
+ 4
3232
+ ],
3233
+ "bits_prop": [
3234
+ 1
3235
+ ],
3236
+ "scale_bits": 4
3237
+ },
3238
+ "up_proj": {
3239
+ "group_size": {
3240
+ "4": 128
3241
+ },
3242
+ "bits": [
3243
+ 4
3244
+ ],
3245
+ "bits_prop": [
3246
+ 1
3247
+ ],
3248
+ "scale_bits": 4
3249
+ },
3250
+ "gate_proj": {
3251
+ "group_size": {
3252
+ "4": 128
3253
+ },
3254
+ "bits": [
3255
+ 4
3256
+ ],
3257
+ "bits_prop": [
3258
+ 1
3259
+ ],
3260
+ "scale_bits": 4
3261
+ },
3262
+ "down_proj": {
3263
+ "group_size": {
3264
+ "4": 128
3265
+ },
3266
+ "bits": [
3267
+ 4
3268
+ ],
3269
+ "bits_prop": [
3270
+ 1
3271
+ ],
3272
+ "scale_bits": 4
3273
+ }
3274
+ }
3275
+ }
3276
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
3
+ size 493443
tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": true,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<unk>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<s>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "2": {
23
+ "content": "</s>",
24
+ "lstrip": false,
25
+ "normalized": false,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": true
29
+ }
30
+ },
31
+ "additional_special_tokens": [],
32
+ "bos_token": "<s>",
33
+ "clean_up_tokenization_spaces": false,
34
+ "eos_token": "</s>",
35
+ "legacy": true,
36
+ "model_max_length": 1000000000000000019884624838656,
37
+ "pad_token": null,
38
+ "sp_model_kwargs": {},
39
+ "spaces_between_special_tokens": false,
40
+ "tokenizer_class": "LlamaTokenizer",
41
+ "unk_token": "<unk>",
42
+ "use_default_system_prompt": false
43
+ }