InternLM-Math commited on
Commit
cb5be48
1 Parent(s): e11b9ea

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: text-generation
3
+ license: other
4
+ language:
5
+ - en
6
+ - zh
7
+ tags:
8
+ - math
9
+ ---
10
+
11
+ # InternLM-Math-Plus
12
+
13
+ <div align="center">
14
+
15
+ <img src="https://raw.githubusercontent.com/InternLM/InternLM/main/assets/logo.svg" width="200"/>
16
+ <div> </div>
17
+ <div align="center">
18
+ <b><font size="5">InternLM-Math</font></b>
19
+ <sup>
20
+ <a href="https://internlm.intern-ai.org.cn/">
21
+ <i><font size="4">Plus</font></i>
22
+ </a>
23
+ </sup>
24
+ <div> </div>
25
+ </div>
26
+
27
+ State-of-the-art bilingual open-sourced Math reasoning LLMs.
28
+ A **solver**, **prover**, **verifier**, **augmentor**.
29
+
30
+ [💻 Github](https://github.com/InternLM/InternLM-Math) [🤗 Demo](https://huggingface.co/spaces/internlm/internlm2-math-7b)
31
+ </div>
32
+
33
+ # News
34
+ - [2024.05.24] We release updated version InternLM2-Math-Plus with 4 sizes and state-of-the-art performances including 1.8B, 7B, 20B, and 8x22B. We improve informal math reasoning performance (chain-of-thought and code-intepreter) and formal math reasoning performance (LEAN 4 translation and LEAN 4 theorem proving) significantly.
35
+ - [2024.02.10] We add tech reports and citation reference.
36
+ - [2024.01.31] We add MiniF2F results with evaluation codes!
37
+ - [2024.01.29] We add checkpoints from ModelScope. Update results about majority voting and Code Intepreter. Tech report is on the way!
38
+ - [2024.01.26] We add checkpoints from OpenXLab, which ease Chinese users to download!
39
+
40
+ # Performance
41
+
42
+ ## Formal Math Reasoning
43
+ We evaluate the performance of InternLM2-Math-Plus on formal math reasoning benchmark MiniF2F-test. The evaluation setting is same as Llemma with LEAN 4.
44
+ | Models | MiniF2F-test |
45
+ | -------------------------------- | ------------ |
46
+ | ReProver | 26.5 |
47
+ | LLMStep | 27.9 |
48
+ | GPT-F | 36.6 |
49
+ | HTPS | 41.0 |
50
+ | Llemma-7B | 26.2 |
51
+ | Llemma-34B | 25.8 |
52
+ | InternLM2-Math-7B-Base | 30.3 |
53
+ | InternLM2-Math-20B-Base | 29.5 |
54
+ | InternLM2-Math-Plus-1.8B | 38.9 |
55
+ | InternLM2-Math-Plus-7B | **43.4** |
56
+ | InternLM2-Math-Plus-20B | 42.6 |
57
+ | InternLM2-Math-Plus-Mixtral8x22B | 37.3 |
58
+
59
+ ## Informal Math Reasoning
60
+ We evaluate the performance of InternLM2-Math-Plus on informal math reasoning benchmark MATH and GSM8K. InternLM2-Math-Plus-1.8B outperforms MiniCPM-2B in the smallest size setting. InternLM2-Math-Plus-7B outperforms Deepseek-Math-7B-RL which is the state-of-the-art math reasoning open source model. InternLM2-Math-Plus-Mixtral8x22B achieves 68.5 on MATH (with Python) and 91.8 on GSM8K.
61
+ | Model | MATH | MATH-Python | GSM8K |
62
+ | -------------------------------- | -------- | ----------- | -------- |
63
+ | MiniCPM-2B | 10.2 | - | 53.8 |
64
+ | InternLM2-Math-Plus-1.8B | **37.0** | **41.5** | **58.8** |
65
+ | InternLM2-Math-7B | 34.6 | 50.9 | 78.1 |
66
+ | Deepseek-Math-7B-RL | 51.7 | 58.8 | **88.2** |
67
+ | InternLM2-Math-Plus-7B | **53.0** | **59.7** | 85.8 |
68
+ | InternLM2-Math-20B | 37.7 | 54.3 | 82.6 |
69
+ | InternLM2-Math-Plus-20B | **53.8** | **61.8** | **87.7** |
70
+ | Mixtral8x22B-Instruct-v0.1 | 41.8 | - | 78.6 |
71
+ | Eurux-8x22B-NCA | 49.0 | - | - |
72
+ | InternLM2-Math-Plus-Mixtral8x22B | **58.1** | **68.5** | **91.8** |
73
+
74
+ We also evaluate models on [MathBench-A](https://github.com/open-compass/MathBench). InternLM2-Math-Plus-Mixtral8x22B has comparable performance compared to Claude 3 Opus.
75
+ | Model | Arithmetic | Primary | Middle | High | College | Average |
76
+ | -------------------------------- | ---------- | ------- | ------ | ---- | ------- | ------- |
77
+ | GPT-4o-0513 | 77.7 | 87.7 | 76.3 | 59.0 | 54.0 | 70.9 |
78
+ | Claude 3 Opus | 85.7 | 85.0 | 58.0 | 42.7 | 43.7 | 63.0 |
79
+ | Qwen-Max-0428 | 72.3 | 86.3 | 65.0 | 45.0 | 27.3 | 59.2 |
80
+ | Qwen-1.5-110B | 70.3 | 82.3 | 64.0 | 47.3 | 28.0 | 58.4 |
81
+ | Deepseek-V2 | 82.7 | 89.3 | 59.0 | 39.3 | 29.3 | 59.9 |
82
+ | Llama-3-70B-Instruct | 70.3 | 86.0 | 53.0 | 38.7 | 34.7 | 56.5 |
83
+ | InternLM2-Math-Plus-Mixtral8x22B | 77.5 | 82.0 | 63.6 | 50.3 | 36.8 | 62.0 |
84
+ | InternLM2-Math-20B | 58.7 | 70.0 | 43.7 | 24.7 | 12.7 | 42.0 |
85
+ | InternLM2-Math-Plus-20B | 65.8 | 79.7 | 59.5 | 47.6 | 24.8 | 55.5 |
86
+ | Llama3-8B-Instruct | 54.7 | 71.0 | 25.0 | 19.0 | 14.0 | 36.7 |
87
+ | InternLM2-Math-7B | 53.7 | 67.0 | 41.3 | 18.3 | 8.0 | 37.7 |
88
+ | Deepseek-Math-7B-RL | 68.0 | 83.3 | 44.3 | 33.0 | 23.0 | 50.3 |
89
+ | InternLM2-Math-Plus-7B | 61.4 | 78.3 | 52.5 | 40.5 | 21.7 | 50.9 |
90
+ | MiniCPM-2B | 49.3 | 51.7 | 18.0 | 8.7 | 3.7 | 26.3 |
91
+ | InternLM2-Math-Plus-1.8B | 43.0 | 43.3 | 25.4 | 18.9 | 4.7 | 27.1 |
92
+
93
+ # Citation and Tech Report
94
+ ```
95
+ @misc{ying2024internlmmath,
96
+ title={InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning},
97
+ author={Huaiyuan Ying and Shuo Zhang and Linyang Li and Zhejian Zhou and Yunfan Shao and Zhaoye Fei and Yichuan Ma and Jiawei Hong and Kuikun Liu and Ziyi Wang and Yudong Wang and Zijian Wu and Shuaibin Li and Fengzhe Zhou and Hongwei Liu and Songyang Zhang and Wenwei Zhang and Hang Yan and Xipeng Qiu and Jiayu Wang and Kai Chen and Dahua Lin},
98
+ year={2024},
99
+ eprint={2402.06332},
100
+ archivePrefix={arXiv},
101
+ primaryClass={cs.CL}
102
+ }
103
+ ```
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "InternLM2ForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_internlm.InternLMConfig",
7
+ "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM",
8
+ "AutoModel": "modeling_internlm2.InternLM2ForCausalLM"
9
+ },
10
+ "bias": false,
11
+ "bos_token_id": 1,
12
+ "eos_token_id": 2,
13
+ "hidden_act": "silu",
14
+ "hidden_size": 6144,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 16384,
17
+ "max_position_embeddings": 8192,
18
+ "model_type": "internlm",
19
+ "num_attention_heads": 48,
20
+ "num_hidden_layers": 48,
21
+ "num_key_value_heads": 8,
22
+ "pad_token_id": 2,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_scaling": {
25
+ "factor": 1.0,
26
+ "type": "dynamic"
27
+ },
28
+ "rope_theta": 1000000,
29
+ "tie_word_embeddings": false,
30
+ "torch_dtype": "bfloat16",
31
+ "transformers_version": "4.40.2",
32
+ "use_cache": true,
33
+ "vocab_size": 92544
34
+ }
configuration_internlm.py ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright (c) InternLM. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ InternLM model configuration"""
21
+
22
+ from transformers.configuration_utils import PretrainedConfig
23
+ from transformers.utils import logging
24
+
25
+ logger = logging.get_logger(__name__)
26
+
27
+ INTERNLM_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
28
+
29
+
30
+ class InternLMConfig(PretrainedConfig):
31
+ r"""
32
+ This is the configuration class to store the configuration of a [`InternLMModel`]. It is used to instantiate
33
+ an InternLM model according to the specified arguments, defining the model architecture. Instantiating a
34
+ configuration with the defaults will yield a similar configuration to that of the InternLM-7B.
35
+
36
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
37
+ documentation from [`PretrainedConfig`] for more information.
38
+
39
+
40
+ Args:
41
+ vocab_size (`int`, *optional*, defaults to 32000):
42
+ Vocabulary size of the InternLM model. Defines the number of different tokens that can be represented by the
43
+ `inputs_ids` passed when calling [`InternLMModel`]
44
+ hidden_size (`int`, *optional*, defaults to 4096):
45
+ Dimension of the hidden representations.
46
+ intermediate_size (`int`, *optional*, defaults to 11008):
47
+ Dimension of the MLP representations.
48
+ num_hidden_layers (`int`, *optional*, defaults to 32):
49
+ Number of hidden layers in the Transformer encoder.
50
+ num_attention_heads (`int`, *optional*, defaults to 32):
51
+ Number of attention heads for each attention layer in the Transformer encoder.
52
+ num_key_value_heads (`int`, *optional*):
53
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
54
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
55
+ `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
56
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
57
+ by meanpooling all the original heads within that group. For more details checkout [this
58
+ paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
59
+ `num_attention_heads`.
60
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
61
+ The non-linear activation function (function or string) in the decoder.
62
+ max_position_embeddings (`int`, *optional*, defaults to 2048):
63
+ The maximum sequence length that this model might ever be used with. Typically set this to something large
64
+ just in case (e.g., 512 or 1024 or 2048).
65
+ initializer_range (`float`, *optional*, defaults to 0.02):
66
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
67
+ rms_norm_eps (`float`, *optional*, defaults to 1e-12):
68
+ The epsilon used by the rms normalization layers.
69
+ use_cache (`bool`, *optional*, defaults to `True`):
70
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
71
+ relevant if `config.is_decoder=True`.
72
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
73
+ Whether to tie weight embeddings
74
+ Example:
75
+
76
+ ```python
77
+ >>> from transformers import InternLMModel, InternLMConfig
78
+
79
+ >>> # Initializing a InternLM internlm-7b style configuration
80
+ >>> configuration = InternLMConfig()
81
+
82
+ >>> # Initializing a model from the internlm-7b style configuration
83
+ >>> model = InternLMModel(configuration)
84
+
85
+ >>> # Accessing the model configuration
86
+ >>> configuration = model.config
87
+ ```"""
88
+ model_type = "internlm"
89
+ _auto_class = "AutoConfig"
90
+
91
+ def __init__( # pylint: disable=W0102
92
+ self,
93
+ vocab_size=103168,
94
+ hidden_size=4096,
95
+ intermediate_size=11008,
96
+ num_hidden_layers=32,
97
+ num_attention_heads=32,
98
+ num_key_value_heads=None,
99
+ hidden_act="silu",
100
+ max_position_embeddings=2048,
101
+ initializer_range=0.02,
102
+ rms_norm_eps=1e-6,
103
+ use_cache=True,
104
+ pad_token_id=0,
105
+ bos_token_id=1,
106
+ eos_token_id=2,
107
+ tie_word_embeddings=False,
108
+ bias=True,
109
+ rope_theta=10000,
110
+ rope_scaling=None,
111
+ **kwargs,
112
+ ):
113
+ self.vocab_size = vocab_size
114
+ self.max_position_embeddings = max_position_embeddings
115
+ self.hidden_size = hidden_size
116
+ self.intermediate_size = intermediate_size
117
+ self.num_hidden_layers = num_hidden_layers
118
+ self.num_attention_heads = num_attention_heads
119
+ self.bias = bias
120
+
121
+ if num_key_value_heads is None:
122
+ num_key_value_heads = num_attention_heads
123
+ self.num_key_value_heads = num_key_value_heads
124
+
125
+ self.hidden_act = hidden_act
126
+ self.initializer_range = initializer_range
127
+ self.rms_norm_eps = rms_norm_eps
128
+ self.use_cache = use_cache
129
+ self.rope_theta = rope_theta
130
+ self.rope_scaling = rope_scaling
131
+ self._rope_scaling_validation()
132
+ super().__init__(
133
+ pad_token_id=pad_token_id,
134
+ bos_token_id=bos_token_id,
135
+ eos_token_id=eos_token_id,
136
+ tie_word_embeddings=tie_word_embeddings,
137
+ **kwargs,
138
+ )
139
+
140
+ def _rope_scaling_validation(self):
141
+ """
142
+ Validate the `rope_scaling` configuration.
143
+ """
144
+ if self.rope_scaling is None:
145
+ return
146
+
147
+ if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
148
+ raise ValueError(
149
+ "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
150
+ f"got {self.rope_scaling}"
151
+ )
152
+ rope_scaling_type = self.rope_scaling.get("type", None)
153
+ rope_scaling_factor = self.rope_scaling.get("factor", None)
154
+ if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
155
+ raise ValueError(
156
+ f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
157
+ )
158
+ if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor < 1.0:
159
+ raise ValueError(f"`rope_scaling`'s factor field must be a float >= 1, got {rope_scaling_factor}")
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 2,
6
+ "transformers_version": "4.40.2"
7
+ }
model-00001-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2db005c496885651e904a96ca3d0da9f4090372f98ceaa585be165bc0d3eda60
3
+ size 3855166232
model-00002-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:409cfe4bfd4fbb0088e4a07ef8eb24b60fb40e34c6e053549b61849ffa541e3c
3
+ size 3900829576
model-00003-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:125177ad28c578f04288ab4b36375c04dd821eb4d5dba1d8d9f56bd4fa3c88bf
3
+ size 3900829600
model-00004-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:543902dcea75dcd54f1879bd0c03ec4b50e83a1d4821bac38a40e17920edfe20
3
+ size 3900829608
model-00005-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf6b883a27fac5bf5a20f43fe44989b6daf3c9d685c3588db0c0613f4aaed7d7
3
+ size 3900829608
model-00006-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b86c3a23f2592dba0dc146537ba53b4b664e4bd9cee2a2843b98cc0500b2c1c4
3
+ size 3900829608
model-00007-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72aecd32ed877a254fb4049c25f6051c99f11f95078af4edb5d19d98c493b18d
3
+ size 3900829608
model-00008-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f2008137193430383579d565c0bec4d3d64483029a4887ff4981c07462d8cc20
3
+ size 3900829608
model-00009-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d5c526007b78b019766ebcfbb1a79538396d75ca5df3a167f598976306734b3e
3
+ size 3900829608
model-00010-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c249fc0057bb11245f792f657864463c3dc4000f935d00332eab51108aae70d2
3
+ size 3523354280
model-00011-of-00011.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:132de72fefca52f439ea5101a1d2d984685a3edcbf5bb2c032063473ab7c0dbd
3
+ size 1137180800
model.safetensors.index.json ADDED
@@ -0,0 +1,346 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 39722299392
4
+ },
5
+ "weight_map": {
6
+ "model.layers.0.attention.wo.weight": "model-00001-of-00011.safetensors",
7
+ "model.layers.0.attention.wqkv.weight": "model-00001-of-00011.safetensors",
8
+ "model.layers.0.attention_norm.weight": "model-00001-of-00011.safetensors",
9
+ "model.layers.0.feed_forward.w1.weight": "model-00001-of-00011.safetensors",
10
+ "model.layers.0.feed_forward.w2.weight": "model-00001-of-00011.safetensors",
11
+ "model.layers.0.feed_forward.w3.weight": "model-00001-of-00011.safetensors",
12
+ "model.layers.0.ffn_norm.weight": "model-00001-of-00011.safetensors",
13
+ "model.layers.1.attention.wo.weight": "model-00001-of-00011.safetensors",
14
+ "model.layers.1.attention.wqkv.weight": "model-00001-of-00011.safetensors",
15
+ "model.layers.1.attention_norm.weight": "model-00001-of-00011.safetensors",
16
+ "model.layers.1.feed_forward.w1.weight": "model-00001-of-00011.safetensors",
17
+ "model.layers.1.feed_forward.w2.weight": "model-00001-of-00011.safetensors",
18
+ "model.layers.1.feed_forward.w3.weight": "model-00001-of-00011.safetensors",
19
+ "model.layers.1.ffn_norm.weight": "model-00001-of-00011.safetensors",
20
+ "model.layers.10.attention.wo.weight": "model-00003-of-00011.safetensors",
21
+ "model.layers.10.attention.wqkv.weight": "model-00003-of-00011.safetensors",
22
+ "model.layers.10.attention_norm.weight": "model-00003-of-00011.safetensors",
23
+ "model.layers.10.feed_forward.w1.weight": "model-00003-of-00011.safetensors",
24
+ "model.layers.10.feed_forward.w2.weight": "model-00003-of-00011.safetensors",
25
+ "model.layers.10.feed_forward.w3.weight": "model-00003-of-00011.safetensors",
26
+ "model.layers.10.ffn_norm.weight": "model-00003-of-00011.safetensors",
27
+ "model.layers.11.attention.wo.weight": "model-00003-of-00011.safetensors",
28
+ "model.layers.11.attention.wqkv.weight": "model-00003-of-00011.safetensors",
29
+ "model.layers.11.attention_norm.weight": "model-00003-of-00011.safetensors",
30
+ "model.layers.11.feed_forward.w1.weight": "model-00003-of-00011.safetensors",
31
+ "model.layers.11.feed_forward.w2.weight": "model-00003-of-00011.safetensors",
32
+ "model.layers.11.feed_forward.w3.weight": "model-00003-of-00011.safetensors",
33
+ "model.layers.11.ffn_norm.weight": "model-00003-of-00011.safetensors",
34
+ "model.layers.12.attention.wo.weight": "model-00003-of-00011.safetensors",
35
+ "model.layers.12.attention.wqkv.weight": "model-00003-of-00011.safetensors",
36
+ "model.layers.12.attention_norm.weight": "model-00003-of-00011.safetensors",
37
+ "model.layers.12.feed_forward.w1.weight": "model-00003-of-00011.safetensors",
38
+ "model.layers.12.feed_forward.w2.weight": "model-00003-of-00011.safetensors",
39
+ "model.layers.12.feed_forward.w3.weight": "model-00003-of-00011.safetensors",
40
+ "model.layers.12.ffn_norm.weight": "model-00003-of-00011.safetensors",
41
+ "model.layers.13.attention.wo.weight": "model-00003-of-00011.safetensors",
42
+ "model.layers.13.attention.wqkv.weight": "model-00003-of-00011.safetensors",
43
+ "model.layers.13.attention_norm.weight": "model-00004-of-00011.safetensors",
44
+ "model.layers.13.feed_forward.w1.weight": "model-00003-of-00011.safetensors",
45
+ "model.layers.13.feed_forward.w2.weight": "model-00004-of-00011.safetensors",
46
+ "model.layers.13.feed_forward.w3.weight": "model-00004-of-00011.safetensors",
47
+ "model.layers.13.ffn_norm.weight": "model-00004-of-00011.safetensors",
48
+ "model.layers.14.attention.wo.weight": "model-00004-of-00011.safetensors",
49
+ "model.layers.14.attention.wqkv.weight": "model-00004-of-00011.safetensors",
50
+ "model.layers.14.attention_norm.weight": "model-00004-of-00011.safetensors",
51
+ "model.layers.14.feed_forward.w1.weight": "model-00004-of-00011.safetensors",
52
+ "model.layers.14.feed_forward.w2.weight": "model-00004-of-00011.safetensors",
53
+ "model.layers.14.feed_forward.w3.weight": "model-00004-of-00011.safetensors",
54
+ "model.layers.14.ffn_norm.weight": "model-00004-of-00011.safetensors",
55
+ "model.layers.15.attention.wo.weight": "model-00004-of-00011.safetensors",
56
+ "model.layers.15.attention.wqkv.weight": "model-00004-of-00011.safetensors",
57
+ "model.layers.15.attention_norm.weight": "model-00004-of-00011.safetensors",
58
+ "model.layers.15.feed_forward.w1.weight": "model-00004-of-00011.safetensors",
59
+ "model.layers.15.feed_forward.w2.weight": "model-00004-of-00011.safetensors",
60
+ "model.layers.15.feed_forward.w3.weight": "model-00004-of-00011.safetensors",
61
+ "model.layers.15.ffn_norm.weight": "model-00004-of-00011.safetensors",
62
+ "model.layers.16.attention.wo.weight": "model-00004-of-00011.safetensors",
63
+ "model.layers.16.attention.wqkv.weight": "model-00004-of-00011.safetensors",
64
+ "model.layers.16.attention_norm.weight": "model-00004-of-00011.safetensors",
65
+ "model.layers.16.feed_forward.w1.weight": "model-00004-of-00011.safetensors",
66
+ "model.layers.16.feed_forward.w2.weight": "model-00004-of-00011.safetensors",
67
+ "model.layers.16.feed_forward.w3.weight": "model-00004-of-00011.safetensors",
68
+ "model.layers.16.ffn_norm.weight": "model-00004-of-00011.safetensors",
69
+ "model.layers.17.attention.wo.weight": "model-00004-of-00011.safetensors",
70
+ "model.layers.17.attention.wqkv.weight": "model-00004-of-00011.safetensors",
71
+ "model.layers.17.attention_norm.weight": "model-00004-of-00011.safetensors",
72
+ "model.layers.17.feed_forward.w1.weight": "model-00004-of-00011.safetensors",
73
+ "model.layers.17.feed_forward.w2.weight": "model-00004-of-00011.safetensors",
74
+ "model.layers.17.feed_forward.w3.weight": "model-00004-of-00011.safetensors",
75
+ "model.layers.17.ffn_norm.weight": "model-00004-of-00011.safetensors",
76
+ "model.layers.18.attention.wo.weight": "model-00004-of-00011.safetensors",
77
+ "model.layers.18.attention.wqkv.weight": "model-00004-of-00011.safetensors",
78
+ "model.layers.18.attention_norm.weight": "model-00005-of-00011.safetensors",
79
+ "model.layers.18.feed_forward.w1.weight": "model-00004-of-00011.safetensors",
80
+ "model.layers.18.feed_forward.w2.weight": "model-00005-of-00011.safetensors",
81
+ "model.layers.18.feed_forward.w3.weight": "model-00005-of-00011.safetensors",
82
+ "model.layers.18.ffn_norm.weight": "model-00005-of-00011.safetensors",
83
+ "model.layers.19.attention.wo.weight": "model-00005-of-00011.safetensors",
84
+ "model.layers.19.attention.wqkv.weight": "model-00005-of-00011.safetensors",
85
+ "model.layers.19.attention_norm.weight": "model-00005-of-00011.safetensors",
86
+ "model.layers.19.feed_forward.w1.weight": "model-00005-of-00011.safetensors",
87
+ "model.layers.19.feed_forward.w2.weight": "model-00005-of-00011.safetensors",
88
+ "model.layers.19.feed_forward.w3.weight": "model-00005-of-00011.safetensors",
89
+ "model.layers.19.ffn_norm.weight": "model-00005-of-00011.safetensors",
90
+ "model.layers.2.attention.wo.weight": "model-00001-of-00011.safetensors",
91
+ "model.layers.2.attention.wqkv.weight": "model-00001-of-00011.safetensors",
92
+ "model.layers.2.attention_norm.weight": "model-00001-of-00011.safetensors",
93
+ "model.layers.2.feed_forward.w1.weight": "model-00001-of-00011.safetensors",
94
+ "model.layers.2.feed_forward.w2.weight": "model-00001-of-00011.safetensors",
95
+ "model.layers.2.feed_forward.w3.weight": "model-00001-of-00011.safetensors",
96
+ "model.layers.2.ffn_norm.weight": "model-00001-of-00011.safetensors",
97
+ "model.layers.20.attention.wo.weight": "model-00005-of-00011.safetensors",
98
+ "model.layers.20.attention.wqkv.weight": "model-00005-of-00011.safetensors",
99
+ "model.layers.20.attention_norm.weight": "model-00005-of-00011.safetensors",
100
+ "model.layers.20.feed_forward.w1.weight": "model-00005-of-00011.safetensors",
101
+ "model.layers.20.feed_forward.w2.weight": "model-00005-of-00011.safetensors",
102
+ "model.layers.20.feed_forward.w3.weight": "model-00005-of-00011.safetensors",
103
+ "model.layers.20.ffn_norm.weight": "model-00005-of-00011.safetensors",
104
+ "model.layers.21.attention.wo.weight": "model-00005-of-00011.safetensors",
105
+ "model.layers.21.attention.wqkv.weight": "model-00005-of-00011.safetensors",
106
+ "model.layers.21.attention_norm.weight": "model-00005-of-00011.safetensors",
107
+ "model.layers.21.feed_forward.w1.weight": "model-00005-of-00011.safetensors",
108
+ "model.layers.21.feed_forward.w2.weight": "model-00005-of-00011.safetensors",
109
+ "model.layers.21.feed_forward.w3.weight": "model-00005-of-00011.safetensors",
110
+ "model.layers.21.ffn_norm.weight": "model-00005-of-00011.safetensors",
111
+ "model.layers.22.attention.wo.weight": "model-00005-of-00011.safetensors",
112
+ "model.layers.22.attention.wqkv.weight": "model-00005-of-00011.safetensors",
113
+ "model.layers.22.attention_norm.weight": "model-00005-of-00011.safetensors",
114
+ "model.layers.22.feed_forward.w1.weight": "model-00005-of-00011.safetensors",
115
+ "model.layers.22.feed_forward.w2.weight": "model-00005-of-00011.safetensors",
116
+ "model.layers.22.feed_forward.w3.weight": "model-00005-of-00011.safetensors",
117
+ "model.layers.22.ffn_norm.weight": "model-00005-of-00011.safetensors",
118
+ "model.layers.23.attention.wo.weight": "model-00005-of-00011.safetensors",
119
+ "model.layers.23.attention.wqkv.weight": "model-00005-of-00011.safetensors",
120
+ "model.layers.23.attention_norm.weight": "model-00006-of-00011.safetensors",
121
+ "model.layers.23.feed_forward.w1.weight": "model-00005-of-00011.safetensors",
122
+ "model.layers.23.feed_forward.w2.weight": "model-00006-of-00011.safetensors",
123
+ "model.layers.23.feed_forward.w3.weight": "model-00006-of-00011.safetensors",
124
+ "model.layers.23.ffn_norm.weight": "model-00006-of-00011.safetensors",
125
+ "model.layers.24.attention.wo.weight": "model-00006-of-00011.safetensors",
126
+ "model.layers.24.attention.wqkv.weight": "model-00006-of-00011.safetensors",
127
+ "model.layers.24.attention_norm.weight": "model-00006-of-00011.safetensors",
128
+ "model.layers.24.feed_forward.w1.weight": "model-00006-of-00011.safetensors",
129
+ "model.layers.24.feed_forward.w2.weight": "model-00006-of-00011.safetensors",
130
+ "model.layers.24.feed_forward.w3.weight": "model-00006-of-00011.safetensors",
131
+ "model.layers.24.ffn_norm.weight": "model-00006-of-00011.safetensors",
132
+ "model.layers.25.attention.wo.weight": "model-00006-of-00011.safetensors",
133
+ "model.layers.25.attention.wqkv.weight": "model-00006-of-00011.safetensors",
134
+ "model.layers.25.attention_norm.weight": "model-00006-of-00011.safetensors",
135
+ "model.layers.25.feed_forward.w1.weight": "model-00006-of-00011.safetensors",
136
+ "model.layers.25.feed_forward.w2.weight": "model-00006-of-00011.safetensors",
137
+ "model.layers.25.feed_forward.w3.weight": "model-00006-of-00011.safetensors",
138
+ "model.layers.25.ffn_norm.weight": "model-00006-of-00011.safetensors",
139
+ "model.layers.26.attention.wo.weight": "model-00006-of-00011.safetensors",
140
+ "model.layers.26.attention.wqkv.weight": "model-00006-of-00011.safetensors",
141
+ "model.layers.26.attention_norm.weight": "model-00006-of-00011.safetensors",
142
+ "model.layers.26.feed_forward.w1.weight": "model-00006-of-00011.safetensors",
143
+ "model.layers.26.feed_forward.w2.weight": "model-00006-of-00011.safetensors",
144
+ "model.layers.26.feed_forward.w3.weight": "model-00006-of-00011.safetensors",
145
+ "model.layers.26.ffn_norm.weight": "model-00006-of-00011.safetensors",
146
+ "model.layers.27.attention.wo.weight": "model-00006-of-00011.safetensors",
147
+ "model.layers.27.attention.wqkv.weight": "model-00006-of-00011.safetensors",
148
+ "model.layers.27.attention_norm.weight": "model-00006-of-00011.safetensors",
149
+ "model.layers.27.feed_forward.w1.weight": "model-00006-of-00011.safetensors",
150
+ "model.layers.27.feed_forward.w2.weight": "model-00006-of-00011.safetensors",
151
+ "model.layers.27.feed_forward.w3.weight": "model-00006-of-00011.safetensors",
152
+ "model.layers.27.ffn_norm.weight": "model-00006-of-00011.safetensors",
153
+ "model.layers.28.attention.wo.weight": "model-00006-of-00011.safetensors",
154
+ "model.layers.28.attention.wqkv.weight": "model-00006-of-00011.safetensors",
155
+ "model.layers.28.attention_norm.weight": "model-00007-of-00011.safetensors",
156
+ "model.layers.28.feed_forward.w1.weight": "model-00006-of-00011.safetensors",
157
+ "model.layers.28.feed_forward.w2.weight": "model-00007-of-00011.safetensors",
158
+ "model.layers.28.feed_forward.w3.weight": "model-00007-of-00011.safetensors",
159
+ "model.layers.28.ffn_norm.weight": "model-00007-of-00011.safetensors",
160
+ "model.layers.29.attention.wo.weight": "model-00007-of-00011.safetensors",
161
+ "model.layers.29.attention.wqkv.weight": "model-00007-of-00011.safetensors",
162
+ "model.layers.29.attention_norm.weight": "model-00007-of-00011.safetensors",
163
+ "model.layers.29.feed_forward.w1.weight": "model-00007-of-00011.safetensors",
164
+ "model.layers.29.feed_forward.w2.weight": "model-00007-of-00011.safetensors",
165
+ "model.layers.29.feed_forward.w3.weight": "model-00007-of-00011.safetensors",
166
+ "model.layers.29.ffn_norm.weight": "model-00007-of-00011.safetensors",
167
+ "model.layers.3.attention.wo.weight": "model-00001-of-00011.safetensors",
168
+ "model.layers.3.attention.wqkv.weight": "model-00001-of-00011.safetensors",
169
+ "model.layers.3.attention_norm.weight": "model-00002-of-00011.safetensors",
170
+ "model.layers.3.feed_forward.w1.weight": "model-00001-of-00011.safetensors",
171
+ "model.layers.3.feed_forward.w2.weight": "model-00002-of-00011.safetensors",
172
+ "model.layers.3.feed_forward.w3.weight": "model-00002-of-00011.safetensors",
173
+ "model.layers.3.ffn_norm.weight": "model-00002-of-00011.safetensors",
174
+ "model.layers.30.attention.wo.weight": "model-00007-of-00011.safetensors",
175
+ "model.layers.30.attention.wqkv.weight": "model-00007-of-00011.safetensors",
176
+ "model.layers.30.attention_norm.weight": "model-00007-of-00011.safetensors",
177
+ "model.layers.30.feed_forward.w1.weight": "model-00007-of-00011.safetensors",
178
+ "model.layers.30.feed_forward.w2.weight": "model-00007-of-00011.safetensors",
179
+ "model.layers.30.feed_forward.w3.weight": "model-00007-of-00011.safetensors",
180
+ "model.layers.30.ffn_norm.weight": "model-00007-of-00011.safetensors",
181
+ "model.layers.31.attention.wo.weight": "model-00007-of-00011.safetensors",
182
+ "model.layers.31.attention.wqkv.weight": "model-00007-of-00011.safetensors",
183
+ "model.layers.31.attention_norm.weight": "model-00007-of-00011.safetensors",
184
+ "model.layers.31.feed_forward.w1.weight": "model-00007-of-00011.safetensors",
185
+ "model.layers.31.feed_forward.w2.weight": "model-00007-of-00011.safetensors",
186
+ "model.layers.31.feed_forward.w3.weight": "model-00007-of-00011.safetensors",
187
+ "model.layers.31.ffn_norm.weight": "model-00007-of-00011.safetensors",
188
+ "model.layers.32.attention.wo.weight": "model-00007-of-00011.safetensors",
189
+ "model.layers.32.attention.wqkv.weight": "model-00007-of-00011.safetensors",
190
+ "model.layers.32.attention_norm.weight": "model-00007-of-00011.safetensors",
191
+ "model.layers.32.feed_forward.w1.weight": "model-00007-of-00011.safetensors",
192
+ "model.layers.32.feed_forward.w2.weight": "model-00007-of-00011.safetensors",
193
+ "model.layers.32.feed_forward.w3.weight": "model-00007-of-00011.safetensors",
194
+ "model.layers.32.ffn_norm.weight": "model-00007-of-00011.safetensors",
195
+ "model.layers.33.attention.wo.weight": "model-00007-of-00011.safetensors",
196
+ "model.layers.33.attention.wqkv.weight": "model-00007-of-00011.safetensors",
197
+ "model.layers.33.attention_norm.weight": "model-00008-of-00011.safetensors",
198
+ "model.layers.33.feed_forward.w1.weight": "model-00007-of-00011.safetensors",
199
+ "model.layers.33.feed_forward.w2.weight": "model-00008-of-00011.safetensors",
200
+ "model.layers.33.feed_forward.w3.weight": "model-00008-of-00011.safetensors",
201
+ "model.layers.33.ffn_norm.weight": "model-00008-of-00011.safetensors",
202
+ "model.layers.34.attention.wo.weight": "model-00008-of-00011.safetensors",
203
+ "model.layers.34.attention.wqkv.weight": "model-00008-of-00011.safetensors",
204
+ "model.layers.34.attention_norm.weight": "model-00008-of-00011.safetensors",
205
+ "model.layers.34.feed_forward.w1.weight": "model-00008-of-00011.safetensors",
206
+ "model.layers.34.feed_forward.w2.weight": "model-00008-of-00011.safetensors",
207
+ "model.layers.34.feed_forward.w3.weight": "model-00008-of-00011.safetensors",
208
+ "model.layers.34.ffn_norm.weight": "model-00008-of-00011.safetensors",
209
+ "model.layers.35.attention.wo.weight": "model-00008-of-00011.safetensors",
210
+ "model.layers.35.attention.wqkv.weight": "model-00008-of-00011.safetensors",
211
+ "model.layers.35.attention_norm.weight": "model-00008-of-00011.safetensors",
212
+ "model.layers.35.feed_forward.w1.weight": "model-00008-of-00011.safetensors",
213
+ "model.layers.35.feed_forward.w2.weight": "model-00008-of-00011.safetensors",
214
+ "model.layers.35.feed_forward.w3.weight": "model-00008-of-00011.safetensors",
215
+ "model.layers.35.ffn_norm.weight": "model-00008-of-00011.safetensors",
216
+ "model.layers.36.attention.wo.weight": "model-00008-of-00011.safetensors",
217
+ "model.layers.36.attention.wqkv.weight": "model-00008-of-00011.safetensors",
218
+ "model.layers.36.attention_norm.weight": "model-00008-of-00011.safetensors",
219
+ "model.layers.36.feed_forward.w1.weight": "model-00008-of-00011.safetensors",
220
+ "model.layers.36.feed_forward.w2.weight": "model-00008-of-00011.safetensors",
221
+ "model.layers.36.feed_forward.w3.weight": "model-00008-of-00011.safetensors",
222
+ "model.layers.36.ffn_norm.weight": "model-00008-of-00011.safetensors",
223
+ "model.layers.37.attention.wo.weight": "model-00008-of-00011.safetensors",
224
+ "model.layers.37.attention.wqkv.weight": "model-00008-of-00011.safetensors",
225
+ "model.layers.37.attention_norm.weight": "model-00008-of-00011.safetensors",
226
+ "model.layers.37.feed_forward.w1.weight": "model-00008-of-00011.safetensors",
227
+ "model.layers.37.feed_forward.w2.weight": "model-00008-of-00011.safetensors",
228
+ "model.layers.37.feed_forward.w3.weight": "model-00008-of-00011.safetensors",
229
+ "model.layers.37.ffn_norm.weight": "model-00008-of-00011.safetensors",
230
+ "model.layers.38.attention.wo.weight": "model-00008-of-00011.safetensors",
231
+ "model.layers.38.attention.wqkv.weight": "model-00008-of-00011.safetensors",
232
+ "model.layers.38.attention_norm.weight": "model-00009-of-00011.safetensors",
233
+ "model.layers.38.feed_forward.w1.weight": "model-00008-of-00011.safetensors",
234
+ "model.layers.38.feed_forward.w2.weight": "model-00009-of-00011.safetensors",
235
+ "model.layers.38.feed_forward.w3.weight": "model-00009-of-00011.safetensors",
236
+ "model.layers.38.ffn_norm.weight": "model-00009-of-00011.safetensors",
237
+ "model.layers.39.attention.wo.weight": "model-00009-of-00011.safetensors",
238
+ "model.layers.39.attention.wqkv.weight": "model-00009-of-00011.safetensors",
239
+ "model.layers.39.attention_norm.weight": "model-00009-of-00011.safetensors",
240
+ "model.layers.39.feed_forward.w1.weight": "model-00009-of-00011.safetensors",
241
+ "model.layers.39.feed_forward.w2.weight": "model-00009-of-00011.safetensors",
242
+ "model.layers.39.feed_forward.w3.weight": "model-00009-of-00011.safetensors",
243
+ "model.layers.39.ffn_norm.weight": "model-00009-of-00011.safetensors",
244
+ "model.layers.4.attention.wo.weight": "model-00002-of-00011.safetensors",
245
+ "model.layers.4.attention.wqkv.weight": "model-00002-of-00011.safetensors",
246
+ "model.layers.4.attention_norm.weight": "model-00002-of-00011.safetensors",
247
+ "model.layers.4.feed_forward.w1.weight": "model-00002-of-00011.safetensors",
248
+ "model.layers.4.feed_forward.w2.weight": "model-00002-of-00011.safetensors",
249
+ "model.layers.4.feed_forward.w3.weight": "model-00002-of-00011.safetensors",
250
+ "model.layers.4.ffn_norm.weight": "model-00002-of-00011.safetensors",
251
+ "model.layers.40.attention.wo.weight": "model-00009-of-00011.safetensors",
252
+ "model.layers.40.attention.wqkv.weight": "model-00009-of-00011.safetensors",
253
+ "model.layers.40.attention_norm.weight": "model-00009-of-00011.safetensors",
254
+ "model.layers.40.feed_forward.w1.weight": "model-00009-of-00011.safetensors",
255
+ "model.layers.40.feed_forward.w2.weight": "model-00009-of-00011.safetensors",
256
+ "model.layers.40.feed_forward.w3.weight": "model-00009-of-00011.safetensors",
257
+ "model.layers.40.ffn_norm.weight": "model-00009-of-00011.safetensors",
258
+ "model.layers.41.attention.wo.weight": "model-00009-of-00011.safetensors",
259
+ "model.layers.41.attention.wqkv.weight": "model-00009-of-00011.safetensors",
260
+ "model.layers.41.attention_norm.weight": "model-00009-of-00011.safetensors",
261
+ "model.layers.41.feed_forward.w1.weight": "model-00009-of-00011.safetensors",
262
+ "model.layers.41.feed_forward.w2.weight": "model-00009-of-00011.safetensors",
263
+ "model.layers.41.feed_forward.w3.weight": "model-00009-of-00011.safetensors",
264
+ "model.layers.41.ffn_norm.weight": "model-00009-of-00011.safetensors",
265
+ "model.layers.42.attention.wo.weight": "model-00009-of-00011.safetensors",
266
+ "model.layers.42.attention.wqkv.weight": "model-00009-of-00011.safetensors",
267
+ "model.layers.42.attention_norm.weight": "model-00009-of-00011.safetensors",
268
+ "model.layers.42.feed_forward.w1.weight": "model-00009-of-00011.safetensors",
269
+ "model.layers.42.feed_forward.w2.weight": "model-00009-of-00011.safetensors",
270
+ "model.layers.42.feed_forward.w3.weight": "model-00009-of-00011.safetensors",
271
+ "model.layers.42.ffn_norm.weight": "model-00009-of-00011.safetensors",
272
+ "model.layers.43.attention.wo.weight": "model-00009-of-00011.safetensors",
273
+ "model.layers.43.attention.wqkv.weight": "model-00009-of-00011.safetensors",
274
+ "model.layers.43.attention_norm.weight": "model-00010-of-00011.safetensors",
275
+ "model.layers.43.feed_forward.w1.weight": "model-00009-of-00011.safetensors",
276
+ "model.layers.43.feed_forward.w2.weight": "model-00010-of-00011.safetensors",
277
+ "model.layers.43.feed_forward.w3.weight": "model-00010-of-00011.safetensors",
278
+ "model.layers.43.ffn_norm.weight": "model-00010-of-00011.safetensors",
279
+ "model.layers.44.attention.wo.weight": "model-00010-of-00011.safetensors",
280
+ "model.layers.44.attention.wqkv.weight": "model-00010-of-00011.safetensors",
281
+ "model.layers.44.attention_norm.weight": "model-00010-of-00011.safetensors",
282
+ "model.layers.44.feed_forward.w1.weight": "model-00010-of-00011.safetensors",
283
+ "model.layers.44.feed_forward.w2.weight": "model-00010-of-00011.safetensors",
284
+ "model.layers.44.feed_forward.w3.weight": "model-00010-of-00011.safetensors",
285
+ "model.layers.44.ffn_norm.weight": "model-00010-of-00011.safetensors",
286
+ "model.layers.45.attention.wo.weight": "model-00010-of-00011.safetensors",
287
+ "model.layers.45.attention.wqkv.weight": "model-00010-of-00011.safetensors",
288
+ "model.layers.45.attention_norm.weight": "model-00010-of-00011.safetensors",
289
+ "model.layers.45.feed_forward.w1.weight": "model-00010-of-00011.safetensors",
290
+ "model.layers.45.feed_forward.w2.weight": "model-00010-of-00011.safetensors",
291
+ "model.layers.45.feed_forward.w3.weight": "model-00010-of-00011.safetensors",
292
+ "model.layers.45.ffn_norm.weight": "model-00010-of-00011.safetensors",
293
+ "model.layers.46.attention.wo.weight": "model-00010-of-00011.safetensors",
294
+ "model.layers.46.attention.wqkv.weight": "model-00010-of-00011.safetensors",
295
+ "model.layers.46.attention_norm.weight": "model-00010-of-00011.safetensors",
296
+ "model.layers.46.feed_forward.w1.weight": "model-00010-of-00011.safetensors",
297
+ "model.layers.46.feed_forward.w2.weight": "model-00010-of-00011.safetensors",
298
+ "model.layers.46.feed_forward.w3.weight": "model-00010-of-00011.safetensors",
299
+ "model.layers.46.ffn_norm.weight": "model-00010-of-00011.safetensors",
300
+ "model.layers.47.attention.wo.weight": "model-00010-of-00011.safetensors",
301
+ "model.layers.47.attention.wqkv.weight": "model-00010-of-00011.safetensors",
302
+ "model.layers.47.attention_norm.weight": "model-00010-of-00011.safetensors",
303
+ "model.layers.47.feed_forward.w1.weight": "model-00010-of-00011.safetensors",
304
+ "model.layers.47.feed_forward.w2.weight": "model-00010-of-00011.safetensors",
305
+ "model.layers.47.feed_forward.w3.weight": "model-00010-of-00011.safetensors",
306
+ "model.layers.47.ffn_norm.weight": "model-00010-of-00011.safetensors",
307
+ "model.layers.5.attention.wo.weight": "model-00002-of-00011.safetensors",
308
+ "model.layers.5.attention.wqkv.weight": "model-00002-of-00011.safetensors",
309
+ "model.layers.5.attention_norm.weight": "model-00002-of-00011.safetensors",
310
+ "model.layers.5.feed_forward.w1.weight": "model-00002-of-00011.safetensors",
311
+ "model.layers.5.feed_forward.w2.weight": "model-00002-of-00011.safetensors",
312
+ "model.layers.5.feed_forward.w3.weight": "model-00002-of-00011.safetensors",
313
+ "model.layers.5.ffn_norm.weight": "model-00002-of-00011.safetensors",
314
+ "model.layers.6.attention.wo.weight": "model-00002-of-00011.safetensors",
315
+ "model.layers.6.attention.wqkv.weight": "model-00002-of-00011.safetensors",
316
+ "model.layers.6.attention_norm.weight": "model-00002-of-00011.safetensors",
317
+ "model.layers.6.feed_forward.w1.weight": "model-00002-of-00011.safetensors",
318
+ "model.layers.6.feed_forward.w2.weight": "model-00002-of-00011.safetensors",
319
+ "model.layers.6.feed_forward.w3.weight": "model-00002-of-00011.safetensors",
320
+ "model.layers.6.ffn_norm.weight": "model-00002-of-00011.safetensors",
321
+ "model.layers.7.attention.wo.weight": "model-00002-of-00011.safetensors",
322
+ "model.layers.7.attention.wqkv.weight": "model-00002-of-00011.safetensors",
323
+ "model.layers.7.attention_norm.weight": "model-00002-of-00011.safetensors",
324
+ "model.layers.7.feed_forward.w1.weight": "model-00002-of-00011.safetensors",
325
+ "model.layers.7.feed_forward.w2.weight": "model-00002-of-00011.safetensors",
326
+ "model.layers.7.feed_forward.w3.weight": "model-00002-of-00011.safetensors",
327
+ "model.layers.7.ffn_norm.weight": "model-00002-of-00011.safetensors",
328
+ "model.layers.8.attention.wo.weight": "model-00002-of-00011.safetensors",
329
+ "model.layers.8.attention.wqkv.weight": "model-00002-of-00011.safetensors",
330
+ "model.layers.8.attention_norm.weight": "model-00003-of-00011.safetensors",
331
+ "model.layers.8.feed_forward.w1.weight": "model-00002-of-00011.safetensors",
332
+ "model.layers.8.feed_forward.w2.weight": "model-00003-of-00011.safetensors",
333
+ "model.layers.8.feed_forward.w3.weight": "model-00003-of-00011.safetensors",
334
+ "model.layers.8.ffn_norm.weight": "model-00003-of-00011.safetensors",
335
+ "model.layers.9.attention.wo.weight": "model-00003-of-00011.safetensors",
336
+ "model.layers.9.attention.wqkv.weight": "model-00003-of-00011.safetensors",
337
+ "model.layers.9.attention_norm.weight": "model-00003-of-00011.safetensors",
338
+ "model.layers.9.feed_forward.w1.weight": "model-00003-of-00011.safetensors",
339
+ "model.layers.9.feed_forward.w2.weight": "model-00003-of-00011.safetensors",
340
+ "model.layers.9.feed_forward.w3.weight": "model-00003-of-00011.safetensors",
341
+ "model.layers.9.ffn_norm.weight": "model-00003-of-00011.safetensors",
342
+ "model.norm.weight": "model-00010-of-00011.safetensors",
343
+ "model.tok_embeddings.weight": "model-00001-of-00011.safetensors",
344
+ "output.weight": "model-00011-of-00011.safetensors"
345
+ }
346
+ }
modeling_internlm2.py ADDED
@@ -0,0 +1,1391 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
2
+ #
3
+ # This code is based on transformers/src/transformers/models/llama/modeling_llama.py
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+ """ PyTorch InternLM2 model."""
17
+ import math
18
+ import queue
19
+ import threading
20
+ import warnings
21
+ from typing import List, Optional, Tuple, Union
22
+
23
+ import torch
24
+ import torch.nn.functional as F
25
+ import torch.utils.checkpoint
26
+ from einops import rearrange
27
+ from torch import nn
28
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
29
+ from transformers.activations import ACT2FN
30
+ from transformers.modeling_outputs import (
31
+ BaseModelOutputWithPast,
32
+ CausalLMOutputWithPast,
33
+ SequenceClassifierOutputWithPast,
34
+ )
35
+ from transformers.modeling_utils import PreTrainedModel
36
+ from transformers.utils import (
37
+ add_start_docstrings,
38
+ add_start_docstrings_to_model_forward,
39
+ logging,
40
+ replace_return_docstrings,
41
+ )
42
+
43
+ try:
44
+ from transformers.generation.streamers import BaseStreamer
45
+ except: # noqa # pylint: disable=bare-except
46
+ BaseStreamer = None
47
+
48
+ from .configuration_internlm2 import InternLM2Config
49
+
50
+ logger = logging.get_logger(__name__)
51
+
52
+ _CONFIG_FOR_DOC = "InternLM2Config"
53
+
54
+ flash_attn_func, flash_attn_varlen_func = None, None
55
+ pad_input, index_first_axis, unpad_input = None, None, None
56
+ def _import_flash_attn():
57
+ global flash_attn_func, flash_attn_varlen_func
58
+ global pad_input, index_first_axis, unpad_input
59
+ try:
60
+ from flash_attn import flash_attn_func as _flash_attn_func, flash_attn_varlen_func as _flash_attn_varlen_func
61
+ from flash_attn.bert_padding import pad_input as _pad_input, index_first_axis as _index_first_axis, unpad_input as _unpad_input
62
+ flash_attn_func, flash_attn_varlen_func = _flash_attn_func, _flash_attn_varlen_func
63
+ pad_input, index_first_axis, unpad_input = _pad_input, _index_first_axis, _unpad_input
64
+ except ImportError:
65
+ raise ImportError("flash_attn is not installed.")
66
+
67
+ # Copied from transformers.models.llama.modeling_llama._get_unpad_data
68
+ def _get_unpad_data(attention_mask):
69
+ seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
70
+ indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
71
+ max_seqlen_in_batch = seqlens_in_batch.max().item()
72
+ cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
73
+ return (
74
+ indices,
75
+ cu_seqlens,
76
+ max_seqlen_in_batch,
77
+ )
78
+
79
+
80
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
81
+ def _make_causal_mask(
82
+ input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
83
+ ):
84
+ """
85
+ Make causal mask used for bi-directional self-attention.
86
+ """
87
+ bsz, tgt_len = input_ids_shape
88
+ mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device)
89
+ mask_cond = torch.arange(mask.size(-1), device=device)
90
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
91
+ mask = mask.to(dtype)
92
+
93
+ if past_key_values_length > 0:
94
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
95
+ return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
96
+
97
+
98
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
99
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
100
+ """
101
+ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
102
+ """
103
+ bsz, src_len = mask.size()
104
+ tgt_len = tgt_len if tgt_len is not None else src_len
105
+
106
+ expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
107
+
108
+ inverted_mask = 1.0 - expanded_mask
109
+
110
+ return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
111
+
112
+
113
+ # Copied from transformers.models.llama.modeling_llama.LlamaRMSNorm with Llama->InternLM2
114
+ class InternLM2RMSNorm(nn.Module):
115
+ def __init__(self, hidden_size, eps=1e-6):
116
+ """
117
+ InternLM2RMSNorm is equivalent to T5LayerNorm
118
+ """
119
+ super().__init__()
120
+ self.weight = nn.Parameter(torch.ones(hidden_size))
121
+ self.variance_epsilon = eps
122
+
123
+ def forward(self, hidden_states):
124
+ input_dtype = hidden_states.dtype
125
+ hidden_states = hidden_states.to(torch.float32)
126
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
127
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
128
+ return self.weight * hidden_states.to(input_dtype)
129
+
130
+
131
+ # Copied from transformers.model.llama.modeling_llama.LlamaRotaryEmbedding with Llama->InternLM2
132
+ class InternLM2RotaryEmbedding(nn.Module):
133
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
134
+ super().__init__()
135
+
136
+ self.dim = dim
137
+ self.max_position_embeddings = max_position_embeddings
138
+ self.base = base
139
+ inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
140
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
141
+
142
+ # Build here to make `torch.jit.trace` work.
143
+ self._set_cos_sin_cache(
144
+ seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
145
+ )
146
+
147
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
148
+ self.max_seq_len_cached = seq_len
149
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
150
+
151
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
152
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
153
+ emb = torch.cat((freqs, freqs), dim=-1)
154
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
155
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
156
+
157
+ def forward(self, x, seq_len=None):
158
+ # x: [bs, num_attention_heads, seq_len, head_size]
159
+ if seq_len > self.max_seq_len_cached:
160
+ self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=torch.float32)
161
+
162
+ return (
163
+ self.cos_cached[:seq_len].to(dtype=x.dtype),
164
+ self.sin_cached[:seq_len].to(dtype=x.dtype),
165
+ )
166
+
167
+
168
+ # Copied from transformers.model.llama.modeling_llama.LlamaLinearScalingRotaryEmbedding with Llama->InternLM2
169
+ class InternLM2LinearScalingRotaryEmbedding(InternLM2RotaryEmbedding):
170
+ """InternLM2RotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev"""
171
+
172
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
173
+ self.scaling_factor = scaling_factor
174
+ super().__init__(dim, max_position_embeddings, base, device)
175
+
176
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
177
+ self.max_seq_len_cached = seq_len
178
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
179
+ t = t / self.scaling_factor
180
+
181
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
182
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
183
+ emb = torch.cat((freqs, freqs), dim=-1)
184
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
185
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
186
+
187
+
188
+ # Copied from transformers.model.llama.modeling_llama.LlamaDynamicNTKScalingRotaryEmbedding with Llama->InternLM2
189
+ class InternLM2DynamicNTKScalingRotaryEmbedding(InternLM2RotaryEmbedding):
190
+ """InternLM2RotaryEmbedding extended with Dynamic NTK scaling.
191
+ Credits to the Reddit users /u/bloc97 and /u/emozilla.
192
+ """
193
+
194
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
195
+ self.scaling_factor = scaling_factor
196
+ super().__init__(dim, max_position_embeddings, base, device)
197
+
198
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
199
+ self.max_seq_len_cached = seq_len
200
+
201
+ if seq_len > self.max_position_embeddings:
202
+ base = self.base * (
203
+ (self.scaling_factor * seq_len / self.max_position_embeddings) - (self.scaling_factor - 1)
204
+ ) ** (self.dim / (self.dim - 2))
205
+ inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
206
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
207
+
208
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
209
+
210
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
211
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
212
+ emb = torch.cat((freqs, freqs), dim=-1)
213
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
214
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
215
+
216
+
217
+ # Copied from transformers.model.llama.modeling_llama.rotate_half
218
+ def rotate_half(x):
219
+ """Rotates half the hidden dims of the input."""
220
+ x1 = x[..., : x.shape[-1] // 2]
221
+ x2 = x[..., x.shape[-1] // 2 :]
222
+ return torch.cat((-x2, x1), dim=-1)
223
+
224
+
225
+ # Copied from transformers.model.llama.modeling_llama.apply_rotary_pos_emb
226
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1):
227
+ """Applies Rotary Position Embedding to the query and key tensors."""
228
+ cos = cos[position_ids].unsqueeze(unsqueeze_dim)
229
+ sin = sin[position_ids].unsqueeze(unsqueeze_dim)
230
+ q_embed = (q * cos) + (rotate_half(q) * sin)
231
+ k_embed = (k * cos) + (rotate_half(k) * sin)
232
+ return q_embed, k_embed
233
+
234
+
235
+ class InternLM2MLP(nn.Module):
236
+ def __init__(self, config):
237
+ super().__init__()
238
+ self.config = config
239
+ self.hidden_size = config.hidden_size
240
+ self.intermediate_size = config.intermediate_size
241
+ self.w1 = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
242
+ self.w3 = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
243
+ self.w2 = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
244
+ self.act_fn = ACT2FN[config.hidden_act]
245
+
246
+ def forward(self, x):
247
+ down_proj = self.w2(self.act_fn(self.w1(x)) * self.w3(x))
248
+
249
+ return down_proj
250
+
251
+
252
+ # Copied from transformers.model.llama.modeling_llama.repeat_kv
253
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
254
+ """
255
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
256
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
257
+ """
258
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
259
+ if n_rep == 1:
260
+ return hidden_states
261
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
262
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
263
+
264
+
265
+ # Modified from transformers.model.llama.modeling_llama.LlamaAttention
266
+ class InternLM2Attention(nn.Module):
267
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
268
+
269
+ def __init__(self, config: InternLM2Config):
270
+ super().__init__()
271
+ self.config = config
272
+ self.hidden_size = config.hidden_size
273
+ self.num_heads = config.num_attention_heads
274
+ self.head_dim = self.hidden_size // self.num_heads
275
+ self.num_key_value_heads = config.num_key_value_heads
276
+ self.num_key_value_groups = self.num_heads // self.num_key_value_heads
277
+ self.max_position_embeddings = config.max_position_embeddings
278
+ self.is_causal = True
279
+
280
+ if (self.head_dim * self.num_heads) != self.hidden_size:
281
+ raise ValueError(
282
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
283
+ f" and `num_heads`: {self.num_heads})."
284
+ )
285
+
286
+ self.wqkv = nn.Linear(
287
+ self.hidden_size,
288
+ (self.num_heads + 2 * self.num_key_value_heads) * self.head_dim,
289
+ bias=config.bias,
290
+ )
291
+
292
+ self.wo = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=config.bias)
293
+ self._init_rope()
294
+
295
+ def _init_rope(self):
296
+ if self.config.rope_scaling is None:
297
+ self.rotary_emb = InternLM2RotaryEmbedding(
298
+ self.head_dim,
299
+ max_position_embeddings=self.max_position_embeddings,
300
+ base=self.config.rope_theta,
301
+ )
302
+ else:
303
+ scaling_type = self.config.rope_scaling["type"]
304
+ scaling_factor = self.config.rope_scaling["factor"]
305
+ if scaling_type == "dynamic":
306
+ self.rotary_emb = InternLM2DynamicNTKScalingRotaryEmbedding(
307
+ self.head_dim,
308
+ max_position_embeddings=self.max_position_embeddings,
309
+ base=self.config.rope_theta,
310
+ scaling_factor=scaling_factor,
311
+ )
312
+ elif scaling_type == "linear":
313
+ self.rotary_emb = InternLM2LinearScalingRotaryEmbedding(
314
+ self.head_dim,
315
+ max_position_embeddings=self.max_position_embeddings,
316
+ base=self.config.rope_theta,
317
+ scaling_factor=scaling_factor,
318
+ )
319
+ else:
320
+ raise ValueError("Currently we only support rotary embedding's type being 'dynamic' or 'linear'.")
321
+ return self.rotary_emb
322
+
323
+ def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
324
+ return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
325
+
326
+ def forward(
327
+ self,
328
+ hidden_states: torch.Tensor,
329
+ attention_mask: Optional[torch.Tensor] = None,
330
+ position_ids: Optional[torch.LongTensor] = None,
331
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
332
+ output_attentions: bool = False,
333
+ use_cache: bool = False,
334
+ **kwargs,
335
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
336
+ if "padding_mask" in kwargs:
337
+ warnings.warn(
338
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. "
339
+ "Please make sure use `attention_mask` instead.`"
340
+ )
341
+
342
+ bsz, q_len, _ = hidden_states.size()
343
+
344
+ qkv_states = self.wqkv(hidden_states)
345
+
346
+ qkv_states = rearrange(
347
+ qkv_states,
348
+ "b q (h gs d) -> b q h gs d",
349
+ gs=2 + self.num_key_value_groups,
350
+ d=self.head_dim,
351
+ )
352
+
353
+ query_states = qkv_states[..., : self.num_key_value_groups, :]
354
+ query_states = rearrange(query_states, "b q h gs d -> b q (h gs) d")
355
+ key_states = qkv_states[..., -2, :]
356
+ value_states = qkv_states[..., -1, :]
357
+
358
+ query_states = query_states.transpose(1, 2)
359
+ key_states = key_states.transpose(1, 2)
360
+ value_states = value_states.transpose(1, 2)
361
+
362
+ kv_seq_len = key_states.shape[-2]
363
+ if past_key_value is not None:
364
+ kv_seq_len += past_key_value[0].shape[-2]
365
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
366
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
367
+
368
+ if past_key_value is not None:
369
+ # reuse k, v, self_attention
370
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
371
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
372
+
373
+ past_key_value = (key_states, value_states) if use_cache else None
374
+
375
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
376
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
377
+
378
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
379
+
380
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
381
+ raise ValueError(
382
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
383
+ f" {attn_weights.size()}"
384
+ )
385
+
386
+ if attention_mask is not None:
387
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
388
+ raise ValueError(
389
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
390
+ )
391
+ attn_weights = attn_weights + attention_mask
392
+
393
+ # upcast attention to fp32
394
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
395
+ attn_output = torch.matmul(attn_weights, value_states)
396
+
397
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
398
+ raise ValueError(
399
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
400
+ f" {attn_output.size()}"
401
+ )
402
+
403
+ attn_output = attn_output.transpose(1, 2).contiguous()
404
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
405
+
406
+ attn_output = self.wo(attn_output)
407
+
408
+ if not output_attentions:
409
+ attn_weights = None
410
+
411
+ return attn_output, attn_weights, past_key_value
412
+
413
+
414
+ # Modified from transformers.model.llama.modeling_llama.InternLM2FlashAttention2
415
+ class InternLM2FlashAttention2(InternLM2Attention):
416
+ """
417
+ InternLM2 flash attention module. This module inherits from `InternLM2Attention` as the weights of the module stays
418
+ untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
419
+ flash attention and deal with padding tokens in case the input contains any of them.
420
+ """
421
+
422
+ def forward(
423
+ self,
424
+ hidden_states: torch.Tensor,
425
+ attention_mask: Optional[torch.LongTensor] = None,
426
+ position_ids: Optional[torch.LongTensor] = None,
427
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
428
+ output_attentions: bool = False,
429
+ use_cache: bool = False,
430
+ **kwargs,
431
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
432
+ # InternLM2FlashAttention2 attention does not support output_attentions
433
+ if "padding_mask" in kwargs:
434
+ warnings.warn(
435
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. "
436
+ "Please make sure use `attention_mask` instead.`"
437
+ )
438
+
439
+ # overwrite attention_mask with padding_mask
440
+ attention_mask = kwargs.pop("padding_mask")
441
+
442
+ output_attentions = False
443
+
444
+ bsz, q_len, _ = hidden_states.size()
445
+
446
+ qkv_states = self.wqkv(hidden_states)
447
+
448
+ qkv_states = rearrange(
449
+ qkv_states,
450
+ "b q (h gs d) -> b q h gs d",
451
+ gs=2 + self.num_key_value_groups,
452
+ d=self.head_dim,
453
+ )
454
+
455
+ query_states = qkv_states[..., : self.num_key_value_groups, :]
456
+ query_states = rearrange(query_states, "b q h gs d -> b q (h gs) d")
457
+ key_states = qkv_states[..., -2, :]
458
+ value_states = qkv_states[..., -1, :]
459
+
460
+ query_states = query_states.transpose(1, 2)
461
+ key_states = key_states.transpose(1, 2)
462
+ value_states = value_states.transpose(1, 2)
463
+
464
+ kv_seq_len = key_states.shape[-2]
465
+ if past_key_value is not None:
466
+ kv_seq_len += past_key_value[0].shape[-2]
467
+
468
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
469
+
470
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
471
+
472
+ if past_key_value is not None:
473
+ # reuse k, v, self_attention
474
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
475
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
476
+
477
+ past_key_value = (key_states, value_states) if use_cache else None
478
+
479
+ query_states = query_states.transpose(1, 2)
480
+ key_states = key_states.transpose(1, 2)
481
+ value_states = value_states.transpose(1, 2)
482
+
483
+ attn_output = self._flash_attention_forward(
484
+ query_states, key_states, value_states, attention_mask, q_len
485
+ )
486
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
487
+ attn_output = self.wo(attn_output)
488
+
489
+ if not output_attentions:
490
+ attn_weights = None
491
+
492
+ return attn_output, attn_weights, past_key_value
493
+
494
+ def _flash_attention_forward(
495
+ self, query_states, key_states, value_states, attention_mask, query_length, dropout=0.0, softmax_scale=None
496
+ ):
497
+ """
498
+ Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
499
+ first unpad the input, then computes the attention scores and pad the final attention scores.
500
+
501
+ Args:
502
+ query_states (`torch.Tensor`):
503
+ Input query states to be passed to Flash Attention API
504
+ key_states (`torch.Tensor`):
505
+ Input key states to be passed to Flash Attention API
506
+ value_states (`torch.Tensor`):
507
+ Input value states to be passed to Flash Attention API
508
+ attention_mask (`torch.Tensor`):
509
+ The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
510
+ position of padding tokens and 1 for the position of non-padding tokens.
511
+ dropout (`int`, *optional*):
512
+ Attention dropout
513
+ softmax_scale (`float`, *optional*):
514
+ The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
515
+ """
516
+ # Contains at least one padding token in the sequence
517
+ causal = self.is_causal and query_length != 1
518
+ if attention_mask is not None:
519
+ batch_size = query_states.shape[0]
520
+ query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._unpad_input(
521
+ query_states, key_states, value_states, attention_mask, query_length
522
+ )
523
+
524
+ cu_seqlens_q, cu_seqlens_k = cu_seq_lens
525
+ max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
526
+
527
+ attn_output_unpad = flash_attn_varlen_func(
528
+ query_states,
529
+ key_states,
530
+ value_states,
531
+ cu_seqlens_q=cu_seqlens_q,
532
+ cu_seqlens_k=cu_seqlens_k,
533
+ max_seqlen_q=max_seqlen_in_batch_q,
534
+ max_seqlen_k=max_seqlen_in_batch_k,
535
+ dropout_p=dropout,
536
+ softmax_scale=softmax_scale,
537
+ causal=causal,
538
+ )
539
+
540
+ attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)
541
+ else:
542
+ attn_output = flash_attn_func(
543
+ query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal
544
+ )
545
+
546
+ return attn_output
547
+
548
+ def _unpad_input(self, query_layer, key_layer, value_layer, attention_mask, query_length):
549
+ indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
550
+ batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape
551
+
552
+ key_layer = index_first_axis(
553
+ key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
554
+ )
555
+ value_layer = index_first_axis(
556
+ value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
557
+ )
558
+
559
+ if query_length == kv_seq_len:
560
+ query_layer = index_first_axis(
561
+ query_layer.reshape(batch_size * kv_seq_len, self.num_heads, head_dim), indices_k
562
+ )
563
+ cu_seqlens_q = cu_seqlens_k
564
+ max_seqlen_in_batch_q = max_seqlen_in_batch_k
565
+ indices_q = indices_k
566
+ elif query_length == 1:
567
+ max_seqlen_in_batch_q = 1
568
+ cu_seqlens_q = torch.arange(
569
+ batch_size + 1, dtype=torch.int32, device=query_layer.device
570
+ ) # There is a memcpy here, that is very bad.
571
+ indices_q = cu_seqlens_q[:-1]
572
+ query_layer = query_layer.squeeze(1)
573
+ else:
574
+ # The -q_len: slice assumes left padding.
575
+ attention_mask = attention_mask[:, -query_length:]
576
+ query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)
577
+
578
+ return (
579
+ query_layer,
580
+ key_layer,
581
+ value_layer,
582
+ indices_q.to(torch.int64),
583
+ (cu_seqlens_q, cu_seqlens_k),
584
+ (max_seqlen_in_batch_q, max_seqlen_in_batch_k),
585
+ )
586
+
587
+ INTERNLM2_ATTENTION_CLASSES = {
588
+ "eager": InternLM2Attention,
589
+ "flash_attention_2": InternLM2FlashAttention2,
590
+ }
591
+
592
+ # Modified from transformers.model.llama.modeling_llama.LlamaDecoderLayer
593
+ class InternLM2DecoderLayer(nn.Module):
594
+ def __init__(self, config: InternLM2Config):
595
+ super().__init__()
596
+ self.hidden_size = config.hidden_size
597
+
598
+ self.attention = INTERNLM2_ATTENTION_CLASSES[config.attn_implementation](config=config)
599
+
600
+ self.feed_forward = InternLM2MLP(config)
601
+ self.attention_norm = InternLM2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
602
+ self.ffn_norm = InternLM2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
603
+
604
+ def forward(
605
+ self,
606
+ hidden_states: torch.Tensor,
607
+ attention_mask: Optional[torch.Tensor] = None,
608
+ position_ids: Optional[torch.LongTensor] = None,
609
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
610
+ output_attentions: Optional[bool] = False,
611
+ use_cache: Optional[bool] = False,
612
+ **kwargs,
613
+ ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
614
+ """
615
+ Args:
616
+ hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
617
+ attention_mask (`torch.FloatTensor`, *optional*):
618
+ attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1,
619
+ query_sequence_length, key_sequence_length)` if default attention is used.
620
+ output_attentions (`bool`, *optional*):
621
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under
622
+ returned tensors for more detail.
623
+ use_cache (`bool`, *optional*):
624
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
625
+ (see `past_key_values`).
626
+ past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
627
+ """
628
+ if "padding_mask" in kwargs:
629
+ warnings.warn(
630
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. "
631
+ "Please make sure use `attention_mask` instead.`"
632
+ )
633
+
634
+ residual = hidden_states
635
+
636
+ hidden_states = self.attention_norm(hidden_states)
637
+
638
+ # Self Attention
639
+ hidden_states, self_attn_weights, present_key_value = self.attention(
640
+ hidden_states=hidden_states,
641
+ attention_mask=attention_mask,
642
+ position_ids=position_ids,
643
+ past_key_value=past_key_value,
644
+ output_attentions=output_attentions,
645
+ use_cache=use_cache,
646
+ **kwargs,
647
+ )
648
+ hidden_states = residual + hidden_states
649
+
650
+ # Fully Connected
651
+ residual = hidden_states
652
+ hidden_states = self.ffn_norm(hidden_states)
653
+ hidden_states = self.feed_forward(hidden_states)
654
+ hidden_states = residual + hidden_states
655
+
656
+ outputs = (hidden_states,)
657
+
658
+ if output_attentions:
659
+ outputs += (self_attn_weights,)
660
+
661
+ if use_cache:
662
+ outputs += (present_key_value,)
663
+
664
+ return outputs
665
+
666
+
667
+ InternLM2_START_DOCSTRING = r"""
668
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
669
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
670
+ etc.)
671
+
672
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
673
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
674
+ and behavior.
675
+
676
+ Parameters:
677
+ config ([`InternLM2Config`]):
678
+ Model configuration class with all the parameters of the model. Initializing with a config file does not
679
+ load the weights associated with the model, only the configuration. Check out the
680
+ [`~PreTrainedModel.from_pretrained`] method to load the model weights.
681
+ """
682
+
683
+
684
+ # Copied from transformers.models.llama.modeling_llama.LlamaPreTrainedModel with Llama->InternLM2
685
+ @add_start_docstrings(
686
+ "The bare InternLM2 Model outputting raw hidden-states without any specific head on top.",
687
+ InternLM2_START_DOCSTRING,
688
+ )
689
+ class InternLM2PreTrainedModel(PreTrainedModel):
690
+ config_class = InternLM2Config
691
+ base_model_prefix = "model"
692
+ supports_gradient_checkpointing = True
693
+ _no_split_modules = ["InternLM2DecoderLayer"]
694
+ _skip_keys_device_placement = "past_key_values"
695
+
696
+ def _init_weights(self, module):
697
+ std = self.config.initializer_range
698
+ if isinstance(module, nn.Linear):
699
+ module.weight.data.normal_(mean=0.0, std=std)
700
+ if module.bias is not None:
701
+ module.bias.data.zero_()
702
+ elif isinstance(module, nn.Embedding):
703
+ module.weight.data.normal_(mean=0.0, std=std)
704
+ if module.padding_idx is not None:
705
+ module.weight.data[module.padding_idx].zero_()
706
+
707
+
708
+ InternLM2_INPUTS_DOCSTRING = r"""
709
+ Args:
710
+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
711
+ Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
712
+ it.
713
+
714
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
715
+ [`PreTrainedTokenizer.__call__`] for details.
716
+
717
+ [What are input IDs?](../glossary#input-ids)
718
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
719
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
720
+
721
+ - 1 for tokens that are **not masked**,
722
+ - 0 for tokens that are **masked**.
723
+
724
+ [What are attention masks?](../glossary#attention-mask)
725
+
726
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
727
+ [`PreTrainedTokenizer.__call__`] for details.
728
+
729
+ If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
730
+ `past_key_values`).
731
+
732
+ If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
733
+ and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
734
+ information on the default strategy.
735
+
736
+ - 1 indicates the head is **not masked**,
737
+ - 0 indicates the head is **masked**.
738
+ position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
739
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
740
+ config.n_positions - 1]`.
741
+
742
+ [What are position IDs?](../glossary#position-ids)
743
+ past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or
744
+ when `config.use_cache=True`):
745
+ Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape
746
+ `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape
747
+ `(batch_size, num_heads, decoder_sequence_length, embed_size_per_head)`.
748
+
749
+ Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
750
+ blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
751
+
752
+ If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
753
+ have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
754
+ of shape `(batch_size, sequence_length)`.
755
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
756
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
757
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
758
+ model's internal embedding lookup matrix.
759
+ use_cache (`bool`, *optional*):
760
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
761
+ `past_key_values`).
762
+ output_attentions (`bool`, *optional*):
763
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
764
+ tensors for more detail.
765
+ output_hidden_states (`bool`, *optional*):
766
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
767
+ more detail.
768
+ return_dict (`bool`, *optional*):
769
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
770
+ """
771
+
772
+
773
+ # Modified from transformers.model.llama.modeling_llama.LlamaModel
774
+ @add_start_docstrings(
775
+ "The bare InternLM2 Model outputting raw hidden-states without any specific head on top.",
776
+ InternLM2_START_DOCSTRING,
777
+ )
778
+ class InternLM2Model(InternLM2PreTrainedModel):
779
+ """
780
+ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`InternLM2DecoderLayer`]
781
+
782
+ Args:
783
+ config: InternLM2Config
784
+ """
785
+
786
+ _auto_class = "AutoModel"
787
+
788
+ def __init__(self, config: InternLM2Config):
789
+ super().__init__(config)
790
+ self.padding_idx = config.pad_token_id
791
+ self.vocab_size = config.vocab_size
792
+ self.config = config
793
+
794
+ self.tok_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
795
+
796
+ self.layers = nn.ModuleList([InternLM2DecoderLayer(config) for _ in range(config.num_hidden_layers)])
797
+ self.norm = InternLM2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
798
+
799
+ self.gradient_checkpointing = False
800
+ # Initialize weights and apply final processing
801
+ self.post_init()
802
+
803
+ def get_input_embeddings(self):
804
+ return self.tok_embeddings
805
+
806
+ def set_input_embeddings(self, value):
807
+ self.tok_embeddings = value
808
+
809
+ def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
810
+ # create causal mask
811
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
812
+ combined_attention_mask = None
813
+ if input_shape[-1] > 1:
814
+ combined_attention_mask = _make_causal_mask(
815
+ input_shape,
816
+ inputs_embeds.dtype,
817
+ device=inputs_embeds.device,
818
+ past_key_values_length=past_key_values_length,
819
+ )
820
+
821
+ if attention_mask is not None:
822
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
823
+ expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
824
+ inputs_embeds.device
825
+ )
826
+ combined_attention_mask = (
827
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
828
+ )
829
+
830
+ return combined_attention_mask
831
+
832
+ @add_start_docstrings_to_model_forward(InternLM2_INPUTS_DOCSTRING)
833
+ def forward(
834
+ self,
835
+ input_ids: torch.LongTensor = None,
836
+ attention_mask: Optional[torch.Tensor] = None,
837
+ position_ids: Optional[torch.LongTensor] = None,
838
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
839
+ inputs_embeds: Optional[torch.FloatTensor] = None,
840
+ use_cache: Optional[bool] = None,
841
+ output_attentions: Optional[bool] = None,
842
+ output_hidden_states: Optional[bool] = None,
843
+ return_dict: Optional[bool] = None,
844
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
845
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
846
+ output_hidden_states = (
847
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
848
+ )
849
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
850
+
851
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
852
+
853
+ if self.config.attn_implementation == "flash_attention_2":
854
+ _import_flash_attn()
855
+
856
+ # retrieve input_ids and inputs_embeds
857
+ if input_ids is not None and inputs_embeds is not None:
858
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
859
+ elif input_ids is not None:
860
+ batch_size, seq_length = input_ids.shape[:2]
861
+ elif inputs_embeds is not None:
862
+ batch_size, seq_length = inputs_embeds.shape[:2]
863
+ else:
864
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
865
+
866
+ seq_length_with_past = seq_length
867
+ past_key_values_length = 0
868
+ if past_key_values is not None:
869
+ past_key_values_length = past_key_values[0][0].shape[2]
870
+ seq_length_with_past = seq_length_with_past + past_key_values_length
871
+
872
+ if position_ids is None:
873
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
874
+ position_ids = torch.arange(
875
+ past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
876
+ )
877
+ position_ids = position_ids.unsqueeze(0)
878
+
879
+ if inputs_embeds is None:
880
+ inputs_embeds = self.tok_embeddings(input_ids)
881
+
882
+ if self.config.attn_implementation == "flash_attention_2":
883
+ # 2d mask is passed through the layers
884
+ attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None
885
+ else:
886
+ if attention_mask is None:
887
+ attention_mask = torch.ones(
888
+ (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device
889
+ )
890
+ attention_mask = self._prepare_decoder_attention_mask(
891
+ attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length
892
+ )
893
+
894
+ # embed positions
895
+ hidden_states = inputs_embeds
896
+
897
+ if self.gradient_checkpointing and self.training:
898
+ if use_cache:
899
+ logger.warning_once(
900
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
901
+ )
902
+ use_cache = False
903
+
904
+ # decoder layers
905
+ all_hidden_states = () if output_hidden_states else None
906
+ all_self_attns = () if output_attentions else None
907
+ next_decoder_cache = () if use_cache else None
908
+
909
+ for idx, decoder_layer in enumerate(self.layers):
910
+ if output_hidden_states:
911
+ all_hidden_states += (hidden_states,)
912
+
913
+ past_key_value = past_key_values[idx] if past_key_values is not None else None
914
+
915
+ if self.gradient_checkpointing and self.training:
916
+
917
+ def create_custom_forward(module):
918
+ def custom_forward(*inputs):
919
+ # None for past_key_value
920
+ return module(*inputs, output_attentions, None)
921
+
922
+ return custom_forward
923
+
924
+ layer_outputs = torch.utils.checkpoint.checkpoint(
925
+ create_custom_forward(decoder_layer),
926
+ hidden_states,
927
+ attention_mask,
928
+ position_ids,
929
+ None,
930
+ )
931
+ else:
932
+ layer_outputs = decoder_layer(
933
+ hidden_states,
934
+ attention_mask=attention_mask,
935
+ position_ids=position_ids,
936
+ past_key_value=past_key_value,
937
+ output_attentions=output_attentions,
938
+ use_cache=use_cache,
939
+ )
940
+
941
+ hidden_states = layer_outputs[0]
942
+
943
+ if use_cache:
944
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
945
+
946
+ if output_attentions:
947
+ all_self_attns += (layer_outputs[1],)
948
+
949
+ hidden_states = self.norm(hidden_states)
950
+
951
+ # add hidden states from the last decoder layer
952
+ if output_hidden_states:
953
+ all_hidden_states += (hidden_states,)
954
+
955
+ next_cache = next_decoder_cache if use_cache else None
956
+ if not return_dict:
957
+ return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
958
+ return BaseModelOutputWithPast(
959
+ last_hidden_state=hidden_states,
960
+ past_key_values=next_cache,
961
+ hidden_states=all_hidden_states,
962
+ attentions=all_self_attns,
963
+ )
964
+
965
+
966
+ # Modified from transformers.model.llama.modeling_llama.LlamaForCausalLM
967
+ class InternLM2ForCausalLM(InternLM2PreTrainedModel):
968
+ _auto_class = "AutoModelForCausalLM"
969
+
970
+ _tied_weights_keys = ["output.weight"]
971
+
972
+ def __init__(self, config):
973
+ super().__init__(config)
974
+ self.model = InternLM2Model(config)
975
+ self.vocab_size = config.vocab_size
976
+ self.output = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
977
+
978
+ # Initialize weights and apply final processing
979
+ self.post_init()
980
+
981
+ def get_input_embeddings(self):
982
+ return self.model.tok_embeddings
983
+
984
+ def set_input_embeddings(self, value):
985
+ self.model.tok_embeddings = value
986
+
987
+ def get_output_embeddings(self):
988
+ return self.output
989
+
990
+ def set_output_embeddings(self, new_embeddings):
991
+ self.output = new_embeddings
992
+
993
+ def set_decoder(self, decoder):
994
+ self.model = decoder
995
+
996
+ def get_decoder(self):
997
+ return self.model
998
+
999
+ @add_start_docstrings_to_model_forward(InternLM2_INPUTS_DOCSTRING)
1000
+ @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
1001
+ def forward(
1002
+ self,
1003
+ input_ids: torch.LongTensor = None,
1004
+ attention_mask: Optional[torch.Tensor] = None,
1005
+ position_ids: Optional[torch.LongTensor] = None,
1006
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1007
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1008
+ labels: Optional[torch.LongTensor] = None,
1009
+ use_cache: Optional[bool] = None,
1010
+ output_attentions: Optional[bool] = None,
1011
+ output_hidden_states: Optional[bool] = None,
1012
+ return_dict: Optional[bool] = None,
1013
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
1014
+ r"""
1015
+ Args:
1016
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1017
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
1018
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
1019
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
1020
+
1021
+ Returns:
1022
+
1023
+ Example:
1024
+
1025
+ ```python
1026
+ >>> from transformers import AutoTokenizer, InternLM2ForCausalLM
1027
+
1028
+ >>> model = InternLM2ForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
1029
+ >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
1030
+
1031
+ >>> prompt = "Hey, are you conscious? Can you talk to me?"
1032
+ >>> inputs = tokenizer(prompt, return_tensors="pt")
1033
+
1034
+ >>> # Generate
1035
+ >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
1036
+ >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
1037
+ "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
1038
+ ```"""
1039
+
1040
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1041
+ output_hidden_states = (
1042
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1043
+ )
1044
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1045
+
1046
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
1047
+ outputs = self.model(
1048
+ input_ids=input_ids,
1049
+ attention_mask=attention_mask,
1050
+ position_ids=position_ids,
1051
+ past_key_values=past_key_values,
1052
+ inputs_embeds=inputs_embeds,
1053
+ use_cache=use_cache,
1054
+ output_attentions=output_attentions,
1055
+ output_hidden_states=output_hidden_states,
1056
+ return_dict=return_dict,
1057
+ )
1058
+
1059
+ hidden_states = outputs[0]
1060
+ logits = self.output(hidden_states)
1061
+ logits = logits.float()
1062
+
1063
+ loss = None
1064
+ if labels is not None:
1065
+ # Shift so that tokens < n predict n
1066
+ shift_logits = logits[..., :-1, :].contiguous()
1067
+ shift_labels = labels[..., 1:].contiguous()
1068
+ # Flatten the tokens
1069
+ loss_fct = CrossEntropyLoss()
1070
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
1071
+ shift_labels = shift_labels.view(-1)
1072
+ # Enable model parallelism
1073
+ shift_labels = shift_labels.to(shift_logits.device)
1074
+ loss = loss_fct(shift_logits, shift_labels)
1075
+
1076
+ if not return_dict:
1077
+ output = (logits,) + outputs[1:]
1078
+ return (loss,) + output if loss is not None else output
1079
+
1080
+ return CausalLMOutputWithPast(
1081
+ loss=loss,
1082
+ logits=logits,
1083
+ past_key_values=outputs.past_key_values,
1084
+ hidden_states=outputs.hidden_states,
1085
+ attentions=outputs.attentions,
1086
+ )
1087
+
1088
+ def prepare_inputs_for_generation(
1089
+ self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
1090
+ ):
1091
+ if past_key_values is not None:
1092
+ past_length = past_key_values[0][0].shape[2]
1093
+
1094
+ # Some generation methods already pass only the last input ID
1095
+ if input_ids.shape[1] > past_length:
1096
+ remove_prefix_length = past_length
1097
+ else:
1098
+ # Default to old behavior: keep only final ID
1099
+ remove_prefix_length = input_ids.shape[1] - 1
1100
+
1101
+ input_ids = input_ids[:, remove_prefix_length:]
1102
+
1103
+ position_ids = kwargs.get("position_ids", None)
1104
+ if attention_mask is not None and position_ids is None:
1105
+ # create position_ids on the fly for batch generation
1106
+ position_ids = attention_mask.long().cumsum(-1) - 1
1107
+ position_ids.masked_fill_(attention_mask == 0, 1)
1108
+ if past_key_values:
1109
+ position_ids = position_ids[:, -input_ids.shape[1] :]
1110
+
1111
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
1112
+ if inputs_embeds is not None and past_key_values is None:
1113
+ model_inputs = {"inputs_embeds": inputs_embeds}
1114
+ else:
1115
+ model_inputs = {"input_ids": input_ids}
1116
+
1117
+ model_inputs.update(
1118
+ {
1119
+ "position_ids": position_ids,
1120
+ "past_key_values": past_key_values,
1121
+ "use_cache": kwargs.get("use_cache"),
1122
+ "attention_mask": attention_mask,
1123
+ }
1124
+ )
1125
+ return model_inputs
1126
+
1127
+ @staticmethod
1128
+ def _reorder_cache(past_key_values, beam_idx):
1129
+ reordered_past = ()
1130
+ for layer_past in past_key_values:
1131
+ reordered_past += (
1132
+ tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
1133
+ )
1134
+ return reordered_past
1135
+
1136
+ def build_inputs(self, tokenizer, query: str, history: List[Tuple[str, str]] = [], meta_instruction=""):
1137
+ if tokenizer.add_bos_token:
1138
+ prompt = ""
1139
+ else:
1140
+ prompt = tokenizer.bos_token
1141
+ if meta_instruction:
1142
+ prompt += f"""<|im_start|>system\n{meta_instruction}<|im_end|>\n"""
1143
+ for record in history:
1144
+ prompt += f"""<|im_start|>user\n{record[0]}<|im_end|>\n<|im_start|>assistant\n{record[1]}<|im_end|>\n"""
1145
+ prompt += f"""<|im_start|>user\n{query}<|im_end|>\n<|im_start|>assistant\n"""
1146
+ return tokenizer([prompt], return_tensors="pt")
1147
+
1148
+ @torch.no_grad()
1149
+ def chat(
1150
+ self,
1151
+ tokenizer,
1152
+ query: str,
1153
+ history: List[Tuple[str, str]] = [],
1154
+ streamer: Optional[BaseStreamer] = None,
1155
+ max_new_tokens: int = 1024,
1156
+ do_sample: bool = True,
1157
+ temperature: float = 0.8,
1158
+ top_p: float = 0.8,
1159
+ meta_instruction: str = "You are an AI assistant whose name is InternLM (书生·浦语).\n"
1160
+ "- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory (上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n"
1161
+ "- InternLM (书生·浦语) can understand and communicate fluently in the language chosen by the user such as English and 中文.",
1162
+ **kwargs,
1163
+ ):
1164
+ inputs = self.build_inputs(tokenizer, query, history, meta_instruction)
1165
+ inputs = {k: v.to(self.device) for k, v in inputs.items() if torch.is_tensor(v)}
1166
+ # also add end-of-assistant token in eos token id to avoid unnecessary generation
1167
+ eos_token_id = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids(["<|im_end|>"])[0]]
1168
+ outputs = self.generate(
1169
+ **inputs,
1170
+ streamer=streamer,
1171
+ max_new_tokens=max_new_tokens,
1172
+ do_sample=do_sample,
1173
+ temperature=temperature,
1174
+ top_p=top_p,
1175
+ eos_token_id=eos_token_id,
1176
+ **kwargs,
1177
+ )
1178
+ outputs = outputs[0].cpu().tolist()[len(inputs["input_ids"][0]) :]
1179
+ response = tokenizer.decode(outputs, skip_special_tokens=True)
1180
+ response = response.split("<|im_end|>")[0]
1181
+ history = history + [(query, response)]
1182
+ return response, history
1183
+
1184
+ @torch.no_grad()
1185
+ def stream_chat(
1186
+ self,
1187
+ tokenizer,
1188
+ query: str,
1189
+ history: List[Tuple[str, str]] = [],
1190
+ max_new_tokens: int = 1024,
1191
+ do_sample: bool = True,
1192
+ temperature: float = 0.8,
1193
+ top_p: float = 0.8,
1194
+ **kwargs,
1195
+ ):
1196
+ """
1197
+ Return a generator in format: (response, history)
1198
+ Eg.
1199
+ ('你好,有什么可以帮助您的吗', [('你好', '你好,有什么可以帮助您的吗')])
1200
+ ('你好,有什么可以帮助您的吗?', [('你好', '你好,有什么可以帮助您的吗?')])
1201
+ """
1202
+ if BaseStreamer is None:
1203
+ raise ModuleNotFoundError(
1204
+ "The version of `transformers` is too low. Please make sure "
1205
+ "that you have installed `transformers>=4.28.0`."
1206
+ )
1207
+
1208
+ response_queue = queue.Queue(maxsize=20)
1209
+
1210
+ class ChatStreamer(BaseStreamer):
1211
+ def __init__(self, tokenizer) -> None:
1212
+ super().__init__()
1213
+ self.tokenizer = tokenizer
1214
+ self.queue = response_queue
1215
+ self.query = query
1216
+ self.history = history
1217
+ self.response = ""
1218
+ self.cache = []
1219
+ self.received_inputs = False
1220
+ self.queue.put((self.response, history + [(self.query, self.response)]))
1221
+
1222
+ def put(self, value):
1223
+ if len(value.shape) > 1 and value.shape[0] > 1:
1224
+ raise ValueError("ChatStreamer only supports batch size 1")
1225
+ elif len(value.shape) > 1:
1226
+ value = value[0]
1227
+
1228
+ if not self.received_inputs:
1229
+ # The first received value is input_ids, ignore here
1230
+ self.received_inputs = True
1231
+ return
1232
+
1233
+ self.cache.extend(value.tolist())
1234
+ token = self.tokenizer.decode(self.cache, skip_special_tokens=True)
1235
+ if token.strip() != "<|im_end|>":
1236
+ self.response = self.response + token
1237
+ history = self.history + [(self.query, self.response)]
1238
+ self.queue.put((self.response, history))
1239
+ self.cache = []
1240
+ else:
1241
+ self.end()
1242
+
1243
+ def end(self):
1244
+ self.queue.put(None)
1245
+
1246
+ def stream_producer():
1247
+ return self.chat(
1248
+ tokenizer=tokenizer,
1249
+ query=query,
1250
+ streamer=ChatStreamer(tokenizer=tokenizer),
1251
+ history=history,
1252
+ max_new_tokens=max_new_tokens,
1253
+ do_sample=do_sample,
1254
+ temperature=temperature,
1255
+ top_p=top_p,
1256
+ **kwargs,
1257
+ )
1258
+
1259
+ def consumer():
1260
+ producer = threading.Thread(target=stream_producer)
1261
+ producer.start()
1262
+ while True:
1263
+ res = response_queue.get()
1264
+ if res is None:
1265
+ return
1266
+ yield res
1267
+
1268
+ return consumer()
1269
+
1270
+
1271
+ # Copied from transformers.model.llama.modeling_llama.LlamaForSequenceClassification with Llama->InternLM2
1272
+ @add_start_docstrings(
1273
+ """
1274
+ The InternLM2 Model transformer with a sequence classification head on top (linear layer).
1275
+
1276
+ [`InternLM2ForSequenceClassification`] uses the last token in order to do the classification,
1277
+ as other causal models (e.g. GPT-2) do.
1278
+
1279
+ Since it does classification on the last token, it requires to know the position of the last token. If a
1280
+ `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If
1281
+ no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
1282
+ padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in
1283
+ each row of the batch).
1284
+ """,
1285
+ InternLM2_START_DOCSTRING,
1286
+ )
1287
+ class InternLM2ForSequenceClassification(InternLM2PreTrainedModel):
1288
+ def __init__(self, config):
1289
+ super().__init__(config)
1290
+ self.num_labels = config.num_labels
1291
+ self.model = InternLM2Model(config)
1292
+ self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)
1293
+
1294
+ # Initialize weights and apply final processing
1295
+ self.post_init()
1296
+
1297
+ def get_input_embeddings(self):
1298
+ return self.model.tok_embeddings
1299
+
1300
+ def set_input_embeddings(self, value):
1301
+ self.model.tok_embeddings = value
1302
+
1303
+ @add_start_docstrings_to_model_forward(InternLM2_INPUTS_DOCSTRING)
1304
+ def forward(
1305
+ self,
1306
+ input_ids: torch.LongTensor = None,
1307
+ attention_mask: Optional[torch.Tensor] = None,
1308
+ position_ids: Optional[torch.LongTensor] = None,
1309
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1310
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1311
+ labels: Optional[torch.LongTensor] = None,
1312
+ use_cache: Optional[bool] = None,
1313
+ output_attentions: Optional[bool] = None,
1314
+ output_hidden_states: Optional[bool] = None,
1315
+ return_dict: Optional[bool] = None,
1316
+ ) -> Union[Tuple, SequenceClassifierOutputWithPast]:
1317
+ r"""
1318
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1319
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
1320
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
1321
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
1322
+ """
1323
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1324
+
1325
+ transformer_outputs = self.model(
1326
+ input_ids,
1327
+ attention_mask=attention_mask,
1328
+ position_ids=position_ids,
1329
+ past_key_values=past_key_values,
1330
+ inputs_embeds=inputs_embeds,
1331
+ use_cache=use_cache,
1332
+ output_attentions=output_attentions,
1333
+ output_hidden_states=output_hidden_states,
1334
+ return_dict=return_dict,
1335
+ )
1336
+ hidden_states = transformer_outputs[0]
1337
+ logits = self.score(hidden_states)
1338
+
1339
+ if input_ids is not None:
1340
+ batch_size = input_ids.shape[0]
1341
+ else:
1342
+ batch_size = inputs_embeds.shape[0]
1343
+
1344
+ if self.config.pad_token_id is None and batch_size != 1:
1345
+ raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
1346
+ if self.config.pad_token_id is None:
1347
+ sequence_lengths = -1
1348
+ else:
1349
+ if input_ids is not None:
1350
+ sequence_lengths = (torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1).to(
1351
+ logits.device
1352
+ )
1353
+ else:
1354
+ sequence_lengths = -1
1355
+
1356
+ pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
1357
+
1358
+ loss = None
1359
+ if labels is not None:
1360
+ labels = labels.to(logits.device)
1361
+ if self.config.problem_type is None:
1362
+ if self.num_labels == 1:
1363
+ self.config.problem_type = "regression"
1364
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
1365
+ self.config.problem_type = "single_label_classification"
1366
+ else:
1367
+ self.config.problem_type = "multi_label_classification"
1368
+
1369
+ if self.config.problem_type == "regression":
1370
+ loss_fct = MSELoss()
1371
+ if self.num_labels == 1:
1372
+ loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
1373
+ else:
1374
+ loss = loss_fct(pooled_logits, labels)
1375
+ elif self.config.problem_type == "single_label_classification":
1376
+ loss_fct = CrossEntropyLoss()
1377
+ loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1))
1378
+ elif self.config.problem_type == "multi_label_classification":
1379
+ loss_fct = BCEWithLogitsLoss()
1380
+ loss = loss_fct(pooled_logits, labels)
1381
+ if not return_dict:
1382
+ output = (pooled_logits,) + transformer_outputs[1:]
1383
+ return ((loss,) + output) if loss is not None else output
1384
+
1385
+ return SequenceClassifierOutputWithPast(
1386
+ loss=loss,
1387
+ logits=pooled_logits,
1388
+ past_key_values=transformer_outputs.past_key_values,
1389
+ hidden_states=transformer_outputs.hidden_states,
1390
+ attentions=transformer_outputs.attentions,
1391
+ )
special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "</s>",
5
+ "unk_token": "<unk>"
6
+ }
tokenization_internlm2.py ADDED
@@ -0,0 +1,236 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on transformers/src/transformers/models/llama/tokenization_llama.py
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+
18
+ """Tokenization classes for InternLM."""
19
+ import os
20
+ from shutil import copyfile
21
+ from typing import Any, Dict, List, Optional, Tuple
22
+
23
+ import sentencepiece as spm
24
+ from transformers.tokenization_utils import PreTrainedTokenizer
25
+ from transformers.utils import logging
26
+
27
+ logger = logging.get_logger(__name__)
28
+
29
+ VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
30
+
31
+ PRETRAINED_VOCAB_FILES_MAP = {}
32
+
33
+
34
+ # Modified from transformers.model.llama.tokenization_llama.LlamaTokenizer
35
+ class InternLM2Tokenizer(PreTrainedTokenizer):
36
+ """
37
+ Construct a InternLM2 tokenizer. Based on byte-level Byte-Pair-Encoding.
38
+
39
+ Args:
40
+ vocab_file (`str`):
41
+ Path to the vocabulary file.
42
+ """
43
+
44
+ vocab_files_names = VOCAB_FILES_NAMES
45
+ pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
46
+ model_input_names = ["input_ids", "attention_mask"]
47
+ _auto_class = "AutoTokenizer"
48
+
49
+ def __init__(
50
+ self,
51
+ vocab_file,
52
+ unk_token="<unk>",
53
+ bos_token="<s>",
54
+ eos_token="</s>",
55
+ pad_token="</s>",
56
+ sp_model_kwargs: Optional[Dict[str, Any]] = None,
57
+ add_bos_token=True,
58
+ add_eos_token=False,
59
+ decode_with_prefix_space=False,
60
+ clean_up_tokenization_spaces=False,
61
+ **kwargs,
62
+ ):
63
+ self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
64
+ self.vocab_file = vocab_file
65
+ self.add_bos_token = add_bos_token
66
+ self.add_eos_token = add_eos_token
67
+ self.decode_with_prefix_space = decode_with_prefix_space
68
+ self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
69
+ self.sp_model.Load(vocab_file)
70
+ self._no_prefix_space_tokens = None
71
+ super().__init__(
72
+ bos_token=bos_token,
73
+ eos_token=eos_token,
74
+ unk_token=unk_token,
75
+ pad_token=pad_token,
76
+ clean_up_tokenization_spaces=clean_up_tokenization_spaces,
77
+ **kwargs,
78
+ )
79
+
80
+ @property
81
+ def no_prefix_space_tokens(self):
82
+ if self._no_prefix_space_tokens is None:
83
+ vocab = self.convert_ids_to_tokens(list(range(self.vocab_size)))
84
+ self._no_prefix_space_tokens = {i for i, tok in enumerate(vocab) if not tok.startswith("▁")}
85
+ return self._no_prefix_space_tokens
86
+
87
+ @property
88
+ def vocab_size(self):
89
+ """Returns vocab size"""
90
+ return self.sp_model.get_piece_size()
91
+
92
+ @property
93
+ def bos_token_id(self) -> Optional[int]:
94
+ return self.sp_model.bos_id()
95
+
96
+ @property
97
+ def eos_token_id(self) -> Optional[int]:
98
+ return self.sp_model.eos_id()
99
+
100
+ def get_vocab(self):
101
+ """Returns vocab as a dict"""
102
+ vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
103
+ vocab.update(self.added_tokens_encoder)
104
+ return vocab
105
+
106
+ def _tokenize(self, text):
107
+ """Returns a tokenized string."""
108
+ return self.sp_model.encode(text, out_type=str)
109
+
110
+ def _convert_token_to_id(self, token):
111
+ """Converts a token (str) in an id using the vocab."""
112
+ return self.sp_model.piece_to_id(token)
113
+
114
+ def _convert_id_to_token(self, index):
115
+ """Converts an index (integer) in a token (str) using the vocab."""
116
+ token = self.sp_model.IdToPiece(index)
117
+ return token
118
+
119
+ def _maybe_add_prefix_space(self, tokens, decoded):
120
+ if tokens and tokens[0] not in self.no_prefix_space_tokens:
121
+ return " " + decoded
122
+ else:
123
+ return decoded
124
+
125
+ def convert_tokens_to_string(self, tokens):
126
+ """Converts a sequence of tokens (string) in a single string."""
127
+ current_sub_tokens = []
128
+ out_string = ""
129
+ prev_is_special = False
130
+ for token in tokens:
131
+ # make sure that special tokens are not decoded using sentencepiece model
132
+ if token in self.all_special_tokens:
133
+ if not prev_is_special:
134
+ out_string += " "
135
+ out_string += self.sp_model.decode(current_sub_tokens) + token
136
+ prev_is_special = True
137
+ current_sub_tokens = []
138
+ else:
139
+ current_sub_tokens.append(token)
140
+ prev_is_special = False
141
+ out_string += self.sp_model.decode(current_sub_tokens)
142
+ out_string = self.clean_up_tokenization(out_string)
143
+ out_string = self._maybe_add_prefix_space(tokens=tokens, decoded=out_string)
144
+ return out_string[1:]
145
+
146
+ def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]:
147
+ """
148
+ Save the vocabulary and special tokens file to a directory.
149
+
150
+ Args:
151
+ save_directory (`str`):
152
+ The directory in which to save the vocabulary.
153
+
154
+ Returns:
155
+ `Tuple(str)`: Paths to the files saved.
156
+ """
157
+ if not os.path.isdir(save_directory):
158
+ logger.error(f"Vocabulary path ({save_directory}) should be a directory")
159
+ return
160
+ out_vocab_file = os.path.join(
161
+ save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
162
+ )
163
+
164
+ if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
165
+ copyfile(self.vocab_file, out_vocab_file)
166
+ elif not os.path.isfile(self.vocab_file):
167
+ with open(out_vocab_file, "wb") as fi:
168
+ content_spiece_model = self.sp_model.serialized_model_proto()
169
+ fi.write(content_spiece_model)
170
+
171
+ return (out_vocab_file,)
172
+
173
+ def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
174
+ if self.add_bos_token:
175
+ bos_token_ids = [self.bos_token_id]
176
+ else:
177
+ bos_token_ids = []
178
+
179
+ output = bos_token_ids + token_ids_0
180
+
181
+ if token_ids_1 is not None:
182
+ output = output + token_ids_1
183
+
184
+ if self.add_eos_token:
185
+ output = output + [self.eos_token_id]
186
+
187
+ return output
188
+
189
+ def get_special_tokens_mask(
190
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
191
+ ) -> List[int]:
192
+ """
193
+ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
194
+ special tokens using the tokenizer `prepare_for_model` method.
195
+
196
+ Args:
197
+ token_ids_0 (`List[int]`):
198
+ List of IDs.
199
+ token_ids_1 (`List[int]`, *optional*):
200
+ Optional second list of IDs for sequence pairs.
201
+ already_has_special_tokens (`bool`, *optional*, defaults to `False`):
202
+ Whether or not the token list is already formatted with special tokens for the model.
203
+
204
+ Returns:
205
+ `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
206
+ """
207
+ if already_has_special_tokens:
208
+ return super().get_special_tokens_mask(
209
+ token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
210
+ )
211
+
212
+ if token_ids_1 is None:
213
+ return [1] + ([0] * len(token_ids_0)) + [1]
214
+ return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
215
+
216
+ def create_token_type_ids_from_sequences(
217
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
218
+ ) -> List[int]:
219
+ """
220
+ Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
221
+ use of token type ids, therefore a list of zeros is returned.
222
+
223
+ Args:
224
+ token_ids_0 (`List[int]`):
225
+ List of IDs.
226
+ token_ids_1 (`List[int]`, *optional*):
227
+ Optional second list of IDs for sequence pairs.
228
+
229
+ Returns:
230
+ `List[int]`: List of zeros.
231
+ """
232
+ eos = [self.eos_token_id]
233
+
234
+ if token_ids_1 is None:
235
+ return len(token_ids_0 + eos) * [0]
236
+ return len(token_ids_0 + eos + token_ids_1 + eos) * [0]
tokenization_internlm2_fast.py ADDED
@@ -0,0 +1,214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on transformers/src/transformers/models/llama/tokenization_llama_fast.py
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+
18
+ """Tokenization Fast class for InternLM."""
19
+ import os
20
+ from shutil import copyfile
21
+ from typing import Any, Dict, Optional, Tuple
22
+
23
+ from tokenizers import processors, decoders, Tokenizer, normalizers
24
+ from tokenizers.models import BPE
25
+
26
+ from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
27
+ from transformers.utils import logging
28
+
29
+ from transformers.convert_slow_tokenizer import (
30
+ SLOW_TO_FAST_CONVERTERS,
31
+ SpmConverter,
32
+ SentencePieceExtractor,
33
+ )
34
+
35
+ from .tokenization_internlm2 import InternLM2Tokenizer
36
+
37
+ logger = logging.get_logger(__name__)
38
+
39
+ VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
40
+
41
+ # Modified from transformers.convert_slow_tokenizer.LlamaConverter
42
+ class InternLM2Converter(SpmConverter):
43
+ handle_byte_fallback = True
44
+
45
+ def vocab(self, proto):
46
+ vocab = [
47
+ ("<unk>", 0.0),
48
+ ("<s>", 0.0),
49
+ ("</s>", 0.0),
50
+ ]
51
+ vocab += [(piece.piece, piece.score) for piece in proto.pieces[3:]]
52
+ return vocab
53
+
54
+ def unk_id(self, proto):
55
+ unk_id = 0
56
+ return unk_id
57
+
58
+ def decoder(self, replacement, add_prefix_space):
59
+ decoders_sequence = [
60
+ decoders.Replace("▁", " "),
61
+ decoders.ByteFallback(),
62
+ decoders.Fuse(),
63
+ ]
64
+ if self.proto.normalizer_spec.add_dummy_prefix:
65
+ decoders_sequence.append(decoders.Strip(content=" ", left=1))
66
+ return decoders.Sequence(decoders_sequence)
67
+
68
+ def tokenizer(self, proto):
69
+ model_type = proto.trainer_spec.model_type
70
+ vocab_scores = self.vocab(proto)
71
+ # special tokens
72
+ added_tokens = self.original_tokenizer.added_tokens_decoder
73
+ for i in range(len(vocab_scores)):
74
+ piece, score = vocab_scores[i]
75
+ if i in added_tokens:
76
+ vocab_scores[i] = (added_tokens[i].content, score)
77
+ if model_type == 1:
78
+ raise RuntimeError("InternLM2 is supposed to be a BPE model!")
79
+
80
+ elif model_type == 2:
81
+ _, merges = SentencePieceExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
82
+ bpe_vocab = {word: i for i, (word, _score) in enumerate(vocab_scores)}
83
+ tokenizer = Tokenizer(
84
+ BPE(bpe_vocab, merges, unk_token=proto.trainer_spec.unk_piece, fuse_unk=True, byte_fallback=True)
85
+ )
86
+ tokenizer.add_special_tokens(
87
+ [ added_token for index, added_token in added_tokens.items()]
88
+ )
89
+ else:
90
+ raise Exception(
91
+ "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
92
+ )
93
+
94
+ return tokenizer
95
+
96
+ def normalizer(self, proto):
97
+ normalizers_list = []
98
+ if proto.normalizer_spec.add_dummy_prefix:
99
+ normalizers_list.append(normalizers.Prepend(prepend="▁"))
100
+ normalizers_list.append(normalizers.Replace(pattern=" ", content="▁"))
101
+ return normalizers.Sequence(normalizers_list)
102
+
103
+ def pre_tokenizer(self, replacement, add_prefix_space):
104
+ return None
105
+
106
+ SLOW_TO_FAST_CONVERTERS["InternLM2Tokenizer"] = InternLM2Converter
107
+
108
+
109
+ # Modified from transformers.model.llama.tokenization_llama_fast.LlamaTokenizerFast -> InternLM2TokenizerFast
110
+ class InternLM2TokenizerFast(PreTrainedTokenizerFast):
111
+ vocab_files_names = VOCAB_FILES_NAMES
112
+ slow_tokenizer_class = InternLM2Tokenizer
113
+ padding_side = "left"
114
+ model_input_names = ["input_ids", "attention_mask"]
115
+ _auto_class = "AutoTokenizer"
116
+
117
+ def __init__(
118
+ self,
119
+ vocab_file,
120
+ unk_token="<unk>",
121
+ bos_token="<s>",
122
+ eos_token="</s>",
123
+ pad_token="</s>",
124
+ sp_model_kwargs: Optional[Dict[str, Any]] = None,
125
+ add_bos_token=True,
126
+ add_eos_token=False,
127
+ decode_with_prefix_space=False,
128
+ clean_up_tokenization_spaces=False,
129
+ **kwargs,
130
+ ):
131
+ super().__init__(
132
+ vocab_file=vocab_file,
133
+ unk_token=unk_token,
134
+ bos_token=bos_token,
135
+ eos_token=eos_token,
136
+ pad_token=pad_token,
137
+ sp_model_kwargs=sp_model_kwargs,
138
+ add_bos_token=add_bos_token,
139
+ add_eos_token=add_eos_token,
140
+ decode_with_prefix_space=decode_with_prefix_space,
141
+ clean_up_tokenization_spaces=clean_up_tokenization_spaces,
142
+ **kwargs,
143
+ )
144
+ self._add_bos_token = add_bos_token
145
+ self._add_eos_token = add_eos_token
146
+ self.update_post_processor()
147
+ self.vocab_file = vocab_file
148
+
149
+ @property
150
+ def can_save_slow_tokenizer(self) -> bool:
151
+ return os.path.isfile(self.vocab_file) if self.vocab_file else False
152
+
153
+ def update_post_processor(self):
154
+ """
155
+ Updates the underlying post processor with the current `bos_token` and `eos_token`.
156
+ """
157
+ bos = self.bos_token
158
+ bos_token_id = self.bos_token_id
159
+ if bos is None and self.add_bos_token:
160
+ raise ValueError("add_bos_token = True but bos_token = None")
161
+
162
+ eos = self.eos_token
163
+ eos_token_id = self.eos_token_id
164
+ if eos is None and self.add_eos_token:
165
+ raise ValueError("add_eos_token = True but eos_token = None")
166
+
167
+ single = f"{(bos+':0 ') if self.add_bos_token else ''}$A:0{(' '+eos+':0') if self.add_eos_token else ''}"
168
+ pair = f"{single}{(' '+bos+':1') if self.add_bos_token else ''} $B:1{(' '+eos+':1') if self.add_eos_token else ''}"
169
+
170
+ special_tokens = []
171
+ if self.add_bos_token:
172
+ special_tokens.append((bos, bos_token_id))
173
+ if self.add_eos_token:
174
+ special_tokens.append((eos, eos_token_id))
175
+ self._tokenizer.post_processor = processors.TemplateProcessing(
176
+ single=single, pair=pair, special_tokens=special_tokens
177
+ )
178
+
179
+ @property
180
+ def add_eos_token(self):
181
+ return self._add_eos_token
182
+
183
+ @property
184
+ def add_bos_token(self):
185
+ return self._add_bos_token
186
+
187
+ @add_eos_token.setter
188
+ def add_eos_token(self, value):
189
+ self._add_eos_token = value
190
+ self.update_post_processor()
191
+
192
+ @add_bos_token.setter
193
+ def add_bos_token(self, value):
194
+ self._add_bos_token = value
195
+ self.update_post_processor()
196
+
197
+ def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
198
+ if not self.can_save_slow_tokenizer:
199
+ raise ValueError(
200
+ "Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
201
+ "tokenizer."
202
+ )
203
+
204
+ if not os.path.isdir(save_directory):
205
+ logger.error(f"Vocabulary path ({save_directory}) should be a directory")
206
+ return
207
+ out_vocab_file = os.path.join(
208
+ save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
209
+ )
210
+
211
+ if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
212
+ copyfile(self.vocab_file, out_vocab_file)
213
+
214
+ return (out_vocab_file,)
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f868398fc4e05ee1e8aeba95ddf18ddcc45b8bce55d5093bead5bbf80429b48b
3
+ size 1477754
tokenizer_config.json ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoTokenizer": [
4
+ "tokenization_internlm2.InternLM2Tokenizer",
5
+ "tokenization_internlm2_fast.InternLM2TokenizerFast"
6
+ ]
7
+ },
8
+ "bos_token": "<s>",
9
+ "clean_up_tokenization_spaces": false,
10
+ "eos_token": "</s>",
11
+ "model_max_length": 1000000000000000019884624838656,
12
+ "pad_token": "</s>",
13
+ "tokenizer_class": "InternLM2Tokenizer",
14
+ "unk_token": "<unk>",
15
+ "added_tokens_decoder": {
16
+ "0": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false,
22
+ "special": true
23
+ },
24
+ "1": {
25
+ "content": "<s>",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false,
30
+ "special": true
31
+ },
32
+ "2": {
33
+ "content": "</s>",
34
+ "lstrip": false,
35
+ "normalized": false,
36
+ "rstrip": false,
37
+ "single_word": false,
38
+ "special": true
39
+ },
40
+ "92543": {
41
+ "content": "<|im_start|>",
42
+ "lstrip": false,
43
+ "normalized": false,
44
+ "rstrip": false,
45
+ "single_word": false,
46
+ "special": true
47
+ },
48
+ "92542": {
49
+ "content": "<|im_end|>",
50
+ "lstrip": false,
51
+ "normalized": false,
52
+ "rstrip": false,
53
+ "single_word": false,
54
+ "special": true
55
+ },
56
+ "92541": {
57
+ "content": "<|action_start|>",
58
+ "lstrip": false,
59
+ "normalized": false,
60
+ "rstrip": false,
61
+ "single_word": false,
62
+ "special": true
63
+ },
64
+ "92540": {
65
+ "content": "<|action_end|>",
66
+ "lstrip": false,
67
+ "normalized": false,
68
+ "rstrip": false,
69
+ "single_word": false,
70
+ "special": true
71
+ },
72
+ "92539": {
73
+ "content": "<|interpreter|>",
74
+ "lstrip": false,
75
+ "normalized": false,
76
+ "rstrip": false,
77
+ "single_word": false,
78
+ "special": true
79
+ },
80
+ "92538": {
81
+ "content": "<|plugin|>",
82
+ "lstrip": false,
83
+ "normalized": false,
84
+ "rstrip": false,
85
+ "single_word": false,
86
+ "special": true
87
+ }
88
+ },
89
+ "chat_template": "{{ bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
90
+ }