rooa commited on
Commit
9aaa0ce
1 Parent(s): 229aec5
README.md CHANGED
@@ -1,3 +1,107 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ # CodeGen2 (CodeGen2-7B)
6
+
7
+ ## Model description
8
+
9
+ [CodeGen2](https://github.com/salesforce/CodeGen2) is a family of autoregressive language models for **program synthesis**, introduced in the paper:
10
+
11
+ [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages]() by Erik Nijkamp\*, Hiroaki Hayashi\*, Caiming Xiong, Silvio Savarese, Yingbo Zhou.
12
+
13
+ Unlike the original CodeGen model family (i.e., CodeGen1), CodeGen2 is capable of infilling, and supports more programming languages.
14
+
15
+ Four model sizes are released: `1B`, `3.7B`, `7B`, `16B`.
16
+
17
+ ## How to use
18
+
19
+ This model can be easily loaded using the `AutoModelForCausalLM` functionality.
20
+
21
+ ### Causal sampling
22
+
23
+ For regular causal sampling, simply generate completions given the context:
24
+ ```python
25
+ from transformers import AutoTokenizer, AutoModelForCausalLM
26
+ tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen2-7B")
27
+ model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen2-7B", trust_remote_code=True, revision="main")
28
+
29
+ text = "def hello_world():"
30
+ input_ids = tokenizer(text, return_tensors="pt").input_ids
31
+ generated_ids = model.generate(input_ids, max_length=128)
32
+ print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
33
+ ```
34
+
35
+ ### Infill sampling
36
+
37
+ For **infill** sampling, we introduce three new special token types:
38
+
39
+ * `<mask_N>`: N-th span to be masked. In practice, use `<mask_1>` to where you want to sample infill.
40
+ * `<sep>`: Seperator token between the suffix and the infilled sample. See below.
41
+ * `<eom>`: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.
42
+
43
+ For example, if we want to generate infill for the following cursor position of a function:
44
+ ```python
45
+ def hello_world():
46
+ |
47
+ return name
48
+ ```
49
+ we construct an input to the model by
50
+
51
+ 1. Inserting `<mask_1>` token in place of cursor position
52
+ 2. Append `<sep>` token to indicate the boundary
53
+ 3. Insert another `<mask_1>` to indicate which mask we want to infill.
54
+
55
+ The final snippet looks as follows:
56
+
57
+ ```python
58
+ from transformers import AutoTokenizer, AutoModelForCausalLM
59
+ tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen2-7B")
60
+ model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen2-7B")
61
+
62
+
63
+ def format(prefix, suffix):
64
+ return prefix + "<mask_1>" + suffix + "<|endoftext|>" + "<sep>" + "<mask_1>"
65
+
66
+
67
+ prefix = "def hello_world():\n "
68
+ suffix = " return name"
69
+ text = format(prefix, suffix)
70
+ input_ids = tokenizer(text, return_tensors="pt").input_ids
71
+ generated_ids = model.generate(input_ids, max_length=128)
72
+ print(tokenizer.decode(generated_ids[0], skip_special_tokens=False)[len(text):])
73
+ ```
74
+
75
+ You might want to truncate the model output with `<eom>`.
76
+
77
+ ## Training data
78
+
79
+ This checkpoint is trained on the stricter permissive subset of [the deduplicated version of the Stack dataset (v1.1)](). Supported languages (and frameworks) are as follows:
80
+ `c`, `c++`, `c-sharp`, `dart`, `go`, `java`, `javascript`, `kotlin`, `lua`, `php`, `python`, `ruby`, `rust`, `scala`, `shell`, `sql`, `swift`, `typescript`, `vue`.
81
+
82
+ ## Training procedure
83
+
84
+ CodeGen2 was trained using cross-entropy loss to maximize the likelihood of sequential inputs.
85
+ The input sequences are formatted in two ways: (1) causal language modeling and (2) file-level span corruption.
86
+ Please refer to the paper for more details.
87
+
88
+ ## Evaluation results
89
+
90
+ We evaluate our models on HumanEval and HumanEval-Infill. Please refer to the [paper]() for more details.
91
+
92
+ ## Intended use and limitations
93
+
94
+ As an autoregressive language model, CodeGen2 is capable of extracting features from given natural language and programming language texts, and calculating the likelihood of them.
95
+ However, the model is intended for and best at **program synthesis**, that is, generating executable code given English prompts, where the prompts should be in the form of a comment string. The model can complete partially-generated code as well.
96
+
97
+
98
+ ## BibTeX entry and citation info
99
+
100
+ ```bibtex
101
+ @article{Nijkamp2023codegen2,
102
+ title={CodeGen2: Lessons for Training LLMs on Programming and Natural Languages},
103
+ author={Nijkamp, Erik and Hayashi, Hiroaki and Xiong, Caiming and Savarese, Silvio and Zhou, Yingbo},
104
+ journal={arXiv preprint},
105
+ year={2022}
106
+ }
107
+ ```
added_tokens.json ADDED
@@ -0,0 +1,945 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "\t\t": 50294,
3
+ "\t\t\t": 50293,
4
+ "\t\t\t\t": 50292,
5
+ "\t\t\t\t\t": 50291,
6
+ "\t\t\t\t\t\t": 50290,
7
+ "\t\t\t\t\t\t\t": 50289,
8
+ "\t\t\t\t\t\t\t\t": 50288,
9
+ "\t\t\t\t\t\t\t\t\t": 50287,
10
+ " ": 50286,
11
+ " ": 50285,
12
+ " ": 50284,
13
+ " ": 50283,
14
+ " ": 50282,
15
+ " ": 50281,
16
+ " ": 50280,
17
+ " ": 50279,
18
+ " ": 50278,
19
+ " ": 50277,
20
+ " ": 50276,
21
+ " ": 50275,
22
+ " ": 50274,
23
+ " ": 50273,
24
+ " ": 50272,
25
+ " ": 50271,
26
+ " ": 50270,
27
+ " ": 50269,
28
+ " ": 50268,
29
+ " ": 50267,
30
+ " ": 50266,
31
+ " ": 50265,
32
+ " ": 50264,
33
+ " ": 50263,
34
+ " ": 50262,
35
+ " ": 50261,
36
+ " ": 50260,
37
+ " ": 50259,
38
+ " ": 50258,
39
+ " ": 50257,
40
+ "<dummy_0>": 50295,
41
+ "<dummy_1>": 50296,
42
+ "<dummy_2>": 50297,
43
+ "<dummy_3>": 50298,
44
+ "<eom>": 50300,
45
+ "<mask_100>": 51100,
46
+ "<mask_101>": 51099,
47
+ "<mask_102>": 51098,
48
+ "<mask_103>": 51097,
49
+ "<mask_104>": 51096,
50
+ "<mask_105>": 51095,
51
+ "<mask_106>": 51094,
52
+ "<mask_107>": 51093,
53
+ "<mask_108>": 51092,
54
+ "<mask_109>": 51091,
55
+ "<mask_10>": 51190,
56
+ "<mask_110>": 51090,
57
+ "<mask_111>": 51089,
58
+ "<mask_112>": 51088,
59
+ "<mask_113>": 51087,
60
+ "<mask_114>": 51086,
61
+ "<mask_115>": 51085,
62
+ "<mask_116>": 51084,
63
+ "<mask_117>": 51083,
64
+ "<mask_118>": 51082,
65
+ "<mask_119>": 51081,
66
+ "<mask_11>": 51189,
67
+ "<mask_120>": 51080,
68
+ "<mask_121>": 51079,
69
+ "<mask_122>": 51078,
70
+ "<mask_123>": 51077,
71
+ "<mask_124>": 51076,
72
+ "<mask_125>": 51075,
73
+ "<mask_126>": 51074,
74
+ "<mask_127>": 51073,
75
+ "<mask_128>": 51072,
76
+ "<mask_129>": 51071,
77
+ "<mask_12>": 51188,
78
+ "<mask_130>": 51070,
79
+ "<mask_131>": 51069,
80
+ "<mask_132>": 51068,
81
+ "<mask_133>": 51067,
82
+ "<mask_134>": 51066,
83
+ "<mask_135>": 51065,
84
+ "<mask_136>": 51064,
85
+ "<mask_137>": 51063,
86
+ "<mask_138>": 51062,
87
+ "<mask_139>": 51061,
88
+ "<mask_13>": 51187,
89
+ "<mask_140>": 51060,
90
+ "<mask_141>": 51059,
91
+ "<mask_142>": 51058,
92
+ "<mask_143>": 51057,
93
+ "<mask_144>": 51056,
94
+ "<mask_145>": 51055,
95
+ "<mask_146>": 51054,
96
+ "<mask_147>": 51053,
97
+ "<mask_148>": 51052,
98
+ "<mask_149>": 51051,
99
+ "<mask_14>": 51186,
100
+ "<mask_150>": 51050,
101
+ "<mask_151>": 51049,
102
+ "<mask_152>": 51048,
103
+ "<mask_153>": 51047,
104
+ "<mask_154>": 51046,
105
+ "<mask_155>": 51045,
106
+ "<mask_156>": 51044,
107
+ "<mask_157>": 51043,
108
+ "<mask_158>": 51042,
109
+ "<mask_159>": 51041,
110
+ "<mask_15>": 51185,
111
+ "<mask_160>": 51040,
112
+ "<mask_161>": 51039,
113
+ "<mask_162>": 51038,
114
+ "<mask_163>": 51037,
115
+ "<mask_164>": 51036,
116
+ "<mask_165>": 51035,
117
+ "<mask_166>": 51034,
118
+ "<mask_167>": 51033,
119
+ "<mask_168>": 51032,
120
+ "<mask_169>": 51031,
121
+ "<mask_16>": 51184,
122
+ "<mask_170>": 51030,
123
+ "<mask_171>": 51029,
124
+ "<mask_172>": 51028,
125
+ "<mask_173>": 51027,
126
+ "<mask_174>": 51026,
127
+ "<mask_175>": 51025,
128
+ "<mask_176>": 51024,
129
+ "<mask_177>": 51023,
130
+ "<mask_178>": 51022,
131
+ "<mask_179>": 51021,
132
+ "<mask_17>": 51183,
133
+ "<mask_180>": 51020,
134
+ "<mask_181>": 51019,
135
+ "<mask_182>": 51018,
136
+ "<mask_183>": 51017,
137
+ "<mask_184>": 51016,
138
+ "<mask_185>": 51015,
139
+ "<mask_186>": 51014,
140
+ "<mask_187>": 51013,
141
+ "<mask_188>": 51012,
142
+ "<mask_189>": 51011,
143
+ "<mask_18>": 51182,
144
+ "<mask_190>": 51010,
145
+ "<mask_191>": 51009,
146
+ "<mask_192>": 51008,
147
+ "<mask_193>": 51007,
148
+ "<mask_194>": 51006,
149
+ "<mask_195>": 51005,
150
+ "<mask_196>": 51004,
151
+ "<mask_197>": 51003,
152
+ "<mask_198>": 51002,
153
+ "<mask_199>": 51001,
154
+ "<mask_19>": 51181,
155
+ "<mask_1>": 51199,
156
+ "<mask_200>": 51000,
157
+ "<mask_201>": 50999,
158
+ "<mask_202>": 50998,
159
+ "<mask_203>": 50997,
160
+ "<mask_204>": 50996,
161
+ "<mask_205>": 50995,
162
+ "<mask_206>": 50994,
163
+ "<mask_207>": 50993,
164
+ "<mask_208>": 50992,
165
+ "<mask_209>": 50991,
166
+ "<mask_20>": 51180,
167
+ "<mask_210>": 50990,
168
+ "<mask_211>": 50989,
169
+ "<mask_212>": 50988,
170
+ "<mask_213>": 50987,
171
+ "<mask_214>": 50986,
172
+ "<mask_215>": 50985,
173
+ "<mask_216>": 50984,
174
+ "<mask_217>": 50983,
175
+ "<mask_218>": 50982,
176
+ "<mask_219>": 50981,
177
+ "<mask_21>": 51179,
178
+ "<mask_220>": 50980,
179
+ "<mask_221>": 50979,
180
+ "<mask_222>": 50978,
181
+ "<mask_223>": 50977,
182
+ "<mask_224>": 50976,
183
+ "<mask_225>": 50975,
184
+ "<mask_226>": 50974,
185
+ "<mask_227>": 50973,
186
+ "<mask_228>": 50972,
187
+ "<mask_229>": 50971,
188
+ "<mask_22>": 51178,
189
+ "<mask_230>": 50970,
190
+ "<mask_231>": 50969,
191
+ "<mask_232>": 50968,
192
+ "<mask_233>": 50967,
193
+ "<mask_234>": 50966,
194
+ "<mask_235>": 50965,
195
+ "<mask_236>": 50964,
196
+ "<mask_237>": 50963,
197
+ "<mask_238>": 50962,
198
+ "<mask_239>": 50961,
199
+ "<mask_23>": 51177,
200
+ "<mask_240>": 50960,
201
+ "<mask_241>": 50959,
202
+ "<mask_242>": 50958,
203
+ "<mask_243>": 50957,
204
+ "<mask_244>": 50956,
205
+ "<mask_245>": 50955,
206
+ "<mask_246>": 50954,
207
+ "<mask_247>": 50953,
208
+ "<mask_248>": 50952,
209
+ "<mask_249>": 50951,
210
+ "<mask_24>": 51176,
211
+ "<mask_250>": 50950,
212
+ "<mask_251>": 50949,
213
+ "<mask_252>": 50948,
214
+ "<mask_253>": 50947,
215
+ "<mask_254>": 50946,
216
+ "<mask_255>": 50945,
217
+ "<mask_256>": 50944,
218
+ "<mask_257>": 50943,
219
+ "<mask_258>": 50942,
220
+ "<mask_259>": 50941,
221
+ "<mask_25>": 51175,
222
+ "<mask_260>": 50940,
223
+ "<mask_261>": 50939,
224
+ "<mask_262>": 50938,
225
+ "<mask_263>": 50937,
226
+ "<mask_264>": 50936,
227
+ "<mask_265>": 50935,
228
+ "<mask_266>": 50934,
229
+ "<mask_267>": 50933,
230
+ "<mask_268>": 50932,
231
+ "<mask_269>": 50931,
232
+ "<mask_26>": 51174,
233
+ "<mask_270>": 50930,
234
+ "<mask_271>": 50929,
235
+ "<mask_272>": 50928,
236
+ "<mask_273>": 50927,
237
+ "<mask_274>": 50926,
238
+ "<mask_275>": 50925,
239
+ "<mask_276>": 50924,
240
+ "<mask_277>": 50923,
241
+ "<mask_278>": 50922,
242
+ "<mask_279>": 50921,
243
+ "<mask_27>": 51173,
244
+ "<mask_280>": 50920,
245
+ "<mask_281>": 50919,
246
+ "<mask_282>": 50918,
247
+ "<mask_283>": 50917,
248
+ "<mask_284>": 50916,
249
+ "<mask_285>": 50915,
250
+ "<mask_286>": 50914,
251
+ "<mask_287>": 50913,
252
+ "<mask_288>": 50912,
253
+ "<mask_289>": 50911,
254
+ "<mask_28>": 51172,
255
+ "<mask_290>": 50910,
256
+ "<mask_291>": 50909,
257
+ "<mask_292>": 50908,
258
+ "<mask_293>": 50907,
259
+ "<mask_294>": 50906,
260
+ "<mask_295>": 50905,
261
+ "<mask_296>": 50904,
262
+ "<mask_297>": 50903,
263
+ "<mask_298>": 50902,
264
+ "<mask_299>": 50901,
265
+ "<mask_29>": 51171,
266
+ "<mask_2>": 51198,
267
+ "<mask_300>": 50900,
268
+ "<mask_301>": 50899,
269
+ "<mask_302>": 50898,
270
+ "<mask_303>": 50897,
271
+ "<mask_304>": 50896,
272
+ "<mask_305>": 50895,
273
+ "<mask_306>": 50894,
274
+ "<mask_307>": 50893,
275
+ "<mask_308>": 50892,
276
+ "<mask_309>": 50891,
277
+ "<mask_30>": 51170,
278
+ "<mask_310>": 50890,
279
+ "<mask_311>": 50889,
280
+ "<mask_312>": 50888,
281
+ "<mask_313>": 50887,
282
+ "<mask_314>": 50886,
283
+ "<mask_315>": 50885,
284
+ "<mask_316>": 50884,
285
+ "<mask_317>": 50883,
286
+ "<mask_318>": 50882,
287
+ "<mask_319>": 50881,
288
+ "<mask_31>": 51169,
289
+ "<mask_320>": 50880,
290
+ "<mask_321>": 50879,
291
+ "<mask_322>": 50878,
292
+ "<mask_323>": 50877,
293
+ "<mask_324>": 50876,
294
+ "<mask_325>": 50875,
295
+ "<mask_326>": 50874,
296
+ "<mask_327>": 50873,
297
+ "<mask_328>": 50872,
298
+ "<mask_329>": 50871,
299
+ "<mask_32>": 51168,
300
+ "<mask_330>": 50870,
301
+ "<mask_331>": 50869,
302
+ "<mask_332>": 50868,
303
+ "<mask_333>": 50867,
304
+ "<mask_334>": 50866,
305
+ "<mask_335>": 50865,
306
+ "<mask_336>": 50864,
307
+ "<mask_337>": 50863,
308
+ "<mask_338>": 50862,
309
+ "<mask_339>": 50861,
310
+ "<mask_33>": 51167,
311
+ "<mask_340>": 50860,
312
+ "<mask_341>": 50859,
313
+ "<mask_342>": 50858,
314
+ "<mask_343>": 50857,
315
+ "<mask_344>": 50856,
316
+ "<mask_345>": 50855,
317
+ "<mask_346>": 50854,
318
+ "<mask_347>": 50853,
319
+ "<mask_348>": 50852,
320
+ "<mask_349>": 50851,
321
+ "<mask_34>": 51166,
322
+ "<mask_350>": 50850,
323
+ "<mask_351>": 50849,
324
+ "<mask_352>": 50848,
325
+ "<mask_353>": 50847,
326
+ "<mask_354>": 50846,
327
+ "<mask_355>": 50845,
328
+ "<mask_356>": 50844,
329
+ "<mask_357>": 50843,
330
+ "<mask_358>": 50842,
331
+ "<mask_359>": 50841,
332
+ "<mask_35>": 51165,
333
+ "<mask_360>": 50840,
334
+ "<mask_361>": 50839,
335
+ "<mask_362>": 50838,
336
+ "<mask_363>": 50837,
337
+ "<mask_364>": 50836,
338
+ "<mask_365>": 50835,
339
+ "<mask_366>": 50834,
340
+ "<mask_367>": 50833,
341
+ "<mask_368>": 50832,
342
+ "<mask_369>": 50831,
343
+ "<mask_36>": 51164,
344
+ "<mask_370>": 50830,
345
+ "<mask_371>": 50829,
346
+ "<mask_372>": 50828,
347
+ "<mask_373>": 50827,
348
+ "<mask_374>": 50826,
349
+ "<mask_375>": 50825,
350
+ "<mask_376>": 50824,
351
+ "<mask_377>": 50823,
352
+ "<mask_378>": 50822,
353
+ "<mask_379>": 50821,
354
+ "<mask_37>": 51163,
355
+ "<mask_380>": 50820,
356
+ "<mask_381>": 50819,
357
+ "<mask_382>": 50818,
358
+ "<mask_383>": 50817,
359
+ "<mask_384>": 50816,
360
+ "<mask_385>": 50815,
361
+ "<mask_386>": 50814,
362
+ "<mask_387>": 50813,
363
+ "<mask_388>": 50812,
364
+ "<mask_389>": 50811,
365
+ "<mask_38>": 51162,
366
+ "<mask_390>": 50810,
367
+ "<mask_391>": 50809,
368
+ "<mask_392>": 50808,
369
+ "<mask_393>": 50807,
370
+ "<mask_394>": 50806,
371
+ "<mask_395>": 50805,
372
+ "<mask_396>": 50804,
373
+ "<mask_397>": 50803,
374
+ "<mask_398>": 50802,
375
+ "<mask_399>": 50801,
376
+ "<mask_39>": 51161,
377
+ "<mask_3>": 51197,
378
+ "<mask_400>": 50800,
379
+ "<mask_401>": 50799,
380
+ "<mask_402>": 50798,
381
+ "<mask_403>": 50797,
382
+ "<mask_404>": 50796,
383
+ "<mask_405>": 50795,
384
+ "<mask_406>": 50794,
385
+ "<mask_407>": 50793,
386
+ "<mask_408>": 50792,
387
+ "<mask_409>": 50791,
388
+ "<mask_40>": 51160,
389
+ "<mask_410>": 50790,
390
+ "<mask_411>": 50789,
391
+ "<mask_412>": 50788,
392
+ "<mask_413>": 50787,
393
+ "<mask_414>": 50786,
394
+ "<mask_415>": 50785,
395
+ "<mask_416>": 50784,
396
+ "<mask_417>": 50783,
397
+ "<mask_418>": 50782,
398
+ "<mask_419>": 50781,
399
+ "<mask_41>": 51159,
400
+ "<mask_420>": 50780,
401
+ "<mask_421>": 50779,
402
+ "<mask_422>": 50778,
403
+ "<mask_423>": 50777,
404
+ "<mask_424>": 50776,
405
+ "<mask_425>": 50775,
406
+ "<mask_426>": 50774,
407
+ "<mask_427>": 50773,
408
+ "<mask_428>": 50772,
409
+ "<mask_429>": 50771,
410
+ "<mask_42>": 51158,
411
+ "<mask_430>": 50770,
412
+ "<mask_431>": 50769,
413
+ "<mask_432>": 50768,
414
+ "<mask_433>": 50767,
415
+ "<mask_434>": 50766,
416
+ "<mask_435>": 50765,
417
+ "<mask_436>": 50764,
418
+ "<mask_437>": 50763,
419
+ "<mask_438>": 50762,
420
+ "<mask_439>": 50761,
421
+ "<mask_43>": 51157,
422
+ "<mask_440>": 50760,
423
+ "<mask_441>": 50759,
424
+ "<mask_442>": 50758,
425
+ "<mask_443>": 50757,
426
+ "<mask_444>": 50756,
427
+ "<mask_445>": 50755,
428
+ "<mask_446>": 50754,
429
+ "<mask_447>": 50753,
430
+ "<mask_448>": 50752,
431
+ "<mask_449>": 50751,
432
+ "<mask_44>": 51156,
433
+ "<mask_450>": 50750,
434
+ "<mask_451>": 50749,
435
+ "<mask_452>": 50748,
436
+ "<mask_453>": 50747,
437
+ "<mask_454>": 50746,
438
+ "<mask_455>": 50745,
439
+ "<mask_456>": 50744,
440
+ "<mask_457>": 50743,
441
+ "<mask_458>": 50742,
442
+ "<mask_459>": 50741,
443
+ "<mask_45>": 51155,
444
+ "<mask_460>": 50740,
445
+ "<mask_461>": 50739,
446
+ "<mask_462>": 50738,
447
+ "<mask_463>": 50737,
448
+ "<mask_464>": 50736,
449
+ "<mask_465>": 50735,
450
+ "<mask_466>": 50734,
451
+ "<mask_467>": 50733,
452
+ "<mask_468>": 50732,
453
+ "<mask_469>": 50731,
454
+ "<mask_46>": 51154,
455
+ "<mask_470>": 50730,
456
+ "<mask_471>": 50729,
457
+ "<mask_472>": 50728,
458
+ "<mask_473>": 50727,
459
+ "<mask_474>": 50726,
460
+ "<mask_475>": 50725,
461
+ "<mask_476>": 50724,
462
+ "<mask_477>": 50723,
463
+ "<mask_478>": 50722,
464
+ "<mask_479>": 50721,
465
+ "<mask_47>": 51153,
466
+ "<mask_480>": 50720,
467
+ "<mask_481>": 50719,
468
+ "<mask_482>": 50718,
469
+ "<mask_483>": 50717,
470
+ "<mask_484>": 50716,
471
+ "<mask_485>": 50715,
472
+ "<mask_486>": 50714,
473
+ "<mask_487>": 50713,
474
+ "<mask_488>": 50712,
475
+ "<mask_489>": 50711,
476
+ "<mask_48>": 51152,
477
+ "<mask_490>": 50710,
478
+ "<mask_491>": 50709,
479
+ "<mask_492>": 50708,
480
+ "<mask_493>": 50707,
481
+ "<mask_494>": 50706,
482
+ "<mask_495>": 50705,
483
+ "<mask_496>": 50704,
484
+ "<mask_497>": 50703,
485
+ "<mask_498>": 50702,
486
+ "<mask_499>": 50701,
487
+ "<mask_49>": 51151,
488
+ "<mask_4>": 51196,
489
+ "<mask_500>": 50700,
490
+ "<mask_501>": 50699,
491
+ "<mask_502>": 50698,
492
+ "<mask_503>": 50697,
493
+ "<mask_504>": 50696,
494
+ "<mask_505>": 50695,
495
+ "<mask_506>": 50694,
496
+ "<mask_507>": 50693,
497
+ "<mask_508>": 50692,
498
+ "<mask_509>": 50691,
499
+ "<mask_50>": 51150,
500
+ "<mask_510>": 50690,
501
+ "<mask_511>": 50689,
502
+ "<mask_512>": 50688,
503
+ "<mask_513>": 50687,
504
+ "<mask_514>": 50686,
505
+ "<mask_515>": 50685,
506
+ "<mask_516>": 50684,
507
+ "<mask_517>": 50683,
508
+ "<mask_518>": 50682,
509
+ "<mask_519>": 50681,
510
+ "<mask_51>": 51149,
511
+ "<mask_520>": 50680,
512
+ "<mask_521>": 50679,
513
+ "<mask_522>": 50678,
514
+ "<mask_523>": 50677,
515
+ "<mask_524>": 50676,
516
+ "<mask_525>": 50675,
517
+ "<mask_526>": 50674,
518
+ "<mask_527>": 50673,
519
+ "<mask_528>": 50672,
520
+ "<mask_529>": 50671,
521
+ "<mask_52>": 51148,
522
+ "<mask_530>": 50670,
523
+ "<mask_531>": 50669,
524
+ "<mask_532>": 50668,
525
+ "<mask_533>": 50667,
526
+ "<mask_534>": 50666,
527
+ "<mask_535>": 50665,
528
+ "<mask_536>": 50664,
529
+ "<mask_537>": 50663,
530
+ "<mask_538>": 50662,
531
+ "<mask_539>": 50661,
532
+ "<mask_53>": 51147,
533
+ "<mask_540>": 50660,
534
+ "<mask_541>": 50659,
535
+ "<mask_542>": 50658,
536
+ "<mask_543>": 50657,
537
+ "<mask_544>": 50656,
538
+ "<mask_545>": 50655,
539
+ "<mask_546>": 50654,
540
+ "<mask_547>": 50653,
541
+ "<mask_548>": 50652,
542
+ "<mask_549>": 50651,
543
+ "<mask_54>": 51146,
544
+ "<mask_550>": 50650,
545
+ "<mask_551>": 50649,
546
+ "<mask_552>": 50648,
547
+ "<mask_553>": 50647,
548
+ "<mask_554>": 50646,
549
+ "<mask_555>": 50645,
550
+ "<mask_556>": 50644,
551
+ "<mask_557>": 50643,
552
+ "<mask_558>": 50642,
553
+ "<mask_559>": 50641,
554
+ "<mask_55>": 51145,
555
+ "<mask_560>": 50640,
556
+ "<mask_561>": 50639,
557
+ "<mask_562>": 50638,
558
+ "<mask_563>": 50637,
559
+ "<mask_564>": 50636,
560
+ "<mask_565>": 50635,
561
+ "<mask_566>": 50634,
562
+ "<mask_567>": 50633,
563
+ "<mask_568>": 50632,
564
+ "<mask_569>": 50631,
565
+ "<mask_56>": 51144,
566
+ "<mask_570>": 50630,
567
+ "<mask_571>": 50629,
568
+ "<mask_572>": 50628,
569
+ "<mask_573>": 50627,
570
+ "<mask_574>": 50626,
571
+ "<mask_575>": 50625,
572
+ "<mask_576>": 50624,
573
+ "<mask_577>": 50623,
574
+ "<mask_578>": 50622,
575
+ "<mask_579>": 50621,
576
+ "<mask_57>": 51143,
577
+ "<mask_580>": 50620,
578
+ "<mask_581>": 50619,
579
+ "<mask_582>": 50618,
580
+ "<mask_583>": 50617,
581
+ "<mask_584>": 50616,
582
+ "<mask_585>": 50615,
583
+ "<mask_586>": 50614,
584
+ "<mask_587>": 50613,
585
+ "<mask_588>": 50612,
586
+ "<mask_589>": 50611,
587
+ "<mask_58>": 51142,
588
+ "<mask_590>": 50610,
589
+ "<mask_591>": 50609,
590
+ "<mask_592>": 50608,
591
+ "<mask_593>": 50607,
592
+ "<mask_594>": 50606,
593
+ "<mask_595>": 50605,
594
+ "<mask_596>": 50604,
595
+ "<mask_597>": 50603,
596
+ "<mask_598>": 50602,
597
+ "<mask_599>": 50601,
598
+ "<mask_59>": 51141,
599
+ "<mask_5>": 51195,
600
+ "<mask_600>": 50600,
601
+ "<mask_601>": 50599,
602
+ "<mask_602>": 50598,
603
+ "<mask_603>": 50597,
604
+ "<mask_604>": 50596,
605
+ "<mask_605>": 50595,
606
+ "<mask_606>": 50594,
607
+ "<mask_607>": 50593,
608
+ "<mask_608>": 50592,
609
+ "<mask_609>": 50591,
610
+ "<mask_60>": 51140,
611
+ "<mask_610>": 50590,
612
+ "<mask_611>": 50589,
613
+ "<mask_612>": 50588,
614
+ "<mask_613>": 50587,
615
+ "<mask_614>": 50586,
616
+ "<mask_615>": 50585,
617
+ "<mask_616>": 50584,
618
+ "<mask_617>": 50583,
619
+ "<mask_618>": 50582,
620
+ "<mask_619>": 50581,
621
+ "<mask_61>": 51139,
622
+ "<mask_620>": 50580,
623
+ "<mask_621>": 50579,
624
+ "<mask_622>": 50578,
625
+ "<mask_623>": 50577,
626
+ "<mask_624>": 50576,
627
+ "<mask_625>": 50575,
628
+ "<mask_626>": 50574,
629
+ "<mask_627>": 50573,
630
+ "<mask_628>": 50572,
631
+ "<mask_629>": 50571,
632
+ "<mask_62>": 51138,
633
+ "<mask_630>": 50570,
634
+ "<mask_631>": 50569,
635
+ "<mask_632>": 50568,
636
+ "<mask_633>": 50567,
637
+ "<mask_634>": 50566,
638
+ "<mask_635>": 50565,
639
+ "<mask_636>": 50564,
640
+ "<mask_637>": 50563,
641
+ "<mask_638>": 50562,
642
+ "<mask_639>": 50561,
643
+ "<mask_63>": 51137,
644
+ "<mask_640>": 50560,
645
+ "<mask_641>": 50559,
646
+ "<mask_642>": 50558,
647
+ "<mask_643>": 50557,
648
+ "<mask_644>": 50556,
649
+ "<mask_645>": 50555,
650
+ "<mask_646>": 50554,
651
+ "<mask_647>": 50553,
652
+ "<mask_648>": 50552,
653
+ "<mask_649>": 50551,
654
+ "<mask_64>": 51136,
655
+ "<mask_650>": 50550,
656
+ "<mask_651>": 50549,
657
+ "<mask_652>": 50548,
658
+ "<mask_653>": 50547,
659
+ "<mask_654>": 50546,
660
+ "<mask_655>": 50545,
661
+ "<mask_656>": 50544,
662
+ "<mask_657>": 50543,
663
+ "<mask_658>": 50542,
664
+ "<mask_659>": 50541,
665
+ "<mask_65>": 51135,
666
+ "<mask_660>": 50540,
667
+ "<mask_661>": 50539,
668
+ "<mask_662>": 50538,
669
+ "<mask_663>": 50537,
670
+ "<mask_664>": 50536,
671
+ "<mask_665>": 50535,
672
+ "<mask_666>": 50534,
673
+ "<mask_667>": 50533,
674
+ "<mask_668>": 50532,
675
+ "<mask_669>": 50531,
676
+ "<mask_66>": 51134,
677
+ "<mask_670>": 50530,
678
+ "<mask_671>": 50529,
679
+ "<mask_672>": 50528,
680
+ "<mask_673>": 50527,
681
+ "<mask_674>": 50526,
682
+ "<mask_675>": 50525,
683
+ "<mask_676>": 50524,
684
+ "<mask_677>": 50523,
685
+ "<mask_678>": 50522,
686
+ "<mask_679>": 50521,
687
+ "<mask_67>": 51133,
688
+ "<mask_680>": 50520,
689
+ "<mask_681>": 50519,
690
+ "<mask_682>": 50518,
691
+ "<mask_683>": 50517,
692
+ "<mask_684>": 50516,
693
+ "<mask_685>": 50515,
694
+ "<mask_686>": 50514,
695
+ "<mask_687>": 50513,
696
+ "<mask_688>": 50512,
697
+ "<mask_689>": 50511,
698
+ "<mask_68>": 51132,
699
+ "<mask_690>": 50510,
700
+ "<mask_691>": 50509,
701
+ "<mask_692>": 50508,
702
+ "<mask_693>": 50507,
703
+ "<mask_694>": 50506,
704
+ "<mask_695>": 50505,
705
+ "<mask_696>": 50504,
706
+ "<mask_697>": 50503,
707
+ "<mask_698>": 50502,
708
+ "<mask_699>": 50501,
709
+ "<mask_69>": 51131,
710
+ "<mask_6>": 51194,
711
+ "<mask_700>": 50500,
712
+ "<mask_701>": 50499,
713
+ "<mask_702>": 50498,
714
+ "<mask_703>": 50497,
715
+ "<mask_704>": 50496,
716
+ "<mask_705>": 50495,
717
+ "<mask_706>": 50494,
718
+ "<mask_707>": 50493,
719
+ "<mask_708>": 50492,
720
+ "<mask_709>": 50491,
721
+ "<mask_70>": 51130,
722
+ "<mask_710>": 50490,
723
+ "<mask_711>": 50489,
724
+ "<mask_712>": 50488,
725
+ "<mask_713>": 50487,
726
+ "<mask_714>": 50486,
727
+ "<mask_715>": 50485,
728
+ "<mask_716>": 50484,
729
+ "<mask_717>": 50483,
730
+ "<mask_718>": 50482,
731
+ "<mask_719>": 50481,
732
+ "<mask_71>": 51129,
733
+ "<mask_720>": 50480,
734
+ "<mask_721>": 50479,
735
+ "<mask_722>": 50478,
736
+ "<mask_723>": 50477,
737
+ "<mask_724>": 50476,
738
+ "<mask_725>": 50475,
739
+ "<mask_726>": 50474,
740
+ "<mask_727>": 50473,
741
+ "<mask_728>": 50472,
742
+ "<mask_729>": 50471,
743
+ "<mask_72>": 51128,
744
+ "<mask_730>": 50470,
745
+ "<mask_731>": 50469,
746
+ "<mask_732>": 50468,
747
+ "<mask_733>": 50467,
748
+ "<mask_734>": 50466,
749
+ "<mask_735>": 50465,
750
+ "<mask_736>": 50464,
751
+ "<mask_737>": 50463,
752
+ "<mask_738>": 50462,
753
+ "<mask_739>": 50461,
754
+ "<mask_73>": 51127,
755
+ "<mask_740>": 50460,
756
+ "<mask_741>": 50459,
757
+ "<mask_742>": 50458,
758
+ "<mask_743>": 50457,
759
+ "<mask_744>": 50456,
760
+ "<mask_745>": 50455,
761
+ "<mask_746>": 50454,
762
+ "<mask_747>": 50453,
763
+ "<mask_748>": 50452,
764
+ "<mask_749>": 50451,
765
+ "<mask_74>": 51126,
766
+ "<mask_750>": 50450,
767
+ "<mask_751>": 50449,
768
+ "<mask_752>": 50448,
769
+ "<mask_753>": 50447,
770
+ "<mask_754>": 50446,
771
+ "<mask_755>": 50445,
772
+ "<mask_756>": 50444,
773
+ "<mask_757>": 50443,
774
+ "<mask_758>": 50442,
775
+ "<mask_759>": 50441,
776
+ "<mask_75>": 51125,
777
+ "<mask_760>": 50440,
778
+ "<mask_761>": 50439,
779
+ "<mask_762>": 50438,
780
+ "<mask_763>": 50437,
781
+ "<mask_764>": 50436,
782
+ "<mask_765>": 50435,
783
+ "<mask_766>": 50434,
784
+ "<mask_767>": 50433,
785
+ "<mask_768>": 50432,
786
+ "<mask_769>": 50431,
787
+ "<mask_76>": 51124,
788
+ "<mask_770>": 50430,
789
+ "<mask_771>": 50429,
790
+ "<mask_772>": 50428,
791
+ "<mask_773>": 50427,
792
+ "<mask_774>": 50426,
793
+ "<mask_775>": 50425,
794
+ "<mask_776>": 50424,
795
+ "<mask_777>": 50423,
796
+ "<mask_778>": 50422,
797
+ "<mask_779>": 50421,
798
+ "<mask_77>": 51123,
799
+ "<mask_780>": 50420,
800
+ "<mask_781>": 50419,
801
+ "<mask_782>": 50418,
802
+ "<mask_783>": 50417,
803
+ "<mask_784>": 50416,
804
+ "<mask_785>": 50415,
805
+ "<mask_786>": 50414,
806
+ "<mask_787>": 50413,
807
+ "<mask_788>": 50412,
808
+ "<mask_789>": 50411,
809
+ "<mask_78>": 51122,
810
+ "<mask_790>": 50410,
811
+ "<mask_791>": 50409,
812
+ "<mask_792>": 50408,
813
+ "<mask_793>": 50407,
814
+ "<mask_794>": 50406,
815
+ "<mask_795>": 50405,
816
+ "<mask_796>": 50404,
817
+ "<mask_797>": 50403,
818
+ "<mask_798>": 50402,
819
+ "<mask_799>": 50401,
820
+ "<mask_79>": 51121,
821
+ "<mask_7>": 51193,
822
+ "<mask_800>": 50400,
823
+ "<mask_801>": 50399,
824
+ "<mask_802>": 50398,
825
+ "<mask_803>": 50397,
826
+ "<mask_804>": 50396,
827
+ "<mask_805>": 50395,
828
+ "<mask_806>": 50394,
829
+ "<mask_807>": 50393,
830
+ "<mask_808>": 50392,
831
+ "<mask_809>": 50391,
832
+ "<mask_80>": 51120,
833
+ "<mask_810>": 50390,
834
+ "<mask_811>": 50389,
835
+ "<mask_812>": 50388,
836
+ "<mask_813>": 50387,
837
+ "<mask_814>": 50386,
838
+ "<mask_815>": 50385,
839
+ "<mask_816>": 50384,
840
+ "<mask_817>": 50383,
841
+ "<mask_818>": 50382,
842
+ "<mask_819>": 50381,
843
+ "<mask_81>": 51119,
844
+ "<mask_820>": 50380,
845
+ "<mask_821>": 50379,
846
+ "<mask_822>": 50378,
847
+ "<mask_823>": 50377,
848
+ "<mask_824>": 50376,
849
+ "<mask_825>": 50375,
850
+ "<mask_826>": 50374,
851
+ "<mask_827>": 50373,
852
+ "<mask_828>": 50372,
853
+ "<mask_829>": 50371,
854
+ "<mask_82>": 51118,
855
+ "<mask_830>": 50370,
856
+ "<mask_831>": 50369,
857
+ "<mask_832>": 50368,
858
+ "<mask_833>": 50367,
859
+ "<mask_834>": 50366,
860
+ "<mask_835>": 50365,
861
+ "<mask_836>": 50364,
862
+ "<mask_837>": 50363,
863
+ "<mask_838>": 50362,
864
+ "<mask_839>": 50361,
865
+ "<mask_83>": 51117,
866
+ "<mask_840>": 50360,
867
+ "<mask_841>": 50359,
868
+ "<mask_842>": 50358,
869
+ "<mask_843>": 50357,
870
+ "<mask_844>": 50356,
871
+ "<mask_845>": 50355,
872
+ "<mask_846>": 50354,
873
+ "<mask_847>": 50353,
874
+ "<mask_848>": 50352,
875
+ "<mask_849>": 50351,
876
+ "<mask_84>": 51116,
877
+ "<mask_850>": 50350,
878
+ "<mask_851>": 50349,
879
+ "<mask_852>": 50348,
880
+ "<mask_853>": 50347,
881
+ "<mask_854>": 50346,
882
+ "<mask_855>": 50345,
883
+ "<mask_856>": 50344,
884
+ "<mask_857>": 50343,
885
+ "<mask_858>": 50342,
886
+ "<mask_859>": 50341,
887
+ "<mask_85>": 51115,
888
+ "<mask_860>": 50340,
889
+ "<mask_861>": 50339,
890
+ "<mask_862>": 50338,
891
+ "<mask_863>": 50337,
892
+ "<mask_864>": 50336,
893
+ "<mask_865>": 50335,
894
+ "<mask_866>": 50334,
895
+ "<mask_867>": 50333,
896
+ "<mask_868>": 50332,
897
+ "<mask_869>": 50331,
898
+ "<mask_86>": 51114,
899
+ "<mask_870>": 50330,
900
+ "<mask_871>": 50329,
901
+ "<mask_872>": 50328,
902
+ "<mask_873>": 50327,
903
+ "<mask_874>": 50326,
904
+ "<mask_875>": 50325,
905
+ "<mask_876>": 50324,
906
+ "<mask_877>": 50323,
907
+ "<mask_878>": 50322,
908
+ "<mask_879>": 50321,
909
+ "<mask_87>": 51113,
910
+ "<mask_880>": 50320,
911
+ "<mask_881>": 50319,
912
+ "<mask_882>": 50318,
913
+ "<mask_883>": 50317,
914
+ "<mask_884>": 50316,
915
+ "<mask_885>": 50315,
916
+ "<mask_886>": 50314,
917
+ "<mask_887>": 50313,
918
+ "<mask_888>": 50312,
919
+ "<mask_889>": 50311,
920
+ "<mask_88>": 51112,
921
+ "<mask_890>": 50310,
922
+ "<mask_891>": 50309,
923
+ "<mask_892>": 50308,
924
+ "<mask_893>": 50307,
925
+ "<mask_894>": 50306,
926
+ "<mask_895>": 50305,
927
+ "<mask_896>": 50304,
928
+ "<mask_897>": 50303,
929
+ "<mask_898>": 50302,
930
+ "<mask_899>": 50301,
931
+ "<mask_89>": 51111,
932
+ "<mask_8>": 51192,
933
+ "<mask_90>": 51110,
934
+ "<mask_91>": 51109,
935
+ "<mask_92>": 51108,
936
+ "<mask_93>": 51107,
937
+ "<mask_94>": 51106,
938
+ "<mask_95>": 51105,
939
+ "<mask_96>": 51104,
940
+ "<mask_97>": 51103,
941
+ "<mask_98>": 51102,
942
+ "<mask_99>": 51101,
943
+ "<mask_9>": 51191,
944
+ "<sep>": 50299
945
+ }
config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "checkpoints/codegen2-7B",
3
+ "activation_function": "gelu_new",
4
+ "architectures": [
5
+ "CodeGenForCausalLM"
6
+ ],
7
+ "attn_pdrop": 0.0,
8
+ "auto_map": {
9
+ "AutoConfig": "configuration_codegen.CodeGenConfig",
10
+ "AutoModel": "modeling_codegen.CodeGenModel",
11
+ "AutoModelForCausalLM": "modeling_codegen.CodeGenForCausalLM"
12
+ },
13
+ "bos_token_id": 1,
14
+ "embd_pdrop": 0.0,
15
+ "eos_token_id": 2,
16
+ "gradient_checkpointing": false,
17
+ "initializer_range": 0.02,
18
+ "layer_norm_epsilon": 1e-05,
19
+ "model_type": "codegen",
20
+ "n_ctx": 2048,
21
+ "n_embd": 4096,
22
+ "n_head": 16,
23
+ "n_inner": null,
24
+ "n_layer": 32,
25
+ "n_positions": 2048,
26
+ "resid_pdrop": 0.0,
27
+ "rotary_dim": 64,
28
+ "scale_attn_weights": true,
29
+ "summary_activation": null,
30
+ "summary_first_dropout": 0.1,
31
+ "summary_proj_to_labels": true,
32
+ "summary_type": "cls_index",
33
+ "summary_use_proj": true,
34
+ "task_specific_params": {
35
+ "text-generation": {
36
+ "do_sample": true,
37
+ "max_length": 50,
38
+ "temperature": 1.0
39
+ }
40
+ },
41
+ "tie_word_embeddings": false,
42
+ "tokenizer_class": "GPT2Tokenizer",
43
+ "torch_dtype": "float32",
44
+ "transformers_version": "4.25.1",
45
+ "use_cache": true,
46
+ "vocab_size": 51200
47
+ }
configuration_codegen.py ADDED
@@ -0,0 +1,236 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 Salesforce authors, The EleutherAI, and HuggingFace Teams. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ CodeGen model configuration"""
16
+ from collections import OrderedDict
17
+ from typing import Any, List, Mapping, Optional
18
+
19
+ from transformers import PreTrainedTokenizer, TensorType, is_torch_available
20
+ from transformers.configuration_utils import PretrainedConfig
21
+ from transformers.onnx import OnnxConfigWithPast, PatchingSpec
22
+ from transformers.utils import logging
23
+
24
+
25
+ logger = logging.get_logger(__name__)
26
+
27
+
28
+ CODEGEN_PRETRAINED_CONFIG_ARCHIVE_MAP = {
29
+ "Salesforce/codegen-350M-nl": "https://huggingface.co/Salesforce/codegen-350M-nl/resolve/main/config.json",
30
+ "Salesforce/codegen-350M-multi": "https://huggingface.co/Salesforce/codegen-350M-multi/resolve/main/config.json",
31
+ "Salesforce/codegen-350M-mono": "https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/config.json",
32
+ "Salesforce/codegen-2B-nl": "https://huggingface.co/Salesforce/codegen-2B-nl/resolve/main/config.json",
33
+ "Salesforce/codegen-2B-multi": "https://huggingface.co/Salesforce/codegen-2B-multi/resolve/main/config.json",
34
+ "Salesforce/codegen-2B-mono": "https://huggingface.co/Salesforce/codegen-2B-mono/resolve/main/config.json",
35
+ "Salesforce/codegen-6B-nl": "https://huggingface.co/Salesforce/codegen-6B-nl/resolve/main/config.json",
36
+ "Salesforce/codegen-6B-multi": "https://huggingface.co/Salesforce/codegen-6B-multi/resolve/main/config.json",
37
+ "Salesforce/codegen-6B-mono": "https://huggingface.co/Salesforce/codegen-6B-mono/resolve/main/config.json",
38
+ "Salesforce/codegen-16B-nl": "https://huggingface.co/Salesforce/codegen-16B-nl/resolve/main/config.json",
39
+ "Salesforce/codegen-16B-multi": "https://huggingface.co/Salesforce/codegen-16B-multi/resolve/main/config.json",
40
+ "Salesforce/codegen-16B-mono": "https://huggingface.co/Salesforce/codegen-16B-mono/resolve/main/config.json",
41
+ }
42
+
43
+
44
+ class CodeGenConfig(PretrainedConfig):
45
+ r"""
46
+ This is the configuration class to store the configuration of a [`CodeGenModel`]. It is used to instantiate a
47
+ CodeGen model according to the specified arguments, defining the model architecture. Instantiating a configuration
48
+ with the defaults will yield a similar configuration to that of the CodeGen
49
+ [Salesforce/codegen-2B-mono](https://huggingface.co/Salesforce/codegen-2B-mono) architecture. Configuration objects
50
+ inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from
51
+ [`PretrainedConfig`] for more information.
52
+
53
+ Args:
54
+ vocab_size (`int`, *optional*, defaults to 50400):
55
+ Vocabulary size of the CodeGen model. Defines the number of different tokens that can be represented by the
56
+ `inputs_ids` passed when calling [`CodeGenModel`].
57
+ n_positions (`int`, *optional*, defaults to 2048):
58
+ The maximum sequence length that this model might ever be used with. Typically set this to something large
59
+ just in case (e.g., 512 or 1024 or 2048).
60
+ n_embd (`int`, *optional*, defaults to 4096):
61
+ Dimensionality of the embeddings and hidden states.
62
+ n_layer (`int`, *optional*, defaults to 28):
63
+ Number of hidden layers in the Transformer encoder.
64
+ n_head (`int`, *optional*, defaults to 16):
65
+ Number of attention heads for each attention layer in the Transformer encoder.
66
+ rotary_dim (`int`, *optional*, defaults to 64):
67
+ Number of dimensions in the embedding that Rotary Position Embedding is applied to.
68
+ n_inner (`int`, *optional*, defaults to None):
69
+ Dimensionality of the inner feed-forward layers. `None` will set it to 4 times n_embd
70
+ activation_function (`str`, *optional*, defaults to `"gelu_new"`):
71
+ Activation function, to be selected in the list `["relu", "silu", "gelu", "tanh", "gelu_new"]`.
72
+ resid_pdrop (`float`, *optional*, defaults to 0.1):
73
+ The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
74
+ embd_pdrop (`int`, *optional*, defaults to 0.1):
75
+ The dropout ratio for the embeddings.
76
+ attn_pdrop (`float`, *optional*, defaults to 0.1):
77
+ The dropout ratio for the attention.
78
+ layer_norm_epsilon (`float`, *optional*, defaults to 1e-5):
79
+ The epsilon to use in the layer normalization layers.
80
+ initializer_range (`float`, *optional*, defaults to 0.02):
81
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
82
+ scale_attn_weights (`bool`, *optional*, defaults to `True`):
83
+ Scale attention weights by dividing by sqrt(hidden_size).
84
+ use_cache (`bool`, *optional*, defaults to `True`):
85
+ Whether or not the model should return the last key/values attentions (not used by all models).
86
+
87
+ Example:
88
+
89
+ ```python
90
+ >>> from transformers import CodeGenModel, CodeGenConfig
91
+
92
+ >>> # Initializing a CodeGen 6B configuration
93
+ >>> configuration = CodeGenConfig()
94
+
95
+ >>> # Initializing a model from the configuration
96
+ >>> model = CodeGenModel(configuration)
97
+
98
+ >>> # Accessing the model configuration
99
+ >>> configuration = model.config
100
+ ```"""
101
+ model_type = "codegen"
102
+ attribute_map = {
103
+ "max_position_embeddings": "n_positions",
104
+ "hidden_size": "n_embd",
105
+ "num_attention_heads": "n_head",
106
+ "num_hidden_layers": "n_layer",
107
+ }
108
+
109
+ def __init__(
110
+ self,
111
+ vocab_size=50400,
112
+ n_positions=2048,
113
+ n_ctx=2048,
114
+ n_embd=4096,
115
+ n_layer=28,
116
+ n_head=16,
117
+ rotary_dim=64,
118
+ n_inner=None,
119
+ activation_function="gelu_new",
120
+ resid_pdrop=0.0,
121
+ embd_pdrop=0.0,
122
+ attn_pdrop=0.0,
123
+ layer_norm_epsilon=1e-5,
124
+ initializer_range=0.02,
125
+ scale_attn_weights=True,
126
+ use_cache=True,
127
+ bos_token_id=50256,
128
+ eos_token_id=50256,
129
+ tie_word_embeddings=False,
130
+ **kwargs
131
+ ):
132
+ self.vocab_size = vocab_size
133
+ self.n_ctx = n_ctx
134
+ self.n_positions = n_positions
135
+ self.n_embd = n_embd
136
+ self.n_layer = n_layer
137
+ self.n_head = n_head
138
+ self.n_inner = n_inner
139
+ self.rotary_dim = rotary_dim
140
+ self.activation_function = activation_function
141
+ self.resid_pdrop = resid_pdrop
142
+ self.embd_pdrop = embd_pdrop
143
+ self.attn_pdrop = attn_pdrop
144
+ self.layer_norm_epsilon = layer_norm_epsilon
145
+ self.initializer_range = initializer_range
146
+ self.scale_attn_weights = scale_attn_weights
147
+ self.use_cache = use_cache
148
+
149
+ self.bos_token_id = bos_token_id
150
+ self.eos_token_id = eos_token_id
151
+
152
+ super().__init__(
153
+ bos_token_id=bos_token_id, eos_token_id=eos_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs
154
+ )
155
+
156
+
157
+ # Copied from transformers.models.gpt2.configuration_gpt2.GPT2OnnxConfig
158
+ class CodeGenOnnxConfig(OnnxConfigWithPast):
159
+ def __init__(
160
+ self,
161
+ config: PretrainedConfig,
162
+ task: str = "default",
163
+ patching_specs: List[PatchingSpec] = None,
164
+ use_past: bool = False,
165
+ ):
166
+ super().__init__(config, task=task, patching_specs=patching_specs, use_past=use_past)
167
+ if not getattr(self._config, "pad_token_id", None):
168
+ # TODO: how to do that better?
169
+ self._config.pad_token_id = 0
170
+
171
+ @property
172
+ def inputs(self) -> Mapping[str, Mapping[int, str]]:
173
+ common_inputs = OrderedDict({"input_ids": {0: "batch", 1: "sequence"}})
174
+ if self.use_past:
175
+ self.fill_with_past_key_values_(common_inputs, direction="inputs")
176
+ common_inputs["attention_mask"] = {0: "batch", 1: "past_sequence + sequence"}
177
+ else:
178
+ common_inputs["attention_mask"] = {0: "batch", 1: "sequence"}
179
+
180
+ return common_inputs
181
+
182
+ @property
183
+ def num_layers(self) -> int:
184
+ return self._config.n_layer
185
+
186
+ @property
187
+ def num_attention_heads(self) -> int:
188
+ return self._config.n_head
189
+
190
+ def generate_dummy_inputs(
191
+ self,
192
+ tokenizer: PreTrainedTokenizer,
193
+ batch_size: int = -1,
194
+ seq_length: int = -1,
195
+ is_pair: bool = False,
196
+ framework: Optional[TensorType] = None,
197
+ ) -> Mapping[str, Any]:
198
+ common_inputs = super(OnnxConfigWithPast, self).generate_dummy_inputs(
199
+ tokenizer, batch_size=batch_size, seq_length=seq_length, is_pair=is_pair, framework=framework
200
+ )
201
+
202
+ # We need to order the input in the way they appears in the forward()
203
+ ordered_inputs = OrderedDict({"input_ids": common_inputs["input_ids"]})
204
+
205
+ # Need to add the past_keys
206
+ if self.use_past:
207
+ if not is_torch_available():
208
+ raise ValueError("Cannot generate dummy past_keys inputs without PyTorch installed.")
209
+ else:
210
+ import torch
211
+
212
+ batch, seqlen = common_inputs["input_ids"].shape
213
+ # Not using the same length for past_key_values
214
+ past_key_values_length = seqlen + 2
215
+ past_shape = (
216
+ batch,
217
+ self.num_attention_heads,
218
+ past_key_values_length,
219
+ self._config.hidden_size // self.num_attention_heads,
220
+ )
221
+ ordered_inputs["past_key_values"] = [
222
+ (torch.zeros(past_shape), torch.zeros(past_shape)) for _ in range(self.num_layers)
223
+ ]
224
+
225
+ ordered_inputs["attention_mask"] = common_inputs["attention_mask"]
226
+ if self.use_past:
227
+ mask_dtype = ordered_inputs["attention_mask"].dtype
228
+ ordered_inputs["attention_mask"] = torch.cat(
229
+ [ordered_inputs["attention_mask"], torch.ones(batch, past_key_values_length, dtype=mask_dtype)], dim=1
230
+ )
231
+
232
+ return ordered_inputs
233
+
234
+ @property
235
+ def default_onnx_opset(self) -> int:
236
+ return 13
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
modeling_codegen.py ADDED
@@ -0,0 +1,747 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 Salesforce authors, The EleutherAI, and HuggingFace Teams. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ PyTorch CodeGen model."""
16
+
17
+ from typing import Optional, Tuple, Union
18
+
19
+ import torch
20
+ import torch.utils.checkpoint
21
+ from torch import nn
22
+ from torch.nn import CrossEntropyLoss
23
+
24
+ from transformers.activations import ACT2FN
25
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
26
+ from transformers.modeling_utils import PreTrainedModel
27
+ from transformers.utils import add_code_sample_docstrings, add_start_docstrings, add_start_docstrings_to_model_forward, logging
28
+ from .configuration_codegen import CodeGenConfig
29
+
30
+
31
+ logger = logging.get_logger(__name__)
32
+
33
+ _CHECKPOINT_FOR_DOC = "Salesforce/codegen-2B-mono"
34
+ _CONFIG_FOR_DOC = "CodeGenConfig"
35
+ _TOKENIZER_FOR_DOC = "GPT2Tokenizer"
36
+
37
+
38
+ CODEGEN_PRETRAINED_MODEL_ARCHIVE_LIST = [
39
+ "Salesforce/codegen-350M-nl",
40
+ "Salesforce/codegen-350M-multi",
41
+ "Salesforce/codegen-350M-mono",
42
+ "Salesforce/codegen-2B-nl",
43
+ "Salesforce/codegen-2B-multi",
44
+ "Salesforce/codegen-2B-mono",
45
+ "Salesforce/codegen-6B-nl",
46
+ "Salesforce/codegen-6B-multi",
47
+ "Salesforce/codegen-6B-mono",
48
+ "Salesforce/codegen-16B-nl",
49
+ "Salesforce/codegen-16B-multi",
50
+ "Salesforce/codegen-16B-mono",
51
+ # See all CodeGen models at https://huggingface.co/models?filter=codegen
52
+ ]
53
+
54
+
55
+ # Copied from transformers.models.gptj.modeling_gptj.fixed_pos_embedding
56
+ def fixed_pos_embedding(x, seq_dim=1, seq_len=None):
57
+ dim = x.shape[-1]
58
+ if seq_len is None:
59
+ seq_len = x.shape[seq_dim]
60
+ inv_freq = 1.0 / (10000 ** (torch.arange(0, dim, 2) / dim))
61
+ sinusoid_inp = (
62
+ torch.einsum("i , j -> i j", torch.arange(seq_len, dtype=torch.float), inv_freq).to(x.device).float()
63
+ )
64
+ return torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)
65
+
66
+
67
+ # Copied from transformers.models.gptj.modeling_gptj.rotate_every_two
68
+ def rotate_every_two(x):
69
+ x1 = x[:, :, :, ::2]
70
+ x2 = x[:, :, :, 1::2]
71
+ x = torch.stack((-x2, x1), dim=-1)
72
+ return x.flatten(-2) # in einsum notation: rearrange(x, '... d j -> ... (d j)')
73
+
74
+
75
+ # Copied from transformers.models.gptj.modeling_gptj.duplicate_interleave
76
+ def duplicate_interleave(m):
77
+ """
78
+ A simple version of `torch.repeat_interleave` for duplicating a matrix while interleaving the copy.
79
+ """
80
+ dim0 = m.shape[0]
81
+ m = m.view(-1, 1) # flatten the matrix
82
+ m = m.repeat(1, 2) # repeat all elements into the 2nd dimension
83
+ m = m.view(dim0, -1) # reshape into a matrix, interleaving the copy
84
+ return m
85
+
86
+
87
+ # Copied from transformers.models.gptj.modeling_gptj.apply_rotary_pos_emb
88
+ def apply_rotary_pos_emb(x, sincos, offset=0):
89
+ sin, cos = map(lambda t: duplicate_interleave(t)[None, offset : x.shape[1] + offset, None, :], sincos)
90
+ # einsum notation for lambda t: repeat(t[offset:x.shape[1]+offset,:], "n d -> () n () (d j)", j=2)
91
+ return (x * cos) + (rotate_every_two(x) * sin)
92
+
93
+
94
+ class CodeGenAttention(nn.Module):
95
+ def __init__(self, config):
96
+ super().__init__()
97
+
98
+ max_positions = config.max_position_embeddings
99
+ self.register_buffer(
100
+ "causal_mask",
101
+ torch.tril(torch.ones((max_positions, max_positions), dtype=torch.bool)).view(
102
+ 1, 1, max_positions, max_positions
103
+ ),
104
+ )
105
+
106
+ self.attn_dropout = nn.Dropout(config.attn_pdrop)
107
+ self.resid_dropout = nn.Dropout(config.resid_pdrop)
108
+
109
+ self.embed_dim = config.hidden_size
110
+ self.num_attention_heads = config.num_attention_heads
111
+ self.head_dim = self.embed_dim // self.num_attention_heads
112
+ if self.head_dim * self.num_attention_heads != self.embed_dim:
113
+ raise ValueError(
114
+ f"embed_dim must be divisible by num_attention_heads (got `embed_dim`: {self.embed_dim} and"
115
+ f" `num_attention_heads`: {self.num_attention_heads})."
116
+ )
117
+ self.scale_attn = torch.sqrt(torch.tensor(self.head_dim, dtype=torch.float32)).to(torch.get_default_dtype())
118
+ self.qkv_proj = nn.Linear(self.embed_dim, self.embed_dim * 3, bias=False)
119
+
120
+ self.out_proj = nn.Linear(self.embed_dim, self.embed_dim, bias=False)
121
+ self.rotary_dim = None
122
+ if config.rotary_dim is not None:
123
+ self.rotary_dim = config.rotary_dim
124
+
125
+ def _split_heads(self, x, n_head, dim_head, mp_num):
126
+ reshaped = x.reshape(x.shape[:-1] + (n_head // mp_num, dim_head))
127
+ reshaped = reshaped.reshape(x.shape[:-2] + (-1,) + reshaped.shape[-1:])
128
+ return reshaped
129
+
130
+ def _merge_heads(self, tensor, num_attention_heads, attn_head_size):
131
+ """
132
+ Merges attn_head_size dim and num_attn_heads dim into n_ctx
133
+ """
134
+ if len(tensor.shape) == 5:
135
+ tensor = tensor.permute(0, 1, 3, 2, 4).contiguous()
136
+ elif len(tensor.shape) == 4:
137
+ tensor = tensor.permute(0, 2, 1, 3).contiguous()
138
+ else:
139
+ raise ValueError(f"Input tensor rank should be one of [4, 5], but is: {len(tensor.shape)}")
140
+ new_shape = tensor.size()[:-2] + (num_attention_heads * attn_head_size,)
141
+ return tensor.view(new_shape)
142
+
143
+ def _attn(
144
+ self,
145
+ query,
146
+ key,
147
+ value,
148
+ attention_mask=None,
149
+ head_mask=None,
150
+ ):
151
+
152
+ # compute causal mask from causal mask buffer
153
+ query_length, key_length = query.size(-2), key.size(-2)
154
+ causal_mask = self.causal_mask[:, :, key_length - query_length : key_length, :key_length]
155
+
156
+ # Keep the attention weights computation in fp32 to avoid overflow issues
157
+ query = query.to(torch.float32)
158
+ key = key.to(torch.float32)
159
+
160
+ attn_weights = torch.matmul(query, key.transpose(-1, -2))
161
+
162
+ attn_weights = attn_weights / self.scale_attn
163
+ mask_value = torch.finfo(attn_weights.dtype).min
164
+ # Need to be a tensor, otherwise we get error: `RuntimeError: expected scalar type float but found double`.
165
+ # Need to be on the same device, otherwise `RuntimeError: ..., x and y to be on the same device`
166
+ mask_value = torch.tensor(mask_value, dtype=attn_weights.dtype).to(attn_weights.device)
167
+ attn_weights = torch.where(causal_mask, attn_weights, mask_value)
168
+
169
+ if attention_mask is not None:
170
+ # Apply the attention mask
171
+ attn_weights = attn_weights + attention_mask
172
+
173
+ attn_weights = nn.Softmax(dim=-1)(attn_weights)
174
+ attn_weights = attn_weights.to(value.dtype)
175
+ attn_weights = self.attn_dropout(attn_weights)
176
+
177
+ # Mask heads if we want to
178
+ if head_mask is not None:
179
+ attn_weights = attn_weights * head_mask
180
+
181
+ attn_output = torch.matmul(attn_weights, value)
182
+
183
+ return attn_output, attn_weights
184
+
185
+ def forward(
186
+ self,
187
+ hidden_states: Optional[torch.FloatTensor],
188
+ attention_mask: Optional[torch.FloatTensor] = None,
189
+ layer_past: Optional[Tuple[torch.Tensor]] = None,
190
+ head_mask: Optional[torch.FloatTensor] = None,
191
+ use_cache: Optional[bool] = False,
192
+ output_attentions: Optional[bool] = False,
193
+ ) -> Union[
194
+ Tuple[torch.Tensor, Tuple[torch.Tensor]],
195
+ Optional[Tuple[torch.Tensor, Tuple[torch.Tensor], Tuple[torch.Tensor, ...]]],
196
+ ]:
197
+
198
+ qkv = self.qkv_proj(hidden_states)
199
+
200
+ # TPU-v4
201
+ mp_num = 4
202
+ qkv_split = qkv.reshape(qkv.shape[:-1] + (mp_num, -1))
203
+
204
+ local_dim = self.head_dim * self.num_attention_heads // mp_num
205
+ query, value, key = torch.split(qkv_split, local_dim, dim=-1)
206
+ query = self._split_heads(query, self.num_attention_heads, self.head_dim, mp_num=mp_num)
207
+ key = self._split_heads(key, self.num_attention_heads, self.head_dim, mp_num=mp_num)
208
+
209
+ value = self._split_heads(value, self.num_attention_heads, self.head_dim, mp_num=mp_num)
210
+ value = value.permute(0, 2, 1, 3)
211
+
212
+ seq_len = key.shape[1]
213
+ offset = 0
214
+
215
+ if layer_past is not None:
216
+ offset = layer_past[0].shape[-2]
217
+ seq_len += offset
218
+
219
+ if self.rotary_dim is not None:
220
+ k_rot = key[:, :, :, : self.rotary_dim]
221
+ k_pass = key[:, :, :, self.rotary_dim :]
222
+
223
+ q_rot = query[:, :, :, : self.rotary_dim]
224
+ q_pass = query[:, :, :, self.rotary_dim :]
225
+
226
+ sincos = fixed_pos_embedding(k_rot, 1, seq_len=seq_len)
227
+ k_rot = apply_rotary_pos_emb(k_rot, sincos, offset=offset)
228
+ q_rot = apply_rotary_pos_emb(q_rot, sincos, offset=offset)
229
+
230
+ key = torch.cat([k_rot, k_pass], dim=-1)
231
+ query = torch.cat([q_rot, q_pass], dim=-1)
232
+ else:
233
+ sincos = fixed_pos_embedding(key, 1, seq_len=seq_len)
234
+ key = apply_rotary_pos_emb(key, sincos, offset=offset)
235
+ query = apply_rotary_pos_emb(query, sincos, offset=offset)
236
+
237
+ key = key.permute(0, 2, 1, 3)
238
+ query = query.permute(0, 2, 1, 3)
239
+
240
+ if layer_past is not None:
241
+ past_key = layer_past[0]
242
+ past_value = layer_past[1]
243
+ key = torch.cat((past_key, key), dim=-2)
244
+ value = torch.cat((past_value, value), dim=-2)
245
+
246
+ if use_cache is True:
247
+ present = (key, value)
248
+ else:
249
+ present = None
250
+
251
+ # compute self-attention: V x Softmax(QK^T)
252
+ attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
253
+
254
+ attn_output = self._merge_heads(attn_output, self.num_attention_heads, self.head_dim)
255
+ attn_output = self.out_proj(attn_output)
256
+ attn_output = self.resid_dropout(attn_output)
257
+
258
+ outputs = (attn_output, present)
259
+ if output_attentions:
260
+ outputs += (attn_weights,)
261
+
262
+ return outputs # a, present, (attentions)
263
+
264
+
265
+ # Copied from transformers.models.gptj.modeling_gptj.GPTJMLP with GPTJ->CodeGen
266
+ class CodeGenMLP(nn.Module):
267
+ def __init__(self, intermediate_size, config): # in MLP: intermediate_size= 4 * embed_dim
268
+ super().__init__()
269
+ embed_dim = config.n_embd
270
+
271
+ self.fc_in = nn.Linear(embed_dim, intermediate_size)
272
+ self.fc_out = nn.Linear(intermediate_size, embed_dim)
273
+
274
+ self.act = ACT2FN[config.activation_function]
275
+ self.dropout = nn.Dropout(config.resid_pdrop)
276
+
277
+ def forward(self, hidden_states: Optional[torch.FloatTensor]) -> torch.FloatTensor:
278
+ hidden_states = self.fc_in(hidden_states)
279
+ hidden_states = self.act(hidden_states)
280
+ hidden_states = self.fc_out(hidden_states)
281
+ hidden_states = self.dropout(hidden_states)
282
+ return hidden_states
283
+
284
+
285
+ # Copied from transformers.models.gptj.modeling_gptj.GPTJBlock with GPTJ->CodeGen
286
+ class CodeGenBlock(nn.Module):
287
+ def __init__(self, config):
288
+ super().__init__()
289
+ inner_dim = config.n_inner if config.n_inner is not None else 4 * config.n_embd
290
+ self.ln_1 = nn.LayerNorm(config.n_embd, eps=config.layer_norm_epsilon)
291
+ self.attn = CodeGenAttention(config)
292
+ self.mlp = CodeGenMLP(inner_dim, config)
293
+
294
+ def forward(
295
+ self,
296
+ hidden_states: Optional[torch.FloatTensor],
297
+ layer_past: Optional[Tuple[torch.Tensor]] = None,
298
+ attention_mask: Optional[torch.FloatTensor] = None,
299
+ head_mask: Optional[torch.FloatTensor] = None,
300
+ use_cache: Optional[bool] = False,
301
+ output_attentions: Optional[bool] = False,
302
+ ) -> Union[Tuple[torch.Tensor], Optional[Tuple[torch.Tensor, Tuple[torch.FloatTensor, ...]]]]:
303
+ residual = hidden_states
304
+ hidden_states = self.ln_1(hidden_states)
305
+ attn_outputs = self.attn(
306
+ hidden_states,
307
+ layer_past=layer_past,
308
+ attention_mask=attention_mask,
309
+ head_mask=head_mask,
310
+ use_cache=use_cache,
311
+ output_attentions=output_attentions,
312
+ )
313
+ attn_output = attn_outputs[0] # output_attn: a, present, (attentions)
314
+ outputs = attn_outputs[1:]
315
+
316
+ feed_forward_hidden_states = self.mlp(hidden_states)
317
+ hidden_states = attn_output + feed_forward_hidden_states + residual
318
+
319
+ if use_cache:
320
+ outputs = (hidden_states,) + outputs
321
+ else:
322
+ outputs = (hidden_states,) + outputs[1:]
323
+
324
+ return outputs # hidden_states, present, (attentions)
325
+
326
+
327
+ class CodeGenPreTrainedModel(PreTrainedModel):
328
+ """
329
+ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
330
+ models.
331
+ """
332
+
333
+ config_class = CodeGenConfig
334
+ base_model_prefix = "transformer"
335
+ supports_gradient_checkpointing = True
336
+ _no_split_modules = ["CodeGenBlock"]
337
+
338
+ def __init__(self, *inputs, **kwargs):
339
+ super().__init__(*inputs, **kwargs)
340
+
341
+ def _init_weights(self, module):
342
+ """Initialize the weights."""
343
+ if isinstance(module, (nn.Linear,)):
344
+ # Slightly different from Mesh Transformer JAX which uses truncated_normal for initialization
345
+ # cf https://github.com/pytorch/pytorch/pull/5617
346
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
347
+ if module.bias is not None:
348
+ module.bias.data.zero_()
349
+ elif isinstance(module, nn.Embedding):
350
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
351
+ if module.padding_idx is not None:
352
+ module.weight.data[module.padding_idx].zero_()
353
+ elif isinstance(module, nn.LayerNorm):
354
+ module.bias.data.zero_()
355
+ module.weight.data.fill_(1.0)
356
+
357
+ def _set_gradient_checkpointing(self, module, value=False):
358
+ if isinstance(module, CodeGenModel):
359
+ module.gradient_checkpointing = value
360
+
361
+
362
+ CODEGEN_START_DOCSTRING = r"""
363
+ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use
364
+ it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
365
+ behavior.
366
+
367
+ Parameters:
368
+ config ([`CodeGenConfig`]): Model configuration class with all the parameters of the model.
369
+ Initializing with a config file does not load the weights associated with the model, only the
370
+ configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
371
+ """
372
+
373
+ CODEGEN_INPUTS_DOCSTRING = r"""
374
+ Args:
375
+ input_ids (`torch.LongTensor` of shape `({0})`):
376
+ Indices of input sequence tokens in the vocabulary.
377
+
378
+ Indices can be obtained using [`GPT2Tokenizer`]. See [`PreTrainedTokenizer.encode`] and
379
+ [`PreTrainedTokenizer.__call__`] for details.
380
+
381
+ [What are input IDs?](../glossary#input-ids)
382
+ attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*):
383
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
384
+
385
+ - 1 for tokens that are **not masked**,
386
+ - 0 for tokens that are **masked**.
387
+
388
+ [What are attention masks?](../glossary#attention-mask)
389
+ token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*):
390
+ Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,
391
+ 1]`:
392
+
393
+ - 0 corresponds to a *sentence A* token,
394
+ - 1 corresponds to a *sentence B* token.
395
+
396
+ [What are token type IDs?](../glossary#token-type-ids)
397
+ position_ids (`torch.LongTensor` of shape `({0})`, *optional*):
398
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
399
+ config.n_positions - 1]`.
400
+
401
+ [What are position IDs?](../glossary#position-ids)
402
+ head_mask (`torch.FloatTensor` of shape `(num_attention_heads,)` or `(n_layer, num_attention_heads)`, *optional*):
403
+ Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
404
+
405
+ - 1 indicates the head is **not masked**,
406
+ - 0 indicates the head is **masked**.
407
+
408
+ inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_dim)`, *optional*):
409
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
410
+ is useful if you want more control over how to convert *input_ids* indices into associated vectors than the
411
+ model's internal embedding lookup matrix.
412
+ output_attentions (`bool`, *optional*):
413
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
414
+ tensors for more detail.
415
+ output_hidden_states (`bool`, *optional*):
416
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
417
+ more detail.
418
+ return_dict (`bool`, *optional*):
419
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
420
+ """
421
+
422
+
423
+ @add_start_docstrings(
424
+ "The bare CodeGen Model transformer outputting raw hidden-states without any specific head on top.",
425
+ CODEGEN_START_DOCSTRING,
426
+ )
427
+ class CodeGenModel(CodeGenPreTrainedModel):
428
+ def __init__(self, config):
429
+ super().__init__(config)
430
+
431
+ self.embed_dim = config.n_embd
432
+ self.vocab_size = config.vocab_size
433
+ self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
434
+ self.drop = nn.Dropout(config.embd_pdrop)
435
+ self.h = nn.ModuleList([CodeGenBlock(config) for _ in range(config.n_layer)])
436
+ self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
437
+ self.rotary_dim = min(config.rotary_dim, config.n_ctx // config.num_attention_heads)
438
+
439
+ self.gradient_checkpointing = False
440
+
441
+ # Initialize weights and apply final processing
442
+ self.post_init()
443
+
444
+ def get_input_embeddings(self):
445
+ return self.wte
446
+
447
+ def set_input_embeddings(self, new_embeddings):
448
+ self.wte = new_embeddings
449
+
450
+ @add_start_docstrings_to_model_forward(CODEGEN_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
451
+ @add_code_sample_docstrings(
452
+ processor_class=_TOKENIZER_FOR_DOC,
453
+ checkpoint=_CHECKPOINT_FOR_DOC,
454
+ output_type=BaseModelOutputWithPast,
455
+ config_class=_CONFIG_FOR_DOC,
456
+ )
457
+ def forward(
458
+ self,
459
+ input_ids: Optional[torch.LongTensor] = None,
460
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
461
+ attention_mask: Optional[torch.FloatTensor] = None,
462
+ token_type_ids: Optional[torch.LongTensor] = None,
463
+ position_ids: Optional[torch.LongTensor] = None,
464
+ head_mask: Optional[torch.FloatTensor] = None,
465
+ inputs_embeds: Optional[torch.FloatTensor] = None,
466
+ use_cache: Optional[bool] = None,
467
+ output_attentions: Optional[bool] = None,
468
+ output_hidden_states: Optional[bool] = None,
469
+ return_dict: Optional[bool] = None,
470
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
471
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
472
+ output_hidden_states = (
473
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
474
+ )
475
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
476
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
477
+
478
+ if input_ids is not None and inputs_embeds is not None:
479
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
480
+ elif input_ids is not None:
481
+ input_shape = input_ids.size()
482
+ input_ids = input_ids.view(-1, input_shape[-1])
483
+ batch_size = input_ids.shape[0]
484
+ elif inputs_embeds is not None:
485
+ input_shape = inputs_embeds.size()[:-1]
486
+ batch_size = inputs_embeds.shape[0]
487
+ else:
488
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
489
+
490
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
491
+
492
+ if token_type_ids is not None:
493
+ token_type_ids = token_type_ids.view(-1, input_shape[-1])
494
+
495
+ if position_ids is not None:
496
+ position_ids = position_ids.view(-1, input_shape[-1])
497
+
498
+ if past_key_values is None:
499
+ past_length = 0
500
+ past_key_values = tuple([None] * len(self.h))
501
+ else:
502
+ past_length = past_key_values[0][0].size(-2)
503
+
504
+ if position_ids is None:
505
+ position_ids = torch.arange(past_length, input_shape[-1] + past_length, dtype=torch.long, device=device)
506
+ position_ids = position_ids.unsqueeze(0).view(-1, input_shape[-1])
507
+
508
+ # Attention mask.
509
+ if attention_mask is not None:
510
+ if batch_size <= 0:
511
+ raise ValueError("batch_size has to be defined and > 0")
512
+ attention_mask = attention_mask.view(batch_size, -1)
513
+ # We create a 3D attention mask from a 2D tensor mask.
514
+ # Sizes are [batch_size, 1, 1, to_seq_length]
515
+ # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
516
+ # this attention mask is more simple than the triangular masking of causal attention
517
+ # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
518
+ attention_mask = attention_mask[:, None, None, :]
519
+
520
+ # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
521
+ # masked positions, this operation will create a tensor which is 0.0 for
522
+ # positions we want to attend and the dtype's smallest value for masked positions.
523
+ # Since we are adding it to the raw scores before the softmax, this is
524
+ # effectively the same as removing these entirely.
525
+ attention_mask = attention_mask.to(dtype=self.dtype) # fp16 compatibility
526
+ attention_mask = (1.0 - attention_mask) * torch.finfo(self.dtype).min
527
+
528
+ # Prepare head mask if needed
529
+ # 1.0 in head_mask indicate we keep the head
530
+ # attention_probs has shape bsz x num_attention_heads x N x N
531
+ # head_mask has shape n_layer x batch x num_attention_heads x N x N
532
+ head_mask = self.get_head_mask(head_mask, self.config.n_layer)
533
+
534
+ if inputs_embeds is None:
535
+ inputs_embeds = self.wte(input_ids)
536
+
537
+ hidden_states = inputs_embeds
538
+
539
+ if token_type_ids is not None:
540
+ token_type_embeds = self.wte(token_type_ids)
541
+ hidden_states = hidden_states + token_type_embeds
542
+
543
+ hidden_states = self.drop(hidden_states)
544
+
545
+ output_shape = input_shape + (hidden_states.size(-1),)
546
+
547
+ presents = () if use_cache else None
548
+ all_self_attentions = () if output_attentions else None
549
+ all_hidden_states = () if output_hidden_states else None
550
+ for i, (block, layer_past) in enumerate(zip(self.h, past_key_values)):
551
+
552
+ if output_hidden_states:
553
+ all_hidden_states = all_hidden_states + (hidden_states,)
554
+
555
+ if self.gradient_checkpointing and self.training:
556
+
557
+ if use_cache:
558
+ logger.warning(
559
+ "`use_cache=True` is incompatible with `config.gradient_checkpointing=True`. Setting "
560
+ "`use_cache=False`..."
561
+ )
562
+ use_cache = False
563
+
564
+ def create_custom_forward(module):
565
+ def custom_forward(*inputs):
566
+ # None for past_key_value
567
+ return module(*inputs, use_cache, output_attentions)
568
+
569
+ return custom_forward
570
+
571
+ outputs = torch.utils.checkpoint.checkpoint(
572
+ create_custom_forward(block),
573
+ hidden_states,
574
+ None,
575
+ attention_mask,
576
+ head_mask[i],
577
+ )
578
+ else:
579
+ outputs = block(
580
+ hidden_states,
581
+ layer_past=layer_past,
582
+ attention_mask=attention_mask,
583
+ head_mask=head_mask[i],
584
+ use_cache=use_cache,
585
+ output_attentions=output_attentions,
586
+ )
587
+
588
+ hidden_states = outputs[0]
589
+ if use_cache is True:
590
+ presents = presents + (outputs[1],)
591
+
592
+ if output_attentions:
593
+ all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)
594
+
595
+ hidden_states = self.ln_f(hidden_states)
596
+
597
+ hidden_states = hidden_states.view(output_shape)
598
+ # Add last hidden state
599
+ if output_hidden_states:
600
+ all_hidden_states = all_hidden_states + (hidden_states,)
601
+
602
+ if not return_dict:
603
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
604
+
605
+ return BaseModelOutputWithPast(
606
+ last_hidden_state=hidden_states,
607
+ past_key_values=presents,
608
+ hidden_states=all_hidden_states,
609
+ attentions=all_self_attentions,
610
+ )
611
+
612
+
613
+ @add_start_docstrings(
614
+ """
615
+ The CodeGen Model transformer with a language modeling head on top.
616
+ """,
617
+ CODEGEN_START_DOCSTRING,
618
+ )
619
+ class CodeGenForCausalLM(CodeGenPreTrainedModel):
620
+ _keys_to_ignore_on_load_missing = [r"h\.\d+\.attn\.masked_bias", r"h\.\d+\.attn\.bias"]
621
+
622
+ def __init__(self, config):
623
+ super().__init__(config)
624
+ self.transformer = CodeGenModel(config)
625
+ self.lm_head = nn.Linear(config.n_embd, config.vocab_size)
626
+
627
+ # Initialize weights and apply final processing
628
+ self.post_init()
629
+
630
+ def get_output_embeddings(self):
631
+ return self.lm_head
632
+
633
+ def set_output_embeddings(self, new_embeddings):
634
+ self.lm_head = new_embeddings
635
+
636
+ def prepare_inputs_for_generation(self, input_ids, past=None, **kwargs):
637
+ token_type_ids = kwargs.get("token_type_ids", None)
638
+ # only last token for inputs_ids if past is defined in kwargs
639
+ if past:
640
+ input_ids = input_ids[:, -1].unsqueeze(-1)
641
+ if token_type_ids is not None:
642
+ token_type_ids = token_type_ids[:, -1].unsqueeze(-1)
643
+
644
+ attention_mask = kwargs.get("attention_mask", None)
645
+ position_ids = kwargs.get("position_ids", None)
646
+
647
+ if attention_mask is not None and position_ids is None:
648
+ # create position_ids on the fly for batch generation
649
+ position_ids = attention_mask.long().cumsum(-1) - 1
650
+ position_ids.masked_fill_(attention_mask == 0, 1)
651
+ if past:
652
+ position_ids = position_ids[:, -1].unsqueeze(-1)
653
+ else:
654
+ position_ids = None
655
+ return {
656
+ "input_ids": input_ids,
657
+ "past_key_values": past,
658
+ "use_cache": kwargs.get("use_cache"),
659
+ "position_ids": position_ids,
660
+ "attention_mask": attention_mask,
661
+ "token_type_ids": token_type_ids,
662
+ }
663
+
664
+ @add_start_docstrings_to_model_forward(CODEGEN_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
665
+ @add_code_sample_docstrings(
666
+ processor_class=_TOKENIZER_FOR_DOC,
667
+ checkpoint=_CHECKPOINT_FOR_DOC,
668
+ output_type=CausalLMOutputWithPast,
669
+ config_class=_CONFIG_FOR_DOC,
670
+ )
671
+ def forward(
672
+ self,
673
+ input_ids: Optional[torch.LongTensor] = None,
674
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
675
+ attention_mask: Optional[torch.FloatTensor] = None,
676
+ token_type_ids: Optional[torch.LongTensor] = None,
677
+ position_ids: Optional[torch.LongTensor] = None,
678
+ head_mask: Optional[torch.FloatTensor] = None,
679
+ inputs_embeds: Optional[torch.FloatTensor] = None,
680
+ labels: Optional[torch.LongTensor] = None,
681
+ use_cache: Optional[bool] = None,
682
+ output_attentions: Optional[bool] = None,
683
+ output_hidden_states: Optional[bool] = None,
684
+ return_dict: Optional[bool] = None,
685
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
686
+ r"""
687
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
688
+ Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
689
+ `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
690
+ are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
691
+ """
692
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
693
+
694
+ transformer_outputs = self.transformer(
695
+ input_ids,
696
+ past_key_values=past_key_values,
697
+ attention_mask=attention_mask,
698
+ token_type_ids=token_type_ids,
699
+ position_ids=position_ids,
700
+ head_mask=head_mask,
701
+ inputs_embeds=inputs_embeds,
702
+ use_cache=use_cache,
703
+ output_attentions=output_attentions,
704
+ output_hidden_states=output_hidden_states,
705
+ return_dict=return_dict,
706
+ )
707
+ hidden_states = transformer_outputs[0]
708
+
709
+ # make sure sampling in fp16 works correctly and
710
+ # compute loss in fp32 to match with mesh-tf version
711
+ # https://github.com/EleutherAI/gpt-neo/blob/89ce74164da2fb16179106f54e2269b5da8db333/models/gpt2/gpt2.py#L179
712
+ lm_logits = self.lm_head(hidden_states).to(torch.float32)
713
+
714
+ loss = None
715
+ if labels is not None:
716
+ # Shift so that tokens < n predict n
717
+ shift_logits = lm_logits[..., :-1, :].contiguous()
718
+ shift_labels = labels[..., 1:].contiguous()
719
+ # Flatten the tokens
720
+ loss_fct = CrossEntropyLoss()
721
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
722
+
723
+ loss = loss.to(hidden_states.dtype)
724
+
725
+ if not return_dict:
726
+ output = (lm_logits,) + transformer_outputs[1:]
727
+ return ((loss,) + output) if loss is not None else output
728
+
729
+ return CausalLMOutputWithPast(
730
+ loss=loss,
731
+ logits=lm_logits,
732
+ past_key_values=transformer_outputs.past_key_values,
733
+ hidden_states=transformer_outputs.hidden_states,
734
+ attentions=transformer_outputs.attentions,
735
+ )
736
+
737
+ @staticmethod
738
+ def _reorder_cache(past: Tuple[Tuple[torch.Tensor]], beam_idx: torch.Tensor) -> Tuple[Tuple[torch.Tensor]]:
739
+ """
740
+ This function is used to re-order the `past_key_values` cache if [`~PretrainedModel.beam_search`] or
741
+ [`~PretrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
742
+ beam_idx at every generation step.
743
+ """
744
+ return tuple(
745
+ tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past)
746
+ for layer_past in past
747
+ )
pytorch_model-00001-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a07728aab3ba46f798215d99d76ca732fafd4f4c6a73f7998a96b1346c66fc8
3
+ size 9950218439
pytorch_model-00002-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:752d8f156068e8e5d84f9fe0f6816428ee38ca99e1073535cf42cfa4be6d808f
3
+ size 9782530030
pytorch_model-00003-of-00003.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:359ff18bf3eb161e7b97cf66980cc210b40f86674c16ad46fc0281fab6b31e57
3
+ size 7853000811
pytorch_model.bin.index.json ADDED
@@ -0,0 +1,300 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 27585650688
4
+ },
5
+ "weight_map": {
6
+ "lm_head.bias": "pytorch_model-00003-of-00003.bin",
7
+ "lm_head.weight": "pytorch_model-00003-of-00003.bin",
8
+ "transformer.h.0.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
9
+ "transformer.h.0.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
10
+ "transformer.h.0.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
11
+ "transformer.h.0.ln_1.bias": "pytorch_model-00001-of-00003.bin",
12
+ "transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00003.bin",
13
+ "transformer.h.0.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
14
+ "transformer.h.0.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
15
+ "transformer.h.0.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
16
+ "transformer.h.0.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
17
+ "transformer.h.1.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
18
+ "transformer.h.1.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
19
+ "transformer.h.1.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
20
+ "transformer.h.1.ln_1.bias": "pytorch_model-00001-of-00003.bin",
21
+ "transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00003.bin",
22
+ "transformer.h.1.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
23
+ "transformer.h.1.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
24
+ "transformer.h.1.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
25
+ "transformer.h.1.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
26
+ "transformer.h.10.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
27
+ "transformer.h.10.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
28
+ "transformer.h.10.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
29
+ "transformer.h.10.ln_1.bias": "pytorch_model-00001-of-00003.bin",
30
+ "transformer.h.10.ln_1.weight": "pytorch_model-00001-of-00003.bin",
31
+ "transformer.h.10.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
32
+ "transformer.h.10.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
33
+ "transformer.h.10.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
34
+ "transformer.h.10.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
35
+ "transformer.h.11.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
36
+ "transformer.h.11.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
37
+ "transformer.h.11.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
38
+ "transformer.h.11.ln_1.bias": "pytorch_model-00001-of-00003.bin",
39
+ "transformer.h.11.ln_1.weight": "pytorch_model-00001-of-00003.bin",
40
+ "transformer.h.11.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
41
+ "transformer.h.11.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
42
+ "transformer.h.11.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
43
+ "transformer.h.11.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
44
+ "transformer.h.12.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
45
+ "transformer.h.12.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
46
+ "transformer.h.12.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
47
+ "transformer.h.12.ln_1.bias": "pytorch_model-00002-of-00003.bin",
48
+ "transformer.h.12.ln_1.weight": "pytorch_model-00002-of-00003.bin",
49
+ "transformer.h.12.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
50
+ "transformer.h.12.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
51
+ "transformer.h.12.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
52
+ "transformer.h.12.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
53
+ "transformer.h.13.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
54
+ "transformer.h.13.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
55
+ "transformer.h.13.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
56
+ "transformer.h.13.ln_1.bias": "pytorch_model-00002-of-00003.bin",
57
+ "transformer.h.13.ln_1.weight": "pytorch_model-00002-of-00003.bin",
58
+ "transformer.h.13.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
59
+ "transformer.h.13.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
60
+ "transformer.h.13.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
61
+ "transformer.h.13.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
62
+ "transformer.h.14.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
63
+ "transformer.h.14.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
64
+ "transformer.h.14.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
65
+ "transformer.h.14.ln_1.bias": "pytorch_model-00002-of-00003.bin",
66
+ "transformer.h.14.ln_1.weight": "pytorch_model-00002-of-00003.bin",
67
+ "transformer.h.14.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
68
+ "transformer.h.14.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
69
+ "transformer.h.14.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
70
+ "transformer.h.14.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
71
+ "transformer.h.15.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
72
+ "transformer.h.15.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
73
+ "transformer.h.15.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
74
+ "transformer.h.15.ln_1.bias": "pytorch_model-00002-of-00003.bin",
75
+ "transformer.h.15.ln_1.weight": "pytorch_model-00002-of-00003.bin",
76
+ "transformer.h.15.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
77
+ "transformer.h.15.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
78
+ "transformer.h.15.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
79
+ "transformer.h.15.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
80
+ "transformer.h.16.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
81
+ "transformer.h.16.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
82
+ "transformer.h.16.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
83
+ "transformer.h.16.ln_1.bias": "pytorch_model-00002-of-00003.bin",
84
+ "transformer.h.16.ln_1.weight": "pytorch_model-00002-of-00003.bin",
85
+ "transformer.h.16.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
86
+ "transformer.h.16.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
87
+ "transformer.h.16.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
88
+ "transformer.h.16.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
89
+ "transformer.h.17.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
90
+ "transformer.h.17.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
91
+ "transformer.h.17.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
92
+ "transformer.h.17.ln_1.bias": "pytorch_model-00002-of-00003.bin",
93
+ "transformer.h.17.ln_1.weight": "pytorch_model-00002-of-00003.bin",
94
+ "transformer.h.17.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
95
+ "transformer.h.17.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
96
+ "transformer.h.17.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
97
+ "transformer.h.17.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
98
+ "transformer.h.18.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
99
+ "transformer.h.18.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
100
+ "transformer.h.18.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
101
+ "transformer.h.18.ln_1.bias": "pytorch_model-00002-of-00003.bin",
102
+ "transformer.h.18.ln_1.weight": "pytorch_model-00002-of-00003.bin",
103
+ "transformer.h.18.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
104
+ "transformer.h.18.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
105
+ "transformer.h.18.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
106
+ "transformer.h.18.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
107
+ "transformer.h.19.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
108
+ "transformer.h.19.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
109
+ "transformer.h.19.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
110
+ "transformer.h.19.ln_1.bias": "pytorch_model-00002-of-00003.bin",
111
+ "transformer.h.19.ln_1.weight": "pytorch_model-00002-of-00003.bin",
112
+ "transformer.h.19.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
113
+ "transformer.h.19.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
114
+ "transformer.h.19.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
115
+ "transformer.h.19.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
116
+ "transformer.h.2.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
117
+ "transformer.h.2.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
118
+ "transformer.h.2.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
119
+ "transformer.h.2.ln_1.bias": "pytorch_model-00001-of-00003.bin",
120
+ "transformer.h.2.ln_1.weight": "pytorch_model-00001-of-00003.bin",
121
+ "transformer.h.2.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
122
+ "transformer.h.2.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
123
+ "transformer.h.2.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
124
+ "transformer.h.2.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
125
+ "transformer.h.20.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
126
+ "transformer.h.20.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
127
+ "transformer.h.20.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
128
+ "transformer.h.20.ln_1.bias": "pytorch_model-00002-of-00003.bin",
129
+ "transformer.h.20.ln_1.weight": "pytorch_model-00002-of-00003.bin",
130
+ "transformer.h.20.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
131
+ "transformer.h.20.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
132
+ "transformer.h.20.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
133
+ "transformer.h.20.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
134
+ "transformer.h.21.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
135
+ "transformer.h.21.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
136
+ "transformer.h.21.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
137
+ "transformer.h.21.ln_1.bias": "pytorch_model-00002-of-00003.bin",
138
+ "transformer.h.21.ln_1.weight": "pytorch_model-00002-of-00003.bin",
139
+ "transformer.h.21.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
140
+ "transformer.h.21.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
141
+ "transformer.h.21.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
142
+ "transformer.h.21.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
143
+ "transformer.h.22.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
144
+ "transformer.h.22.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
145
+ "transformer.h.22.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
146
+ "transformer.h.22.ln_1.bias": "pytorch_model-00002-of-00003.bin",
147
+ "transformer.h.22.ln_1.weight": "pytorch_model-00002-of-00003.bin",
148
+ "transformer.h.22.mlp.fc_in.bias": "pytorch_model-00002-of-00003.bin",
149
+ "transformer.h.22.mlp.fc_in.weight": "pytorch_model-00002-of-00003.bin",
150
+ "transformer.h.22.mlp.fc_out.bias": "pytorch_model-00002-of-00003.bin",
151
+ "transformer.h.22.mlp.fc_out.weight": "pytorch_model-00002-of-00003.bin",
152
+ "transformer.h.23.attn.causal_mask": "pytorch_model-00002-of-00003.bin",
153
+ "transformer.h.23.attn.out_proj.weight": "pytorch_model-00002-of-00003.bin",
154
+ "transformer.h.23.attn.qkv_proj.weight": "pytorch_model-00002-of-00003.bin",
155
+ "transformer.h.23.ln_1.bias": "pytorch_model-00002-of-00003.bin",
156
+ "transformer.h.23.ln_1.weight": "pytorch_model-00002-of-00003.bin",
157
+ "transformer.h.23.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
158
+ "transformer.h.23.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
159
+ "transformer.h.23.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
160
+ "transformer.h.23.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
161
+ "transformer.h.24.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
162
+ "transformer.h.24.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
163
+ "transformer.h.24.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
164
+ "transformer.h.24.ln_1.bias": "pytorch_model-00003-of-00003.bin",
165
+ "transformer.h.24.ln_1.weight": "pytorch_model-00003-of-00003.bin",
166
+ "transformer.h.24.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
167
+ "transformer.h.24.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
168
+ "transformer.h.24.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
169
+ "transformer.h.24.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
170
+ "transformer.h.25.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
171
+ "transformer.h.25.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
172
+ "transformer.h.25.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
173
+ "transformer.h.25.ln_1.bias": "pytorch_model-00003-of-00003.bin",
174
+ "transformer.h.25.ln_1.weight": "pytorch_model-00003-of-00003.bin",
175
+ "transformer.h.25.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
176
+ "transformer.h.25.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
177
+ "transformer.h.25.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
178
+ "transformer.h.25.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
179
+ "transformer.h.26.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
180
+ "transformer.h.26.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
181
+ "transformer.h.26.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
182
+ "transformer.h.26.ln_1.bias": "pytorch_model-00003-of-00003.bin",
183
+ "transformer.h.26.ln_1.weight": "pytorch_model-00003-of-00003.bin",
184
+ "transformer.h.26.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
185
+ "transformer.h.26.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
186
+ "transformer.h.26.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
187
+ "transformer.h.26.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
188
+ "transformer.h.27.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
189
+ "transformer.h.27.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
190
+ "transformer.h.27.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
191
+ "transformer.h.27.ln_1.bias": "pytorch_model-00003-of-00003.bin",
192
+ "transformer.h.27.ln_1.weight": "pytorch_model-00003-of-00003.bin",
193
+ "transformer.h.27.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
194
+ "transformer.h.27.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
195
+ "transformer.h.27.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
196
+ "transformer.h.27.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
197
+ "transformer.h.28.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
198
+ "transformer.h.28.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
199
+ "transformer.h.28.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
200
+ "transformer.h.28.ln_1.bias": "pytorch_model-00003-of-00003.bin",
201
+ "transformer.h.28.ln_1.weight": "pytorch_model-00003-of-00003.bin",
202
+ "transformer.h.28.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
203
+ "transformer.h.28.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
204
+ "transformer.h.28.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
205
+ "transformer.h.28.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
206
+ "transformer.h.29.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
207
+ "transformer.h.29.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
208
+ "transformer.h.29.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
209
+ "transformer.h.29.ln_1.bias": "pytorch_model-00003-of-00003.bin",
210
+ "transformer.h.29.ln_1.weight": "pytorch_model-00003-of-00003.bin",
211
+ "transformer.h.29.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
212
+ "transformer.h.29.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
213
+ "transformer.h.29.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
214
+ "transformer.h.29.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
215
+ "transformer.h.3.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
216
+ "transformer.h.3.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
217
+ "transformer.h.3.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
218
+ "transformer.h.3.ln_1.bias": "pytorch_model-00001-of-00003.bin",
219
+ "transformer.h.3.ln_1.weight": "pytorch_model-00001-of-00003.bin",
220
+ "transformer.h.3.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
221
+ "transformer.h.3.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
222
+ "transformer.h.3.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
223
+ "transformer.h.3.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
224
+ "transformer.h.30.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
225
+ "transformer.h.30.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
226
+ "transformer.h.30.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
227
+ "transformer.h.30.ln_1.bias": "pytorch_model-00003-of-00003.bin",
228
+ "transformer.h.30.ln_1.weight": "pytorch_model-00003-of-00003.bin",
229
+ "transformer.h.30.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
230
+ "transformer.h.30.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
231
+ "transformer.h.30.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
232
+ "transformer.h.30.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
233
+ "transformer.h.31.attn.causal_mask": "pytorch_model-00003-of-00003.bin",
234
+ "transformer.h.31.attn.out_proj.weight": "pytorch_model-00003-of-00003.bin",
235
+ "transformer.h.31.attn.qkv_proj.weight": "pytorch_model-00003-of-00003.bin",
236
+ "transformer.h.31.ln_1.bias": "pytorch_model-00003-of-00003.bin",
237
+ "transformer.h.31.ln_1.weight": "pytorch_model-00003-of-00003.bin",
238
+ "transformer.h.31.mlp.fc_in.bias": "pytorch_model-00003-of-00003.bin",
239
+ "transformer.h.31.mlp.fc_in.weight": "pytorch_model-00003-of-00003.bin",
240
+ "transformer.h.31.mlp.fc_out.bias": "pytorch_model-00003-of-00003.bin",
241
+ "transformer.h.31.mlp.fc_out.weight": "pytorch_model-00003-of-00003.bin",
242
+ "transformer.h.4.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
243
+ "transformer.h.4.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
244
+ "transformer.h.4.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
245
+ "transformer.h.4.ln_1.bias": "pytorch_model-00001-of-00003.bin",
246
+ "transformer.h.4.ln_1.weight": "pytorch_model-00001-of-00003.bin",
247
+ "transformer.h.4.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
248
+ "transformer.h.4.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
249
+ "transformer.h.4.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
250
+ "transformer.h.4.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
251
+ "transformer.h.5.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
252
+ "transformer.h.5.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
253
+ "transformer.h.5.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
254
+ "transformer.h.5.ln_1.bias": "pytorch_model-00001-of-00003.bin",
255
+ "transformer.h.5.ln_1.weight": "pytorch_model-00001-of-00003.bin",
256
+ "transformer.h.5.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
257
+ "transformer.h.5.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
258
+ "transformer.h.5.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
259
+ "transformer.h.5.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
260
+ "transformer.h.6.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
261
+ "transformer.h.6.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
262
+ "transformer.h.6.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
263
+ "transformer.h.6.ln_1.bias": "pytorch_model-00001-of-00003.bin",
264
+ "transformer.h.6.ln_1.weight": "pytorch_model-00001-of-00003.bin",
265
+ "transformer.h.6.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
266
+ "transformer.h.6.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
267
+ "transformer.h.6.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
268
+ "transformer.h.6.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
269
+ "transformer.h.7.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
270
+ "transformer.h.7.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
271
+ "transformer.h.7.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
272
+ "transformer.h.7.ln_1.bias": "pytorch_model-00001-of-00003.bin",
273
+ "transformer.h.7.ln_1.weight": "pytorch_model-00001-of-00003.bin",
274
+ "transformer.h.7.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
275
+ "transformer.h.7.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
276
+ "transformer.h.7.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
277
+ "transformer.h.7.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
278
+ "transformer.h.8.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
279
+ "transformer.h.8.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
280
+ "transformer.h.8.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
281
+ "transformer.h.8.ln_1.bias": "pytorch_model-00001-of-00003.bin",
282
+ "transformer.h.8.ln_1.weight": "pytorch_model-00001-of-00003.bin",
283
+ "transformer.h.8.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
284
+ "transformer.h.8.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
285
+ "transformer.h.8.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
286
+ "transformer.h.8.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
287
+ "transformer.h.9.attn.causal_mask": "pytorch_model-00001-of-00003.bin",
288
+ "transformer.h.9.attn.out_proj.weight": "pytorch_model-00001-of-00003.bin",
289
+ "transformer.h.9.attn.qkv_proj.weight": "pytorch_model-00001-of-00003.bin",
290
+ "transformer.h.9.ln_1.bias": "pytorch_model-00001-of-00003.bin",
291
+ "transformer.h.9.ln_1.weight": "pytorch_model-00001-of-00003.bin",
292
+ "transformer.h.9.mlp.fc_in.bias": "pytorch_model-00001-of-00003.bin",
293
+ "transformer.h.9.mlp.fc_in.weight": "pytorch_model-00001-of-00003.bin",
294
+ "transformer.h.9.mlp.fc_out.bias": "pytorch_model-00001-of-00003.bin",
295
+ "transformer.h.9.mlp.fc_out.weight": "pytorch_model-00001-of-00003.bin",
296
+ "transformer.ln_f.bias": "pytorch_model-00003-of-00003.bin",
297
+ "transformer.ln_f.weight": "pytorch_model-00003-of-00003.bin",
298
+ "transformer.wte.weight": "pytorch_model-00001-of-00003.bin"
299
+ }
300
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<|endoftext|>",
3
+ "eos_token": "<|endoftext|>",
4
+ "unk_token": "<|endoftext|>"
5
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<|endoftext|>",
4
+ "eos_token": "<|endoftext|>",
5
+ "model_max_length": 1024,
6
+ "name_or_path": "gpt2",
7
+ "special_tokens_map_file": null,
8
+ "tokenizer_class": "GPT2Tokenizer",
9
+ "unk_token": "<|endoftext|>"
10
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff