lysandre HF staff commited on
Commit
f5e146e
1 Parent(s): 65a62d7

Update code samples

Browse files
Files changed (1) hide show
  1. README.md +71 -68
README.md CHANGED
@@ -10,8 +10,7 @@ datasets:
10
 
11
  Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
12
  [this paper](https://arxiv.org/abs/1810.04805) and first released in
13
- [this repository](https://github.com/google-research/bert). This model is cased: it does not make a difference
14
- between english and English.
15
 
16
  Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.
17
 
@@ -59,32 +58,36 @@ You can use this model directly with a pipeline for masked language modeling:
59
  >>> unmasker = pipeline('fill-mask', model='bert-large-cased-whole-word-masking')
60
  >>> unmasker("Hello I'm a [MASK] model.")
61
  [
62
- {
63
- 'sequence': "[CLS] hello i'm a fashion model. [SEP]",
64
- 'score': 0.15813860297203064,
65
- 'token': 4827,
66
- 'token_str': 'fashion'
67
- }, {
68
- 'sequence': "[CLS] hello i'm a cover model. [SEP]",
69
- 'score': 0.10551052540540695,
70
- 'token': 3104,
71
- 'token_str': 'cover'
72
- }, {
73
- 'sequence': "[CLS] hello i'm a male model. [SEP]",
74
- 'score': 0.08340442180633545,
75
- 'token': 3287,
76
- 'token_str': 'male'
77
- }, {
78
- 'sequence': "[CLS] hello i'm a super model. [SEP]",
79
- 'score': 0.036381796002388,
80
- 'token': 3565,
81
- 'token_str': 'super'
82
- }, {
83
- 'sequence': "[CLS] hello i'm a top model. [SEP]",
84
- 'score': 0.03609578311443329,
85
- 'token': 2327,
86
- 'token_str': 'top'
87
- }
 
 
 
 
88
  ]
89
  ```
90
 
@@ -121,68 +124,69 @@ predictions:
121
  >>> unmasker("The man worked as a [MASK].")
122
  [
123
  {
124
- "sequence":"[CLS] the man worked as a waiter. [SEP]",
125
- "score":0.09823174774646759,
126
- "token":15610,
127
- "token_str":"waiter"
128
  },
129
  {
130
- "sequence":"[CLS] the man worked as a carpenter. [SEP]",
131
- "score":0.08976428955793381,
132
- "token":10533,
133
- "token_str":"carpenter"
134
  },
135
  {
136
- "sequence":"[CLS] the man worked as a mechanic. [SEP]",
137
- "score":0.06550426036119461,
138
- "token":15893,
139
  "token_str":"mechanic"
140
  },
141
  {
142
- "sequence":"[CLS] the man worked as a butcher. [SEP]",
143
- "score":0.04142395779490471,
144
- "token":14998,
145
- "token_str":"butcher"
146
  },
147
  {
148
- "sequence":"[CLS] the man worked as a barber. [SEP]",
149
- "score":0.03680137172341347,
150
- "token":13362,
151
- "token_str":"barber"
152
  }
153
  ]
154
 
 
155
  >>> unmasker("The woman worked as a [MASK].")
156
  [
157
  {
158
- "sequence":"[CLS] the woman worked as a waitress. [SEP]",
159
- "score":0.2669651508331299,
160
- "token":13877,
161
- "token_str":"waitress"
162
  },
163
  {
164
- "sequence":"[CLS] the woman worked as a maid. [SEP]",
165
- "score":0.13054853677749634,
166
- "token":10850,
167
- "token_str":"maid"
168
  },
169
  {
170
- "sequence":"[CLS] the woman worked as a nurse. [SEP]",
171
- "score":0.07987703382968903,
172
- "token":6821,
173
  "token_str":"nurse"
174
  },
175
  {
176
- "sequence":"[CLS] the woman worked as a prostitute. [SEP]",
177
- "score":0.058545831590890884,
178
- "token":19215,
179
- "token_str":"prostitute"
180
  },
181
  {
182
- "sequence":"[CLS] the woman worked as a cleaner. [SEP]",
183
- "score":0.03834161534905434,
184
- "token":20133,
185
- "token_str":"cleaner"
186
  }
187
  ]
188
  ```
@@ -230,8 +234,7 @@ When fine-tuned on downstream tasks, this model achieves the following results:
230
 
231
  Model | SQUAD 1.1 F1/EM | Multi NLI Accuracy
232
  ---------------------------------------- | :-------------: | :----------------:
233
- BERT-Large, Uncased (Whole Word Masking) | 92.8/86.7 | 87.07
234
-
235
 
236
  ### BibTeX entry and citation info
237
 
 
10
 
11
  Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
12
  [this paper](https://arxiv.org/abs/1810.04805) and first released in
13
+ [this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.
 
14
 
15
  Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.
16
 
 
58
  >>> unmasker = pipeline('fill-mask', model='bert-large-cased-whole-word-masking')
59
  >>> unmasker("Hello I'm a [MASK] model.")
60
  [
61
+ {
62
+ "sequence":"[CLS] Hello I'm a fashion model. [SEP]",
63
+ "score":0.1474294513463974,
64
+ "token":4633,
65
+ "token_str":"fashion"
66
+ },
67
+ {
68
+ "sequence":"[CLS] Hello I'm a magazine model. [SEP]",
69
+ "score":0.05430116504430771,
70
+ "token":2435,
71
+ "token_str":"magazine"
72
+ },
73
+ {
74
+ "sequence":"[CLS] Hello I'm a male model. [SEP]",
75
+ "score":0.039395421743392944,
76
+ "token":2581,
77
+ "token_str":"male"
78
+ },
79
+ {
80
+ "sequence":"[CLS] Hello I'm a former model. [SEP]",
81
+ "score":0.036936815828084946,
82
+ "token":1393,
83
+ "token_str":"former"
84
+ },
85
+ {
86
+ "sequence":"[CLS] Hello I'm a professional model. [SEP]",
87
+ "score":0.03663451969623566,
88
+ "token":1848,
89
+ "token_str":"professional"
90
+ }
91
  ]
92
  ```
93
 
 
124
  >>> unmasker("The man worked as a [MASK].")
125
  [
126
  {
127
+ "sequence":"[CLS] The man worked as a carpenter. [SEP]",
128
+ "score":0.09021259099245071,
129
+ "token":25169,
130
+ "token_str":"carpenter"
131
  },
132
  {
133
+ "sequence":"[CLS] The man worked as a cook. [SEP]",
134
+ "score":0.08125395327806473,
135
+ "token":9834,
136
+ "token_str":"cook"
137
  },
138
  {
139
+ "sequence":"[CLS] The man worked as a mechanic. [SEP]",
140
+ "score":0.07524766772985458,
141
+ "token":19459,
142
  "token_str":"mechanic"
143
  },
144
  {
145
+ "sequence":"[CLS] The man worked as a waiter. [SEP]",
146
+ "score":0.07397029548883438,
147
+ "token":17989,
148
+ "token_str":"waiter"
149
  },
150
  {
151
+ "sequence":"[CLS] The man worked as a guard. [SEP]",
152
+ "score":0.05848982185125351,
153
+ "token":3542,
154
+ "token_str":"guard"
155
  }
156
  ]
157
 
158
+
159
  >>> unmasker("The woman worked as a [MASK].")
160
  [
161
  {
162
+ "sequence":"[CLS] The woman worked as a maid. [SEP]",
163
+ "score":0.19436432421207428,
164
+ "token":13487,
165
+ "token_str":"maid"
166
  },
167
  {
168
+ "sequence":"[CLS] The woman worked as a waitress. [SEP]",
169
+ "score":0.16161060333251953,
170
+ "token":15098,
171
+ "token_str":"waitress"
172
  },
173
  {
174
+ "sequence":"[CLS] The woman worked as a nurse. [SEP]",
175
+ "score":0.14942803978919983,
176
+ "token":7439,
177
  "token_str":"nurse"
178
  },
179
  {
180
+ "sequence":"[CLS] The woman worked as a secretary. [SEP]",
181
+ "score":0.10373266786336899,
182
+ "token":4848,
183
+ "token_str":"secretary"
184
  },
185
  {
186
+ "sequence":"[CLS] The woman worked as a cook. [SEP]",
187
+ "score":0.06384387612342834,
188
+ "token":9834,
189
+ "token_str":"cook"
190
  }
191
  ]
192
  ```
 
234
 
235
  Model | SQUAD 1.1 F1/EM | Multi NLI Accuracy
236
  ---------------------------------------- | :-------------: | :----------------:
237
+ BERT-Large, Cased (Whole Word Masking) | 92.9/86.7 | 86.46
 
238
 
239
  ### BibTeX entry and citation info
240