Hugging Face's logo

cardiffnlp
/
twitter-roberta-base-2022-154m

Fill-Mask
Transformers
PyTorch
English
roberta
timelms
twitter
Inference Endpoints
Model card Files Community
3
danlou commited on
Commit
ac520b1
•
1 Parent(s): fd99dd7

Upload 14 files

Browse files
Files changed (14) hide show
  1. README.md +161 -0
  2. config.json +27 -0
  3. merges.txt +0 -0
  4. optimizer.pt +3 -0
  5. pytorch_model.bin +3 -0
  6. rng_state.pth +3 -0
  7. scaler.pt +3 -0
  8. scheduler.pt +3 -0
  9. special_tokens_map.json +15 -0
  10. tokenizer.json +0 -0
  11. tokenizer_config.json +16 -0
  12. trainer_state.json +907 -0
  13. training_args.bin +3 -0
  14. vocab.json +0 -0
README.md ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - timelms
5
+ - twitter
6
+ license: mit
7
+ datasets:
8
+ - twitter-api
9
+ ---
10
+
11
+ # Twitter 2022 154M (RoBERTa-base, 154M - full update)
12
+
13
+ This is a RoBERTa-base model trained on 154M tweets until the end of December 2022 (from original checkpoint, no incremental updates).
14
+ These 154M tweets result from filtering 220M tweets obtained exclusively from the Twitter Academic API, covering every month between 2018-01 and 2022-12.
15
+ Filtering and preprocessing details are available in the [TimeLMs paper](https://arxiv.org/abs/2202.03829).
16
+
17
+ Below, we provide some usage examples using the standard Transformers interface. For another interface more suited to comparing predictions and perplexity scores between models trained at different temporal intervals, check the [TimeLMs repository](https://github.com/cardiffnlp/timelms).
18
+
19
+ For other models trained until different periods, check this [table](https://github.com/cardiffnlp/timelms#released-models).
20
+
21
+ ## Preprocess Text
22
+ Replace usernames and links for placeholders: "@user" and "http".
23
+ If you're interested in retaining verified users which were also retained during training, you may keep the users listed [here](https://github.com/cardiffnlp/timelms/tree/main/data).
24
+ ```python
25
+ def preprocess(text):
26
+ preprocessed_text = []
27
+ for t in text.split():
28
+ if len(t) > 1:
29
+ t = '@user' if t[0] == '@' and t.count('@') == 1 else t
30
+ t = 'http' if t.startswith('http') else t
31
+ preprocessed_text.append(t)
32
+ return ' '.join(preprocessed_text)
33
+ ```
34
+
35
+ ## Example Masked Language Model
36
+
37
+ ```python
38
+ from transformers import pipeline, AutoTokenizer
39
+
40
+ MODEL = "cardiffnlp/twitter-roberta-base-2022-154m"
41
+ fill_mask = pipeline("fill-mask", model=MODEL, tokenizer=MODEL)
42
+ tokenizer = AutoTokenizer.from_pretrained(MODEL)
43
+
44
+ def pprint(candidates, n):
45
+ for i in range(n):
46
+ token = tokenizer.decode(candidates[i]['token'])
47
+ score = candidates[i]['score']
48
+ print("%d) %.5f %s" % (i+1, score, token))
49
+
50
+ texts = [
51
+ "So glad I'm <mask> vaccinated.",
52
+ "I keep forgetting to bring a <mask>.",
53
+ "Looking forward to watching <mask> Game tonight!",
54
+ ]
55
+ for text in texts:
56
+ t = preprocess(text)
57
+ print(f"{'-'*30}\n{t}")
58
+ candidates = fill_mask(t)
59
+ pprint(candidates, 5)
60
+ ```
61
+
62
+ Output:
63
+
64
+ ```
65
+ ------------------------------
66
+ So glad I'm <mask> vaccinated.
67
+ 1) 0.60140 not
68
+ 2) 0.15077 getting
69
+ 3) 0.12119 fully
70
+ 4) 0.02203 still
71
+ 5) 0.01020 all
72
+ ------------------------------
73
+ I keep forgetting to bring a <mask>.
74
+ 1) 0.05812 charger
75
+ 2) 0.05040 backpack
76
+ 3) 0.05004 book
77
+ 4) 0.04548 bag
78
+ 5) 0.03992 lighter
79
+ ------------------------------
80
+ Looking forward to watching <mask> Game tonight!
81
+ 1) 0.39552 the
82
+ 2) 0.28083 The
83
+ 3) 0.02029 End
84
+ 4) 0.01878 Squid
85
+ 5) 0.01438 this
86
+ ```
87
+
88
+ ## Example Tweet Embeddings
89
+ ```python
90
+ from transformers import AutoTokenizer, AutoModel, TFAutoModel
91
+ import numpy as np
92
+ from scipy.spatial.distance import cosine
93
+ from collections import Counter
94
+
95
+ def get_embedding(text): # naive approach for demonstration
96
+ text = preprocess(text)
97
+ encoded_input = tokenizer(text, return_tensors='pt')
98
+ features = model(**encoded_input)
99
+ features = features[0].detach().cpu().numpy()
100
+ return np.mean(features[0], axis=0)
101
+
102
+
103
+ MODEL = "cardiffnlp/twitter-roberta-base-2022-154m"
104
+ tokenizer = AutoTokenizer.from_pretrained(MODEL)
105
+ model = AutoModel.from_pretrained(MODEL)
106
+
107
+ query = "The book was awesome"
108
+ tweets = ["I just ordered fried chicken 🐣",
109
+ "The movie was great",
110
+ "What time is the next game?",
111
+ "Just finished reading 'Embeddings in NLP'"]
112
+
113
+ sims = Counter()
114
+ for tweet in tweets:
115
+ sim = 1 - cosine(get_embedding(query), get_embedding(tweet))
116
+ sims[tweet] = sim
117
+
118
+ print('Most similar to: ', query)
119
+ print(f"{'-'*30}")
120
+ for idx, (tweet, sim) in enumerate(sims.most_common()):
121
+ print("%d) %.5f %s" % (idx+1, sim, tweet))
122
+ ```
123
+ Output:
124
+
125
+ ```
126
+ Most similar to: The book was awesome
127
+ ------------------------------
128
+ 1) 0.98914 The movie was great
129
+ 2) 0.96194 Just finished reading 'Embeddings in NLP'
130
+ 3) 0.94603 What time is the next game?
131
+ 4) 0.94580 I just ordered fried chicken 🐣
132
+ ```
133
+
134
+ ## Example Feature Extraction
135
+
136
+ ```python
137
+ from transformers import AutoTokenizer, AutoModel, TFAutoModel
138
+ import numpy as np
139
+
140
+ MODEL = "cardiffnlp/twitter-roberta-base-2022-154m"
141
+ tokenizer = AutoTokenizer.from_pretrained(MODEL)
142
+
143
+ text = "Good night 😊"
144
+ text = preprocess(text)
145
+
146
+ # Pytorch
147
+ model = AutoModel.from_pretrained(MODEL)
148
+ encoded_input = tokenizer(text, return_tensors='pt')
149
+ features = model(**encoded_input)
150
+ features = features[0].detach().cpu().numpy()
151
+ features_mean = np.mean(features[0], axis=0)
152
+ #features_max = np.max(features[0], axis=0)
153
+
154
+ # # Tensorflow
155
+ # model = TFAutoModel.from_pretrained(MODEL)
156
+ # encoded_input = tokenizer(text, return_tensors='tf')
157
+ # features = model(encoded_input)
158
+ # features = features[0].numpy()
159
+ # features_mean = np.mean(features[0], axis=0)
160
+ # #features_max = np.max(features[0], axis=0)
161
+ ```
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "roberta-base",
3
+ "architectures": [
4
+ "RobertaForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": 0,
8
+ "classifier_dropout": null,
9
+ "eos_token_id": 2,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-05,
16
+ "max_position_embeddings": 514,
17
+ "model_type": "roberta",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 12,
20
+ "pad_token_id": 1,
21
+ "position_embedding_type": "absolute",
22
+ "torch_dtype": "float32",
23
+ "transformers_version": "4.27.0.dev0",
24
+ "type_vocab_size": 1,
25
+ "use_cache": true,
26
+ "vocab_size": 50265
27
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee102536c94f31edde08526cf9727ec7cda55a1ffe050c90cf59c914978ec101
3
+ size 997696473
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:222d589484e0b5b5a69fb3305e171ddd46456c662cc34644e6f80a228c8a47fc
3
+ size 498861675
rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f899aba43676eaadd99f526ba5c91001b01065923c37e1aebe33dc8746e60a7e
3
+ size 21579
scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7db3637ff6d8fdf8d724111c86ee2fb5f41dedf8d3cb14acfa63d0ed830a0d68
3
+ size 559
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:693f659bcad9eaafb36a940b28a69a24887719e7d09301b507e68c3c611acafa
3
+ size 623
special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "cls_token": "<s>",
4
+ "eos_token": "</s>",
5
+ "mask_token": {
6
+ "content": "<mask>",
7
+ "lstrip": true,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "pad_token": "<pad>",
13
+ "sep_token": "</s>",
14
+ "unk_token": "<unk>"
15
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<s>",
4
+ "cls_token": "<s>",
5
+ "eos_token": "</s>",
6
+ "errors": "replace",
7
+ "mask_token": "<mask>",
8
+ "model_max_length": 512,
9
+ "name_or_path": "roberta-base",
10
+ "pad_token": "<pad>",
11
+ "sep_token": "</s>",
12
+ "special_tokens_map_file": null,
13
+ "tokenizer_class": "RobertaTokenizer",
14
+ "trim_offsets": true,
15
+ "unk_token": "<unk>"
16
+ }
trainer_state.json ADDED
@@ -0,0 +1,907 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.5714025899946678,
5
+ "global_step": 54000,
6
+ "is_hyper_param_search": false,
7
+ "is_local_process_zero": true,
8
+ "is_world_process_zero": true,
9
+ "log_history": [
10
+ {
11
+ "epoch": 0.03,
12
+ "learning_rate": 5e-06,
13
+ "loss": 2.5739,
14
+ "step": 500
15
+ },
16
+ {
17
+ "epoch": 0.07,
18
+ "learning_rate": 1e-05,
19
+ "loss": 2.3712,
20
+ "step": 1000
21
+ },
22
+ {
23
+ "epoch": 0.1,
24
+ "learning_rate": 9.999669290296979e-06,
25
+ "loss": 2.3057,
26
+ "step": 1500
27
+ },
28
+ {
29
+ "epoch": 0.13,
30
+ "learning_rate": 9.999338580593956e-06,
31
+ "loss": 2.2676,
32
+ "step": 2000
33
+ },
34
+ {
35
+ "epoch": 0.13,
36
+ "eval_accuracy": 0.624435573588116,
37
+ "eval_loss": 2.00943922996521,
38
+ "eval_runtime": 4.2994,
39
+ "eval_samples_per_second": 930.363,
40
+ "eval_steps_per_second": 1.628,
41
+ "step": 2000
42
+ },
43
+ {
44
+ "epoch": 0.17,
45
+ "learning_rate": 9.999007870890934e-06,
46
+ "loss": 2.2392,
47
+ "step": 2500
48
+ },
49
+ {
50
+ "epoch": 0.2,
51
+ "learning_rate": 9.99867716118791e-06,
52
+ "loss": 2.2181,
53
+ "step": 3000
54
+ },
55
+ {
56
+ "epoch": 0.23,
57
+ "learning_rate": 9.998346451484888e-06,
58
+ "loss": 2.2015,
59
+ "step": 3500
60
+ },
61
+ {
62
+ "epoch": 0.26,
63
+ "learning_rate": 9.998016403201271e-06,
64
+ "loss": 2.1881,
65
+ "step": 4000
66
+ },
67
+ {
68
+ "epoch": 0.26,
69
+ "eval_accuracy": 0.6230075777371309,
70
+ "eval_loss": 1.980251431465149,
71
+ "eval_runtime": 4.1905,
72
+ "eval_samples_per_second": 954.545,
73
+ "eval_steps_per_second": 1.67,
74
+ "step": 4000
75
+ },
76
+ {
77
+ "epoch": 0.3,
78
+ "learning_rate": 9.997685693498247e-06,
79
+ "loss": 2.1754,
80
+ "step": 4500
81
+ },
82
+ {
83
+ "epoch": 0.33,
84
+ "learning_rate": 9.997354983795225e-06,
85
+ "loss": 2.1646,
86
+ "step": 5000
87
+ },
88
+ {
89
+ "epoch": 0.36,
90
+ "learning_rate": 9.997024274092203e-06,
91
+ "loss": 2.155,
92
+ "step": 5500
93
+ },
94
+ {
95
+ "epoch": 0.4,
96
+ "learning_rate": 9.99669356438918e-06,
97
+ "loss": 2.1462,
98
+ "step": 6000
99
+ },
100
+ {
101
+ "epoch": 0.4,
102
+ "eval_accuracy": 0.6319083064356762,
103
+ "eval_loss": 1.9461042881011963,
104
+ "eval_runtime": 4.2092,
105
+ "eval_samples_per_second": 950.295,
106
+ "eval_steps_per_second": 1.663,
107
+ "step": 6000
108
+ },
109
+ {
110
+ "epoch": 0.43,
111
+ "learning_rate": 9.996362854686157e-06,
112
+ "loss": 2.1369,
113
+ "step": 6500
114
+ },
115
+ {
116
+ "epoch": 0.46,
117
+ "learning_rate": 9.996032144983134e-06,
118
+ "loss": 2.129,
119
+ "step": 7000
120
+ },
121
+ {
122
+ "epoch": 0.5,
123
+ "learning_rate": 9.995701435280112e-06,
124
+ "loss": 2.1228,
125
+ "step": 7500
126
+ },
127
+ {
128
+ "epoch": 0.53,
129
+ "learning_rate": 9.99537072557709e-06,
130
+ "loss": 2.1163,
131
+ "step": 8000
132
+ },
133
+ {
134
+ "epoch": 0.53,
135
+ "eval_accuracy": 0.630312540865699,
136
+ "eval_loss": 1.9446306228637695,
137
+ "eval_runtime": 4.1812,
138
+ "eval_samples_per_second": 956.66,
139
+ "eval_steps_per_second": 1.674,
140
+ "step": 8000
141
+ },
142
+ {
143
+ "epoch": 0.56,
144
+ "learning_rate": 9.995040677293473e-06,
145
+ "loss": 2.1093,
146
+ "step": 8500
147
+ },
148
+ {
149
+ "epoch": 0.6,
150
+ "learning_rate": 9.994709967590451e-06,
151
+ "loss": 2.1049,
152
+ "step": 9000
153
+ },
154
+ {
155
+ "epoch": 0.63,
156
+ "learning_rate": 9.994379257887427e-06,
157
+ "loss": 2.0986,
158
+ "step": 9500
159
+ },
160
+ {
161
+ "epoch": 0.66,
162
+ "learning_rate": 9.994048548184405e-06,
163
+ "loss": 2.0949,
164
+ "step": 10000
165
+ },
166
+ {
167
+ "epoch": 0.66,
168
+ "eval_accuracy": 0.6310234052317577,
169
+ "eval_loss": 1.9502341747283936,
170
+ "eval_runtime": 4.3076,
171
+ "eval_samples_per_second": 928.598,
172
+ "eval_steps_per_second": 1.625,
173
+ "step": 10000
174
+ },
175
+ {
176
+ "epoch": 0.69,
177
+ "learning_rate": 9.993717838481381e-06,
178
+ "loss": 2.09,
179
+ "step": 10500
180
+ },
181
+ {
182
+ "epoch": 0.73,
183
+ "learning_rate": 9.993387790197764e-06,
184
+ "loss": 2.0861,
185
+ "step": 11000
186
+ },
187
+ {
188
+ "epoch": 0.76,
189
+ "learning_rate": 9.993057080494742e-06,
190
+ "loss": 2.08,
191
+ "step": 11500
192
+ },
193
+ {
194
+ "epoch": 0.79,
195
+ "learning_rate": 9.99272637079172e-06,
196
+ "loss": 2.0752,
197
+ "step": 12000
198
+ },
199
+ {
200
+ "epoch": 0.79,
201
+ "eval_accuracy": 0.6285732890538517,
202
+ "eval_loss": 1.9644430875778198,
203
+ "eval_runtime": 4.1884,
204
+ "eval_samples_per_second": 955.011,
205
+ "eval_steps_per_second": 1.671,
206
+ "step": 12000
207
+ },
208
+ {
209
+ "epoch": 0.83,
210
+ "learning_rate": 9.992395661088698e-06,
211
+ "loss": 2.0717,
212
+ "step": 12500
213
+ },
214
+ {
215
+ "epoch": 0.86,
216
+ "learning_rate": 9.992065612805081e-06,
217
+ "loss": 2.0677,
218
+ "step": 13000
219
+ },
220
+ {
221
+ "epoch": 0.89,
222
+ "learning_rate": 9.991734903102059e-06,
223
+ "loss": 2.0643,
224
+ "step": 13500
225
+ },
226
+ {
227
+ "epoch": 0.93,
228
+ "learning_rate": 9.991404193399035e-06,
229
+ "loss": 2.0609,
230
+ "step": 14000
231
+ },
232
+ {
233
+ "epoch": 0.93,
234
+ "eval_accuracy": 0.6438428177158628,
235
+ "eval_loss": 1.8919312953948975,
236
+ "eval_runtime": 5.6046,
237
+ "eval_samples_per_second": 713.696,
238
+ "eval_steps_per_second": 1.249,
239
+ "step": 14000
240
+ },
241
+ {
242
+ "epoch": 0.96,
243
+ "learning_rate": 9.991073483696012e-06,
244
+ "loss": 2.055,
245
+ "step": 14500
246
+ },
247
+ {
248
+ "epoch": 0.99,
249
+ "learning_rate": 9.99074277399299e-06,
250
+ "loss": 2.052,
251
+ "step": 15000
252
+ },
253
+ {
254
+ "epoch": 1.03,
255
+ "learning_rate": 9.990412725709374e-06,
256
+ "loss": 2.0516,
257
+ "step": 15500
258
+ },
259
+ {
260
+ "epoch": 1.06,
261
+ "learning_rate": 9.990082016006351e-06,
262
+ "loss": 2.0461,
263
+ "step": 16000
264
+ },
265
+ {
266
+ "epoch": 1.06,
267
+ "eval_accuracy": 0.6411957950065703,
268
+ "eval_loss": 1.8524950742721558,
269
+ "eval_runtime": 4.1951,
270
+ "eval_samples_per_second": 953.502,
271
+ "eval_steps_per_second": 1.669,
272
+ "step": 16000
273
+ },
274
+ {
275
+ "epoch": 1.09,
276
+ "learning_rate": 9.989751306303327e-06,
277
+ "loss": 2.0429,
278
+ "step": 16500
279
+ },
280
+ {
281
+ "epoch": 1.12,
282
+ "learning_rate": 9.989420596600305e-06,
283
+ "loss": 2.0391,
284
+ "step": 17000
285
+ },
286
+ {
287
+ "epoch": 1.16,
288
+ "learning_rate": 9.989089886897281e-06,
289
+ "loss": 2.0364,
290
+ "step": 17500
291
+ },
292
+ {
293
+ "epoch": 1.19,
294
+ "learning_rate": 9.988759177194259e-06,
295
+ "loss": 2.0347,
296
+ "step": 18000
297
+ },
298
+ {
299
+ "epoch": 1.19,
300
+ "eval_accuracy": 0.646603611349957,
301
+ "eval_loss": 1.842495322227478,
302
+ "eval_runtime": 4.1943,
303
+ "eval_samples_per_second": 953.666,
304
+ "eval_steps_per_second": 1.669,
305
+ "step": 18000
306
+ },
307
+ {
308
+ "epoch": 1.22,
309
+ "learning_rate": 9.988428467491237e-06,
310
+ "loss": 2.0311,
311
+ "step": 18500
312
+ },
313
+ {
314
+ "epoch": 1.26,
315
+ "learning_rate": 9.98809841920762e-06,
316
+ "loss": 2.0298,
317
+ "step": 19000
318
+ },
319
+ {
320
+ "epoch": 1.29,
321
+ "learning_rate": 9.987767709504598e-06,
322
+ "loss": 2.0263,
323
+ "step": 19500
324
+ },
325
+ {
326
+ "epoch": 1.32,
327
+ "learning_rate": 9.987436999801576e-06,
328
+ "loss": 2.0227,
329
+ "step": 20000
330
+ },
331
+ {
332
+ "epoch": 1.32,
333
+ "eval_accuracy": 0.6428384906645777,
334
+ "eval_loss": 1.8519748449325562,
335
+ "eval_runtime": 4.4809,
336
+ "eval_samples_per_second": 892.668,
337
+ "eval_steps_per_second": 1.562,
338
+ "step": 20000
339
+ },
340
+ {
341
+ "epoch": 1.36,
342
+ "learning_rate": 9.987106290098553e-06,
343
+ "loss": 2.0213,
344
+ "step": 20500
345
+ },
346
+ {
347
+ "epoch": 1.39,
348
+ "learning_rate": 9.98677558039553e-06,
349
+ "loss": 2.0202,
350
+ "step": 21000
351
+ },
352
+ {
353
+ "epoch": 1.42,
354
+ "learning_rate": 9.986444870692507e-06,
355
+ "loss": 2.0176,
356
+ "step": 21500
357
+ },
358
+ {
359
+ "epoch": 1.46,
360
+ "learning_rate": 9.986114160989483e-06,
361
+ "loss": 2.0156,
362
+ "step": 22000
363
+ },
364
+ {
365
+ "epoch": 1.46,
366
+ "eval_accuracy": 0.6465079984393289,
367
+ "eval_loss": 1.838840126991272,
368
+ "eval_runtime": 5.6638,
369
+ "eval_samples_per_second": 706.234,
370
+ "eval_steps_per_second": 1.236,
371
+ "step": 22000
372
+ },
373
+ {
374
+ "epoch": 1.49,
375
+ "learning_rate": 9.985783451286461e-06,
376
+ "loss": 2.0135,
377
+ "step": 22500
378
+ },
379
+ {
380
+ "epoch": 1.52,
381
+ "learning_rate": 9.985452741583439e-06,
382
+ "loss": 2.0108,
383
+ "step": 23000
384
+ },
385
+ {
386
+ "epoch": 1.55,
387
+ "learning_rate": 9.985122031880417e-06,
388
+ "loss": 2.0083,
389
+ "step": 23500
390
+ },
391
+ {
392
+ "epoch": 1.59,
393
+ "learning_rate": 9.984792645016205e-06,
394
+ "loss": 2.0061,
395
+ "step": 24000
396
+ },
397
+ {
398
+ "epoch": 1.59,
399
+ "eval_accuracy": 0.6455588887439676,
400
+ "eval_loss": 1.8356744050979614,
401
+ "eval_runtime": 4.1905,
402
+ "eval_samples_per_second": 954.546,
403
+ "eval_steps_per_second": 1.67,
404
+ "step": 24000
405
+ },
406
+ {
407
+ "epoch": 1.62,
408
+ "learning_rate": 9.984461935313183e-06,
409
+ "loss": 2.0035,
410
+ "step": 24500
411
+ },
412
+ {
413
+ "epoch": 1.65,
414
+ "learning_rate": 9.98413122561016e-06,
415
+ "loss": 2.0021,
416
+ "step": 25000
417
+ },
418
+ {
419
+ "epoch": 1.69,
420
+ "learning_rate": 9.983800515907137e-06,
421
+ "loss": 2.001,
422
+ "step": 25500
423
+ },
424
+ {
425
+ "epoch": 1.72,
426
+ "learning_rate": 9.983469806204115e-06,
427
+ "loss": 1.9985,
428
+ "step": 26000
429
+ },
430
+ {
431
+ "epoch": 1.72,
432
+ "eval_accuracy": 0.6489613463055482,
433
+ "eval_loss": 1.7983965873718262,
434
+ "eval_runtime": 4.2683,
435
+ "eval_samples_per_second": 937.141,
436
+ "eval_steps_per_second": 1.64,
437
+ "step": 26000
438
+ },
439
+ {
440
+ "epoch": 1.75,
441
+ "learning_rate": 9.983139096501093e-06,
442
+ "loss": 1.9962,
443
+ "step": 26500
444
+ },
445
+ {
446
+ "epoch": 1.79,
447
+ "learning_rate": 9.98280838679807e-06,
448
+ "loss": 1.9948,
449
+ "step": 27000
450
+ },
451
+ {
452
+ "epoch": 1.82,
453
+ "learning_rate": 9.982477677095046e-06,
454
+ "loss": 1.9947,
455
+ "step": 27500
456
+ },
457
+ {
458
+ "epoch": 1.85,
459
+ "learning_rate": 9.982146967392024e-06,
460
+ "loss": 1.9909,
461
+ "step": 28000
462
+ },
463
+ {
464
+ "epoch": 1.85,
465
+ "eval_accuracy": 0.6503962176200306,
466
+ "eval_loss": 1.7985965013504028,
467
+ "eval_runtime": 4.4068,
468
+ "eval_samples_per_second": 907.681,
469
+ "eval_steps_per_second": 1.588,
470
+ "step": 28000
471
+ },
472
+ {
473
+ "epoch": 1.88,
474
+ "learning_rate": 9.981816919108407e-06,
475
+ "loss": 1.9888,
476
+ "step": 28500
477
+ },
478
+ {
479
+ "epoch": 1.92,
480
+ "learning_rate": 9.98148687082479e-06,
481
+ "loss": 1.9877,
482
+ "step": 29000
483
+ },
484
+ {
485
+ "epoch": 1.95,
486
+ "learning_rate": 9.981156161121769e-06,
487
+ "loss": 1.9862,
488
+ "step": 29500
489
+ },
490
+ {
491
+ "epoch": 1.98,
492
+ "learning_rate": 9.980825451418745e-06,
493
+ "loss": 1.9856,
494
+ "step": 30000
495
+ },
496
+ {
497
+ "epoch": 1.98,
498
+ "eval_accuracy": 0.647755693036725,
499
+ "eval_loss": 1.8175112009048462,
500
+ "eval_runtime": 5.8254,
501
+ "eval_samples_per_second": 686.652,
502
+ "eval_steps_per_second": 1.202,
503
+ "step": 30000
504
+ },
505
+ {
506
+ "epoch": 2.02,
507
+ "learning_rate": 9.980494741715722e-06,
508
+ "loss": 1.9859,
509
+ "step": 30500
510
+ },
511
+ {
512
+ "epoch": 2.05,
513
+ "learning_rate": 9.9801640320127e-06,
514
+ "loss": 1.982,
515
+ "step": 31000
516
+ },
517
+ {
518
+ "epoch": 2.08,
519
+ "learning_rate": 9.979833322309678e-06,
520
+ "loss": 1.979,
521
+ "step": 31500
522
+ },
523
+ {
524
+ "epoch": 2.12,
525
+ "learning_rate": 9.979502612606656e-06,
526
+ "loss": 1.9805,
527
+ "step": 32000
528
+ },
529
+ {
530
+ "epoch": 2.12,
531
+ "eval_accuracy": 0.6516655780535597,
532
+ "eval_loss": 1.8021340370178223,
533
+ "eval_runtime": 4.2089,
534
+ "eval_samples_per_second": 950.37,
535
+ "eval_steps_per_second": 1.663,
536
+ "step": 32000
537
+ },
538
+ {
539
+ "epoch": 2.15,
540
+ "learning_rate": 9.979171902903632e-06,
541
+ "loss": 1.9771,
542
+ "step": 32500
543
+ },
544
+ {
545
+ "epoch": 2.18,
546
+ "learning_rate": 9.978841854620015e-06,
547
+ "loss": 1.9767,
548
+ "step": 33000
549
+ },
550
+ {
551
+ "epoch": 2.22,
552
+ "learning_rate": 9.978511144916993e-06,
553
+ "loss": 1.9738,
554
+ "step": 33500
555
+ },
556
+ {
557
+ "epoch": 2.25,
558
+ "learning_rate": 9.978181096633376e-06,
559
+ "loss": 1.9732,
560
+ "step": 34000
561
+ },
562
+ {
563
+ "epoch": 2.25,
564
+ "eval_accuracy": 0.6534042413498785,
565
+ "eval_loss": 1.7741553783416748,
566
+ "eval_runtime": 5.6645,
567
+ "eval_samples_per_second": 706.155,
568
+ "eval_steps_per_second": 1.236,
569
+ "step": 34000
570
+ },
571
+ {
572
+ "epoch": 2.28,
573
+ "learning_rate": 9.977850386930354e-06,
574
+ "loss": 1.9711,
575
+ "step": 34500
576
+ },
577
+ {
578
+ "epoch": 2.31,
579
+ "learning_rate": 9.977519677227332e-06,
580
+ "loss": 1.9697,
581
+ "step": 35000
582
+ },
583
+ {
584
+ "epoch": 2.35,
585
+ "learning_rate": 9.977188967524308e-06,
586
+ "loss": 1.969,
587
+ "step": 35500
588
+ },
589
+ {
590
+ "epoch": 2.38,
591
+ "learning_rate": 9.976858257821285e-06,
592
+ "loss": 1.968,
593
+ "step": 36000
594
+ },
595
+ {
596
+ "epoch": 2.38,
597
+ "eval_accuracy": 0.6549699887870193,
598
+ "eval_loss": 1.7642747163772583,
599
+ "eval_runtime": 5.7171,
600
+ "eval_samples_per_second": 699.657,
601
+ "eval_steps_per_second": 1.224,
602
+ "step": 36000
603
+ },
604
+ {
605
+ "epoch": 2.41,
606
+ "learning_rate": 9.976527548118262e-06,
607
+ "loss": 1.9671,
608
+ "step": 36500
609
+ },
610
+ {
611
+ "epoch": 2.45,
612
+ "learning_rate": 9.97619683841524e-06,
613
+ "loss": 1.9664,
614
+ "step": 37000
615
+ },
616
+ {
617
+ "epoch": 2.48,
618
+ "learning_rate": 9.975866128712217e-06,
619
+ "loss": 1.9634,
620
+ "step": 37500
621
+ },
622
+ {
623
+ "epoch": 2.51,
624
+ "learning_rate": 9.975535419009195e-06,
625
+ "loss": 1.9618,
626
+ "step": 38000
627
+ },
628
+ {
629
+ "epoch": 2.51,
630
+ "eval_accuracy": 0.6490381480273884,
631
+ "eval_loss": 1.8036186695098877,
632
+ "eval_runtime": 4.2126,
633
+ "eval_samples_per_second": 949.526,
634
+ "eval_steps_per_second": 1.662,
635
+ "step": 38000
636
+ },
637
+ {
638
+ "epoch": 2.55,
639
+ "learning_rate": 9.975205370725578e-06,
640
+ "loss": 1.9605,
641
+ "step": 38500
642
+ },
643
+ {
644
+ "epoch": 2.58,
645
+ "learning_rate": 9.974874661022556e-06,
646
+ "loss": 1.9593,
647
+ "step": 39000
648
+ },
649
+ {
650
+ "epoch": 2.61,
651
+ "learning_rate": 9.974543951319534e-06,
652
+ "loss": 1.9598,
653
+ "step": 39500
654
+ },
655
+ {
656
+ "epoch": 2.65,
657
+ "learning_rate": 9.97421324161651e-06,
658
+ "loss": 1.9582,
659
+ "step": 40000
660
+ },
661
+ {
662
+ "epoch": 2.65,
663
+ "eval_accuracy": 0.6469189617594611,
664
+ "eval_loss": 1.80427885055542,
665
+ "eval_runtime": 4.2765,
666
+ "eval_samples_per_second": 935.338,
667
+ "eval_steps_per_second": 1.637,
668
+ "step": 40000
669
+ },
670
+ {
671
+ "epoch": 2.68,
672
+ "learning_rate": 9.973883193332893e-06,
673
+ "loss": 1.9555,
674
+ "step": 40500
675
+ },
676
+ {
677
+ "epoch": 2.71,
678
+ "learning_rate": 9.973552483629871e-06,
679
+ "loss": 1.9546,
680
+ "step": 41000
681
+ },
682
+ {
683
+ "epoch": 2.74,
684
+ "learning_rate": 9.973221773926847e-06,
685
+ "loss": 1.9543,
686
+ "step": 41500
687
+ },
688
+ {
689
+ "epoch": 2.78,
690
+ "learning_rate": 9.972891064223825e-06,
691
+ "loss": 1.9533,
692
+ "step": 42000
693
+ },
694
+ {
695
+ "epoch": 2.78,
696
+ "eval_accuracy": 0.6523191823899371,
697
+ "eval_loss": 1.8008100986480713,
698
+ "eval_runtime": 4.4543,
699
+ "eval_samples_per_second": 898.013,
700
+ "eval_steps_per_second": 1.572,
701
+ "step": 42000
702
+ },
703
+ {
704
+ "epoch": 2.81,
705
+ "learning_rate": 9.972560354520802e-06,
706
+ "loss": 1.9528,
707
+ "step": 42500
708
+ },
709
+ {
710
+ "epoch": 2.84,
711
+ "learning_rate": 9.972230306237186e-06,
712
+ "loss": 1.9514,
713
+ "step": 43000
714
+ },
715
+ {
716
+ "epoch": 2.88,
717
+ "learning_rate": 9.971900257953569e-06,
718
+ "loss": 1.9504,
719
+ "step": 43500
720
+ },
721
+ {
722
+ "epoch": 2.91,
723
+ "learning_rate": 9.971569548250547e-06,
724
+ "loss": 1.9472,
725
+ "step": 44000
726
+ },
727
+ {
728
+ "epoch": 2.91,
729
+ "eval_accuracy": 0.6498401704848162,
730
+ "eval_loss": 1.7917245626449585,
731
+ "eval_runtime": 4.1847,
732
+ "eval_samples_per_second": 955.859,
733
+ "eval_steps_per_second": 1.673,
734
+ "step": 44000
735
+ },
736
+ {
737
+ "epoch": 2.94,
738
+ "learning_rate": 9.971238838547523e-06,
739
+ "loss": 1.9484,
740
+ "step": 44500
741
+ },
742
+ {
743
+ "epoch": 2.98,
744
+ "learning_rate": 9.9709081288445e-06,
745
+ "loss": 1.9464,
746
+ "step": 45000
747
+ },
748
+ {
749
+ "epoch": 3.01,
750
+ "learning_rate": 9.970577419141478e-06,
751
+ "loss": 1.9474,
752
+ "step": 45500
753
+ },
754
+ {
755
+ "epoch": 3.04,
756
+ "learning_rate": 9.970246709438456e-06,
757
+ "loss": 1.9443,
758
+ "step": 46000
759
+ },
760
+ {
761
+ "epoch": 3.04,
762
+ "eval_accuracy": 0.6572224802601022,
763
+ "eval_loss": 1.7718559503555298,
764
+ "eval_runtime": 4.1961,
765
+ "eval_samples_per_second": 953.273,
766
+ "eval_steps_per_second": 1.668,
767
+ "step": 46000
768
+ },
769
+ {
770
+ "epoch": 3.08,
771
+ "learning_rate": 9.969915999735434e-06,
772
+ "loss": 1.9433,
773
+ "step": 46500
774
+ },
775
+ {
776
+ "epoch": 3.11,
777
+ "learning_rate": 9.96958529003241e-06,
778
+ "loss": 1.9424,
779
+ "step": 47000
780
+ },
781
+ {
782
+ "epoch": 3.14,
783
+ "learning_rate": 9.969254580329388e-06,
784
+ "loss": 1.9409,
785
+ "step": 47500
786
+ },
787
+ {
788
+ "epoch": 3.17,
789
+ "learning_rate": 9.968923870626364e-06,
790
+ "loss": 1.9394,
791
+ "step": 48000
792
+ },
793
+ {
794
+ "epoch": 3.17,
795
+ "eval_accuracy": 0.6642529728293364,
796
+ "eval_loss": 1.721533179283142,
797
+ "eval_runtime": 4.2728,
798
+ "eval_samples_per_second": 936.147,
799
+ "eval_steps_per_second": 1.638,
800
+ "step": 48000
801
+ },
802
+ {
803
+ "epoch": 3.21,
804
+ "learning_rate": 9.968593822342749e-06,
805
+ "loss": 1.9394,
806
+ "step": 48500
807
+ },
808
+ {
809
+ "epoch": 3.24,
810
+ "learning_rate": 9.968263112639725e-06,
811
+ "loss": 1.9388,
812
+ "step": 49000
813
+ },
814
+ {
815
+ "epoch": 3.27,
816
+ "learning_rate": 9.967932402936703e-06,
817
+ "loss": 1.9376,
818
+ "step": 49500
819
+ },
820
+ {
821
+ "epoch": 3.31,
822
+ "learning_rate": 9.96760169323368e-06,
823
+ "loss": 1.9372,
824
+ "step": 50000
825
+ },
826
+ {
827
+ "epoch": 3.31,
828
+ "eval_accuracy": 0.6554556120453656,
829
+ "eval_loss": 1.7481720447540283,
830
+ "eval_runtime": 4.2868,
831
+ "eval_samples_per_second": 933.107,
832
+ "eval_steps_per_second": 1.633,
833
+ "step": 50000
834
+ },
835
+ {
836
+ "epoch": 3.34,
837
+ "learning_rate": 9.967270983530658e-06,
838
+ "loss": 1.936,
839
+ "step": 50500
840
+ },
841
+ {
842
+ "epoch": 3.37,
843
+ "learning_rate": 9.966940273827634e-06,
844
+ "loss": 1.9339,
845
+ "step": 51000
846
+ },
847
+ {
848
+ "epoch": 3.41,
849
+ "learning_rate": 9.966609564124612e-06,
850
+ "loss": 1.9333,
851
+ "step": 51500
852
+ },
853
+ {
854
+ "epoch": 3.44,
855
+ "learning_rate": 9.966279515840995e-06,
856
+ "loss": 1.9324,
857
+ "step": 52000
858
+ },
859
+ {
860
+ "epoch": 3.44,
861
+ "eval_accuracy": 0.6522052559145423,
862
+ "eval_loss": 1.7548049688339233,
863
+ "eval_runtime": 5.7821,
864
+ "eval_samples_per_second": 691.794,
865
+ "eval_steps_per_second": 1.211,
866
+ "step": 52000
867
+ },
868
+ {
869
+ "epoch": 3.47,
870
+ "learning_rate": 9.965948806137973e-06,
871
+ "loss": 1.9317,
872
+ "step": 52500
873
+ },
874
+ {
875
+ "epoch": 3.51,
876
+ "learning_rate": 9.965618096434951e-06,
877
+ "loss": 1.9323,
878
+ "step": 53000
879
+ },
880
+ {
881
+ "epoch": 3.54,
882
+ "learning_rate": 9.965287386731927e-06,
883
+ "loss": 1.9303,
884
+ "step": 53500
885
+ },
886
+ {
887
+ "epoch": 3.57,
888
+ "learning_rate": 9.964956677028905e-06,
889
+ "loss": 1.9297,
890
+ "step": 54000
891
+ },
892
+ {
893
+ "epoch": 3.57,
894
+ "eval_accuracy": 0.6580935017580414,
895
+ "eval_loss": 1.7288295030593872,
896
+ "eval_runtime": 4.2876,
897
+ "eval_samples_per_second": 932.929,
898
+ "eval_steps_per_second": 1.633,
899
+ "step": 54000
900
+ }
901
+ ],
902
+ "max_steps": 15120000,
903
+ "num_train_epochs": 1000,
904
+ "total_flos": 1.8197541890074767e+19,
905
+ "trial_name": null,
906
+ "trial_params": null
907
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9af316f54eb87b541f90db3947421d832747328366874c7f37f02f3b2264ec9b
3
+ size 3503
vocab.json ADDED
The diff for this file is too large to render. See raw diff