tiedeman commited on
Commit
32e7571
1 Parent(s): 857717c

Initial commit

Browse files
.gitattributes CHANGED
@@ -29,3 +29,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
29
  *.zip filter=lfs diff=lfs merge=lfs -text
30
  *.zst filter=lfs diff=lfs merge=lfs -text
31
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
29
  *.zip filter=lfs diff=lfs merge=lfs -text
30
  *.zst filter=lfs diff=lfs merge=lfs -text
31
  *tfevents* filter=lfs diff=lfs merge=lfs -text
32
+ *.spm filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,286 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - da
4
+ - de
5
+ - gmq
6
+ - is
7
+ - nb
8
+ - nn
9
+ - no
10
+ - sv
11
+
12
+ tags:
13
+ - translation
14
+ - opus-mt-tc
15
+
16
+ license: cc-by-4.0
17
+ model-index:
18
+ - name: opus-mt-tc-big-de-gmq
19
+ results:
20
+ - task:
21
+ name: Translation deu-dan
22
+ type: translation
23
+ args: deu-dan
24
+ dataset:
25
+ name: flores101-devtest
26
+ type: flores_101
27
+ args: deu dan devtest
28
+ metrics:
29
+ - name: BLEU
30
+ type: bleu
31
+ value: 35.6
32
+ - name: chr-F
33
+ type: chrf
34
+ value: 0.62363
35
+ - task:
36
+ name: Translation deu-isl
37
+ type: translation
38
+ args: deu-isl
39
+ dataset:
40
+ name: flores101-devtest
41
+ type: flores_101
42
+ args: deu isl devtest
43
+ metrics:
44
+ - name: BLEU
45
+ type: bleu
46
+ value: 20.6
47
+ - name: chr-F
48
+ type: chrf
49
+ value: 0.48691
50
+ - task:
51
+ name: Translation deu-nob
52
+ type: translation
53
+ args: deu-nob
54
+ dataset:
55
+ name: flores101-devtest
56
+ type: flores_101
57
+ args: deu nob devtest
58
+ metrics:
59
+ - name: BLEU
60
+ type: bleu
61
+ value: 25.3
62
+ - name: chr-F
63
+ type: chrf
64
+ value: 0.55765
65
+ - task:
66
+ name: Translation deu-swe
67
+ type: translation
68
+ args: deu-swe
69
+ dataset:
70
+ name: flores101-devtest
71
+ type: flores_101
72
+ args: deu swe devtest
73
+ metrics:
74
+ - name: BLEU
75
+ type: bleu
76
+ value: 34.7
77
+ - name: chr-F
78
+ type: chrf
79
+ value: 0.62323
80
+ - task:
81
+ name: Translation deu-dan
82
+ type: translation
83
+ args: deu-dan
84
+ dataset:
85
+ name: tatoeba-test-v2021-08-07
86
+ type: tatoeba_mt
87
+ args: deu-dan
88
+ metrics:
89
+ - name: BLEU
90
+ type: bleu
91
+ value: 58.7
92
+ - name: chr-F
93
+ type: chrf
94
+ value: 0.74306
95
+ - task:
96
+ name: Translation deu-isl
97
+ type: translation
98
+ args: deu-isl
99
+ dataset:
100
+ name: tatoeba-test-v2021-08-07
101
+ type: tatoeba_mt
102
+ args: deu-isl
103
+ metrics:
104
+ - name: BLEU
105
+ type: bleu
106
+ value: 47.1
107
+ - name: chr-F
108
+ type: chrf
109
+ value: 0.65180
110
+ - task:
111
+ name: Translation deu-nob
112
+ type: translation
113
+ args: deu-nob
114
+ dataset:
115
+ name: tatoeba-test-v2021-08-07
116
+ type: tatoeba_mt
117
+ args: deu-nob
118
+ metrics:
119
+ - name: BLEU
120
+ type: bleu
121
+ value: 52.5
122
+ - name: chr-F
123
+ type: chrf
124
+ value: 0.71062
125
+ - task:
126
+ name: Translation deu-swe
127
+ type: translation
128
+ args: deu-swe
129
+ dataset:
130
+ name: tatoeba-test-v2021-08-07
131
+ type: tatoeba_mt
132
+ args: deu-swe
133
+ metrics:
134
+ - name: BLEU
135
+ type: bleu
136
+ value: 58.3
137
+ - name: chr-F
138
+ type: chrf
139
+ value: 0.72658
140
+ ---
141
+ # opus-mt-tc-big-de-gmq
142
+
143
+ ## Table of Contents
144
+ - [Model Details](#model-details)
145
+ - [Uses](#uses)
146
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
147
+ - [How to Get Started With the Model](#how-to-get-started-with-the-model)
148
+ - [Training](#training)
149
+ - [Evaluation](#evaluation)
150
+ - [Citation Information](#citation-information)
151
+ - [Acknowledgements](#acknowledgements)
152
+
153
+ ## Model Details
154
+
155
+ Neural machine translation model for translating from German (de) to North Germanic languages (gmq).
156
+
157
+ This model is part of the [OPUS-MT project](https://github.com/Helsinki-NLP/Opus-MT), an effort to make neural machine translation models widely available and accessible for many languages in the world. All models are originally trained using the amazing framework of [Marian NMT](https://marian-nmt.github.io/), an efficient NMT implementation written in pure C++. The models have been converted to pyTorch using the transformers library by huggingface. Training data is taken from [OPUS](https://opus.nlpl.eu/) and training pipelines use the procedures of [OPUS-MT-train](https://github.com/Helsinki-NLP/Opus-MT-train).
158
+ **Model Description:**
159
+ - **Developed by:** Language Technology Research Group at the University of Helsinki
160
+ - **Model Type:** Translation (transformer-big)
161
+ - **Release**: 2022-07-25
162
+ - **License:** CC-BY-4.0
163
+ - **Language(s):**
164
+ - Source Language(s): deu
165
+ - Target Language(s): dan isl nno nob nor swe
166
+ - Valid Target Language Labels: >>dan<< >>isl<< >>nno<< >>nob<< >>nor<< >>swe<<
167
+ - **Original Model**: [opusTCv20210807_transformer-big_2022-07-25.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/deu-gmq/opusTCv20210807_transformer-big_2022-07-25.zip)
168
+ - **Resources for more information:**
169
+ - [OPUS-MT-train GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
170
+ - More information about released models for this language pair: [OPUS-MT deu-gmq README](https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/deu-gmq/README.md)
171
+ - [More information about MarianNMT models in the transformers library](https://huggingface.co/docs/transformers/model_doc/marian)
172
+ - [Tatoeba Translation Challenge](https://github.com/Helsinki-NLP/Tatoeba-Challenge/
173
+
174
+ This is a multilingual translation model with multiple target languages. A sentence initial language token is required in the form of `>>id<<` (id = valid target language ID), e.g. `>>dan<<`
175
+
176
+ ## Uses
177
+
178
+ This model can be used for translation and text-to-text generation.
179
+
180
+ ## Risks, Limitations and Biases
181
+
182
+ **CONTENT WARNING: Readers should be aware that the model is trained on various public data sets that may contain content that is disturbing, offensive, and can propagate historical and current stereotypes.**
183
+
184
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
185
+
186
+ ## How to Get Started With the Model
187
+
188
+ A short example code:
189
+
190
+ ```python
191
+ from transformers import MarianMTModel, MarianTokenizer
192
+
193
+ src_text = [
194
+ ">>dan<< Ich hätte fast meinen Pass vergessen.",
195
+ ">>dan<< Dieses Fenster hier ist schusssicher."
196
+ ]
197
+
198
+ model_name = "pytorch-models/opus-mt-tc-big-de-gmq"
199
+ tokenizer = MarianTokenizer.from_pretrained(model_name)
200
+ model = MarianMTModel.from_pretrained(model_name)
201
+ translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
202
+
203
+ for t in translated:
204
+ print( tokenizer.decode(t, skip_special_tokens=True) )
205
+
206
+ # expected output:
207
+ # Jeg havde næsten glemt mit pas.
208
+ # Dette vindue er skudsikkert.
209
+ ```
210
+
211
+ You can also use OPUS-MT models with the transformers pipelines, for example:
212
+
213
+ ```python
214
+ from transformers import pipeline
215
+ pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-big-de-gmq")
216
+ print(pipe(">>dan<< Ich hätte fast meinen Pass vergessen."))
217
+
218
+ # expected output: Jeg havde næsten glemt mit pas.
219
+ ```
220
+
221
+ ## Training
222
+
223
+ - **Data**: opusTCv20210807 ([source](https://github.com/Helsinki-NLP/Tatoeba-Challenge))
224
+ - **Pre-processing**: SentencePiece (spm32k,spm32k)
225
+ - **Model Type:** transformer-big
226
+ - **Original MarianNMT Model**: [opusTCv20210807_transformer-big_2022-07-25.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/deu-gmq/opusTCv20210807_transformer-big_2022-07-25.zip)
227
+ - **Training Scripts**: [GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
228
+
229
+ ## Evaluation
230
+
231
+ * test set translations: [opusTCv20210807_transformer-big_2022-07-25.test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/deu-gmq/opusTCv20210807_transformer-big_2022-07-25.test.txt)
232
+ * test set scores: [opusTCv20210807_transformer-big_2022-07-25.eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/deu-gmq/opusTCv20210807_transformer-big_2022-07-25.eval.txt)
233
+ * benchmark results: [benchmark_results.txt](benchmark_results.txt)
234
+ * benchmark output: [benchmark_translations.zip](benchmark_translations.zip)
235
+
236
+ | langpair | testset | chr-F | BLEU | #sent | #words |
237
+ |----------|---------|-------|-------|-------|--------|
238
+ | deu-dan | tatoeba-test-v2021-08-07 | 0.74306 | 58.7 | 9998 | 74644 |
239
+ | deu-isl | tatoeba-test-v2021-08-07 | 0.65180 | 47.1 | 969 | 5951 |
240
+ | deu-nob | tatoeba-test-v2021-08-07 | 0.71062 | 52.5 | 3525 | 31978 |
241
+ | deu-swe | tatoeba-test-v2021-08-07 | 0.72658 | 58.3 | 3410 | 22701 |
242
+ | deu-dan | flores101-devtest | 0.62363 | 35.6 | 1012 | 24638 |
243
+ | deu-isl | flores101-devtest | 0.48691 | 20.6 | 1012 | 22834 |
244
+ | deu-nob | flores101-devtest | 0.55765 | 25.3 | 1012 | 23873 |
245
+ | deu-swe | flores101-devtest | 0.62323 | 34.7 | 1012 | 23121 |
246
+
247
+ ## Citation Information
248
+
249
+ * Publications: [OPUS-MT – Building open translation services for the World](https://aclanthology.org/2020.eamt-1.61/) and [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://aclanthology.org/2020.wmt-1.139/) (Please, cite if you use this model.)
250
+
251
+ ```
252
+ @inproceedings{tiedemann-thottingal-2020-opus,
253
+ title = "{OPUS}-{MT} {--} Building open translation services for the World",
254
+ author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh},
255
+ booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
256
+ month = nov,
257
+ year = "2020",
258
+ address = "Lisboa, Portugal",
259
+ publisher = "European Association for Machine Translation",
260
+ url = "https://aclanthology.org/2020.eamt-1.61",
261
+ pages = "479--480",
262
+ }
263
+
264
+ @inproceedings{tiedemann-2020-tatoeba,
265
+ title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
266
+ author = {Tiedemann, J{\"o}rg},
267
+ booktitle = "Proceedings of the Fifth Conference on Machine Translation",
268
+ month = nov,
269
+ year = "2020",
270
+ address = "Online",
271
+ publisher = "Association for Computational Linguistics",
272
+ url = "https://aclanthology.org/2020.wmt-1.139",
273
+ pages = "1174--1182",
274
+ }
275
+ ```
276
+
277
+ ## Acknowledgements
278
+
279
+ The work is supported by the [European Language Grid](https://www.european-language-grid.eu/) as [pilot project 2866](https://live.european-language-grid.eu/catalogue/#/resource/projects/2866), by the [FoTran project](https://www.helsinki.fi/en/researchgroups/natural-language-understanding-with-cross-lingual-grounding), funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 771113), and the [MeMAD project](https://memad.eu/), funded by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement No 780069. We are also grateful for the generous computational resources and IT infrastructure provided by [CSC -- IT Center for Science](https://www.csc.fi/), Finland.
280
+
281
+ ## Model conversion info
282
+
283
+ * transformers version: 4.16.2
284
+ * OPUS-MT git hash: 8b9f0b0
285
+ * port time: Fri Aug 12 18:54:58 EEST 2022
286
+ * port machine: LM0-400-22516.local
benchmark_results.txt ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ deu-dan flores101-dev 0.61917 36.1 997 23685
2
+ deu-isl flores101-dev 0.48506 20.1 997 21857
3
+ deu-nob flores101-dev 0.55184 25.5 997 23157
4
+ deu-swe flores101-dev 0.62102 34.8 997 22417
5
+ deu-dan flores101-devtest 0.62363 35.6 1012 24638
6
+ deu-isl flores101-devtest 0.48691 20.6 1012 22834
7
+ deu-nob flores101-devtest 0.55765 25.3 1012 23873
8
+ deu-swe flores101-devtest 0.62323 34.7 1012 23121
9
+ deu-dan tatoeba-test-v2020-07-28 0.74305 58.7 10000 74655
10
+ deu-isl tatoeba-test-v2020-07-28 0.65276 47.2 968 5939
11
+ deu-dan tatoeba-test-v2021-03-30 0.74305 58.7 10000 74655
12
+ deu-isl tatoeba-test-v2021-03-30 0.65393 47.3 990 6127
13
+ deu-dan tatoeba-test-v2021-08-07 0.74306 58.7 9998 74644
14
+ deu-isl tatoeba-test-v2021-08-07 0.65180 47.1 969 5951
15
+ deu-nob tatoeba-test-v2021-08-07 0.71062 52.5 3525 31978
16
+ deu-swe tatoeba-test-v2021-08-07 0.72658 58.3 3410 22701
benchmark_translations.zip ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8cd1261fb584ea7713d5dea17ddaf3fb5be08dd0f7762e278871d73da923c81e
3
+ size 2720078
config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "activation_dropout": 0.0,
3
+ "activation_function": "relu",
4
+ "architectures": [
5
+ "MarianMTModel"
6
+ ],
7
+ "attention_dropout": 0.0,
8
+ "bad_words_ids": [
9
+ [
10
+ 56727
11
+ ]
12
+ ],
13
+ "bos_token_id": 0,
14
+ "classifier_dropout": 0.0,
15
+ "d_model": 1024,
16
+ "decoder_attention_heads": 16,
17
+ "decoder_ffn_dim": 4096,
18
+ "decoder_layerdrop": 0.0,
19
+ "decoder_layers": 6,
20
+ "decoder_start_token_id": 56727,
21
+ "decoder_vocab_size": 56728,
22
+ "dropout": 0.1,
23
+ "encoder_attention_heads": 16,
24
+ "encoder_ffn_dim": 4096,
25
+ "encoder_layerdrop": 0.0,
26
+ "encoder_layers": 6,
27
+ "eos_token_id": 41373,
28
+ "forced_eos_token_id": 41373,
29
+ "init_std": 0.02,
30
+ "is_encoder_decoder": true,
31
+ "max_length": 512,
32
+ "max_position_embeddings": 1024,
33
+ "model_type": "marian",
34
+ "normalize_embedding": false,
35
+ "num_beams": 4,
36
+ "num_hidden_layers": 6,
37
+ "pad_token_id": 56727,
38
+ "scale_embedding": true,
39
+ "share_encoder_decoder_embeddings": true,
40
+ "static_position_embeddings": true,
41
+ "torch_dtype": "float16",
42
+ "transformers_version": "4.18.0.dev0",
43
+ "use_cache": true,
44
+ "vocab_size": 56728
45
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db3fcc4f08b00f84532ba66c66511b3be3649a81f34af2516079e8ea7e0dcbe4
3
+ size 585277763
source.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06a1b553146ea2267fa9c35c807334729e5da52891fed3f4586543db9dd18c37
3
+ size 823934
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}
target.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:abe1d5af0e56a45bcd1310b56ea9d5c571caab24e2d319556b3f10e414824fdd
3
+ size 805921
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"source_lang": "de", "target_lang": "gmq", "unk_token": "<unk>", "eos_token": "</s>", "pad_token": "<pad>", "model_max_length": 512, "sp_model_kwargs": {}, "separate_vocabs": false, "special_tokens_map_file": null, "name_or_path": "marian-models/opusTCv20210807_transformer-big_2022-07-25/de-gmq", "tokenizer_class": "MarianTokenizer"}
vocab.json ADDED
The diff for this file is too large to render. See raw diff