jramompichel
commited on
Commit
•
9b024b5
1
Parent(s):
42647dd
Update README.md
Browse files
README.md
CHANGED
@@ -5,6 +5,7 @@ license: mit
|
|
5 |
**Descrición do Modelo / Model description**
|
6 |
|
7 |
Modelo feito con OpenNMT para o par español-galego utilizando unha arquitectura transformer.
|
|
|
8 |
Model developed with OpenNMT for the Spanish-Galician pair using a transformer architecture.
|
9 |
|
10 |
**Como utilizar**
|
@@ -22,44 +23,44 @@ onmt_translate -src input_text -model NOS-MT-es-gl -output ./output_file.txt -r
|
|
22 |
**Adestramento / Training**
|
23 |
|
24 |
No adestramento, utilizamos corpora auténticos e sintéticos. Os primeiros son corpora de traducións feitas directamente por tradutores humanos. Os segundos son corpora de traducións español-portugués e inglés-portugués, que convertemos en español-galego e inglés-galego a través da tradución automática portugués-galego con Opentrad/Apertium e transliteración para palabras fóra de vocabulário.
|
|
|
25 |
In the training we have used authentic and synthetic corpora. The former are corpora of translations directly produced by human translators. The latter are corpora of Spanish-Portuguese and English-Portuguese translations, which we have converted into Spanish-Galician and English-Galician by means of Portuguese-Galician translation with Opentrad/Apertium and transliteration for out-of-vocabulary words.
|
26 |
|
27 |
|
28 |
-
**Procedemento de adestramento**
|
29 |
|
30 |
-
+ Tokenization dos datasets feita co tokenizador de linguakit https://github.com/citiususc/Linguakit
|
31 |
|
32 |
-
+ O vocabulario para os modelos foi xerado a través do script [learn_bpe.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/tools/learn_bpe.py) da open NMT
|
33 |
|
34 |
-
+ Usando o .yaml neste repositorio pode replicar o proceso de adestramento do seguinte xeito
|
35 |
|
36 |
```bash
|
37 |
onmt_build_vocab -config bpe-es-gl_emb.yaml -n_sample 100000
|
38 |
onmt_train -config bpe-es-gl_emb.yaml
|
39 |
```
|
40 |
|
41 |
-
**Hiperparámetros**
|
42 |
|
43 |
Os parámetros usados para o desenvolvimento do modelo poden ser consultados directamente no mesmo ficheiro .yaml bpe-es-gl_emb.yaml
|
44 |
|
|
|
|
|
|
|
45 |
|
46 |
-
|
47 |
-
A avalación dos modelos é feita cunha mistura de tests desenvolvidos internamente
|
48 |
-
(gold1, gold2, test-suite) con outros datasets disponíbeis en galego (Flores).
|
49 |
|
50 |
| GOLD 1 | GOLD 2 | FLORES | TEST-SUITE|
|
51 |
| ------------- |:-------------:| -------:|----------:|
|
52 |
| 79.6 | 43.3 | 21.8 | 74.3 |
|
53 |
|
54 |
-
|
55 |
-
|
56 |
**Información adicional**
|
57 |
|
58 |
Licensing information
|
59 |
|
60 |
-
|
61 |
|
62 |
-
**Financiamento**
|
63 |
|
64 |
This research was funded by the project "Nós: Galician in the society and economy of artificial intelligence", agreement between Xunta de Galicia and University of Santiago de Compostela, and grant ED431G2019/04 by the Galician Ministry of Education, University and Professional Training, and the European Regional Development Fund (ERDF/FEDER program), and Groups of Reference: ED431C 2020/21.
|
65 |
|
|
|
5 |
**Descrición do Modelo / Model description**
|
6 |
|
7 |
Modelo feito con OpenNMT para o par español-galego utilizando unha arquitectura transformer.
|
8 |
+
|
9 |
Model developed with OpenNMT for the Spanish-Galician pair using a transformer architecture.
|
10 |
|
11 |
**Como utilizar**
|
|
|
23 |
**Adestramento / Training**
|
24 |
|
25 |
No adestramento, utilizamos corpora auténticos e sintéticos. Os primeiros son corpora de traducións feitas directamente por tradutores humanos. Os segundos son corpora de traducións español-portugués e inglés-portugués, que convertemos en español-galego e inglés-galego a través da tradución automática portugués-galego con Opentrad/Apertium e transliteración para palabras fóra de vocabulário.
|
26 |
+
|
27 |
In the training we have used authentic and synthetic corpora. The former are corpora of translations directly produced by human translators. The latter are corpora of Spanish-Portuguese and English-Portuguese translations, which we have converted into Spanish-Galician and English-Galician by means of Portuguese-Galician translation with Opentrad/Apertium and transliteration for out-of-vocabulary words.
|
28 |
|
29 |
|
30 |
+
**Procedemento de adestramento / Training process**
|
31 |
|
32 |
+
+ Tokenization dos datasets feita co tokenizador de linguakit / Tokenization of the datasets made with linguakit tokeniser https://github.com/citiususc/Linguakit
|
33 |
|
34 |
+
+ O vocabulario para os modelos foi xerado a través do script / Vocabulary for the models was created by the script [learn_bpe.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/tools/learn_bpe.py) da open NMT
|
35 |
|
36 |
+
+ Usando o .yaml neste repositorio pode replicar o proceso de adestramento do seguinte xeito / Using the .yaml in this repository you can replicate the training process as follows
|
37 |
|
38 |
```bash
|
39 |
onmt_build_vocab -config bpe-es-gl_emb.yaml -n_sample 100000
|
40 |
onmt_train -config bpe-es-gl_emb.yaml
|
41 |
```
|
42 |
|
43 |
+
**Hiperparámetros / Hyper-parameters**
|
44 |
|
45 |
Os parámetros usados para o desenvolvimento do modelo poden ser consultados directamente no mesmo ficheiro .yaml bpe-es-gl_emb.yaml
|
46 |
|
47 |
+
The parameters used for the development of the model can be directly viewed in the same .yaml file bpe-es-gl_emb.yaml
|
48 |
+
|
49 |
+
**Avaliación / Evaluation**
|
50 |
|
51 |
+
A avalación dos modelos é feita cunha mistura de tests desenvolvidos internamente (gold1, gold2, test-suite) con outros datasets disponíbeis en galego (Flores).
|
|
|
|
|
52 |
|
53 |
| GOLD 1 | GOLD 2 | FLORES | TEST-SUITE|
|
54 |
| ------------- |:-------------:| -------:|----------:|
|
55 |
| 79.6 | 43.3 | 21.8 | 74.3 |
|
56 |
|
|
|
|
|
57 |
**Información adicional**
|
58 |
|
59 |
Licensing information
|
60 |
|
61 |
+
MIT
|
62 |
|
63 |
+
**Financiamento / Funding**
|
64 |
|
65 |
This research was funded by the project "Nós: Galician in the society and economy of artificial intelligence", agreement between Xunta de Galicia and University of Santiago de Compostela, and grant ED431G2019/04 by the Galician Ministry of Education, University and Professional Training, and the European Regional Development Fund (ERDF/FEDER program), and Groups of Reference: ED431C 2020/21.
|
66 |
|