Marissa commited on
Commit
75af979
1 Parent(s): 9bf199e

Add model card

Browse files



@Ezi



@Meg

Files changed (1) hide show
  1. README.md +122 -0
README.md ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ - en
5
+ - de
6
+ ---
7
+
8
+ # xlm-mlm-ende-1024
9
+
10
+ # Table of Contents
11
+
12
+ 1. [Model Details](#model-details)
13
+ 2. [Uses](#uses)
14
+ 3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
15
+ 4. [Training](#training)
16
+ 5. [Evaluation](#evaluation)
17
+ 6. [Environmental Impact](#environmental-impact)
18
+ 7. [Technical Specifications](#technical-specifications)
19
+ 8. [Citation](#citation)
20
+ 9. [Model Card Authors](#model-card-authors)
21
+ 10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
22
+
23
+
24
+ # Model Details
25
+
26
+ The XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample, Alexis Conneau. xlm-mlm-ende-1024 is a transformer pretrained using a masked language modeling (MLM) objective for English-German. This model uses language embeddings to specify the language used at inference. See the [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) for further details.
27
+
28
+ ## Model Description
29
+
30
+ - **Developed by:** Guillaume Lample, Alexis Conneau, see [associated paper](https://arxiv.org/abs/1901.07291)
31
+ - **Model type:** Language model
32
+ - **Language(s) (NLP) or Countries (images):** English-German
33
+ - **License:** Unknown
34
+ - **Related Models:** [xlm-clm-enfr-1024](https://huggingface.co/xlm-clm-enfr-1024), [xlm-clm-ende-1024](https://huggingface.co/xlm-clm-ende-1024), [xlm-mlm-enfr-1024](https://huggingface.co/xlm-mlm-enfr-1024), [xlm-mlm-enro-1024](https://huggingface.co/xlm-mlm-enro-1024)
35
+ - **Resources for more information:**
36
+ - [Associated paper](https://arxiv.org/abs/1901.07291)
37
+ - [GitHub Repo](https://github.com/facebookresearch/XLM)
38
+ - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
39
+
40
+ # Uses
41
+
42
+ ## Direct Use
43
+
44
+ The model is a language model. The model can be used for masked language modeling.
45
+
46
+ ## Downstream Use
47
+
48
+ To learn more about this task and potential downstream uses, see the Hugging Face [fill mask docs](https://huggingface.co/tasks/fill-mask) and the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs.
49
+
50
+ ## Out-of-Scope Use
51
+
52
+ The model should not be used to intentionally create hostile or alienating environments for people.
53
+
54
+ # Bias, Risks, and Limitations
55
+
56
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
57
+
58
+ ## Recommendations
59
+
60
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
61
+
62
+ # Training
63
+
64
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for details on the training data and training procedure.
65
+
66
+ The model developers also write that:
67
+
68
+ > If you use these models, you should use the same data preprocessing / BPE codes to preprocess your data.
69
+
70
+ See the associated [GitHub Repo](https://github.com/facebookresearch/XLM#ii-cross-lingual-language-model-pretraining-xlm) for further details.
71
+
72
+ # Evaluation
73
+
74
+ ## Testing Data, Factors & Metrics
75
+
76
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for details on the testing data, factors and metrics.
77
+
78
+ ## Results
79
+
80
+ For xlm-mlm-ende-1024 results, see Table 1 and Table 2 of the [associated paper](https://arxiv.org/pdf/1901.07291.pdf).
81
+
82
+ # Environmental Impact
83
+
84
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
85
+
86
+ - **Hardware Type:** More information needed
87
+ - **Hours used:** More information needed
88
+ - **Cloud Provider:** More information needed
89
+ - **Compute Region:** More information needed
90
+ - **Carbon Emitted:** More information needed
91
+
92
+ # Technical Specifications
93
+
94
+ The model developers write:
95
+
96
+ > We implement all our models in PyTorch (Paszke et al., 2017), and train them on 64 Volta GPUs for the language modeling tasks, and 8 GPUs for the MT tasks. We use float16 operations to speed up training and to reduce the memory usage of our models.
97
+
98
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for further details.
99
+
100
+ # Citation
101
+
102
+ **BibTeX:**
103
+
104
+ ```bibtex
105
+ @article{lample2019cross,
106
+ title={Cross-lingual language model pretraining},
107
+ author={Lample, Guillaume and Conneau, Alexis},
108
+ journal={arXiv preprint arXiv:1901.07291},
109
+ year={2019}
110
+ }
111
+ ```
112
+
113
+ **APA:**
114
+ - Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
115
+
116
+ # Model Card Authors
117
+
118
+ This model card was written by the team at Hugging Face.
119
+
120
+ # How to Get Started with the Model
121
+
122
+ More information needed. This model uses language embeddings to specify the language used at inference. See the [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) for further details.