sgugger Marissa commited on
Commit
bfe8606
1 Parent(s): 4137612

Initial model card (#1)

Browse files

- Initial model card (8d35a8ff705200c04d8eeb7c408e48f518a3cdfd)
- Update README.md (ee36cc9eea0cb26f65eb7cf8d35a2c9714f74544)
- Update README.md (88345f72fc3e3e23fb2404467ff59bdd7992e897)


Co-authored-by: Marissa Gerchick <Marissa@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +139 -0
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ - en
5
+ - de
6
+ ---
7
+
8
+ # xlm-clm-ende-1024
9
+
10
+ # Table of Contents
11
+
12
+ 1. [Model Details](#model-details)
13
+ 2. [Uses](#uses)
14
+ 3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
15
+ 4. [Training](#training)
16
+ 5. [Evaluation](#evaluation)
17
+ 6. [Environmental Impact](#environmental-impact)
18
+ 7. [Technical Specifications](#technical-specifications)
19
+ 8. [Citation](#citation)
20
+ 9. [Model Card Authors](#model-card-authors)
21
+ 10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
22
+
23
+
24
+ # Model Details
25
+
26
+ The XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample, Alexis Conneau. xlm-clm-ende-1024 is a transformer pretrained using a causal language modeling (CLM) objective (next token prediction) for English-German.
27
+
28
+ ## Model Description
29
+
30
+ - **Developed by:** Guillaume Lample, Alexis Conneau, see [associated paper](https://arxiv.org/abs/1901.07291)
31
+ - **Model type:** Language model
32
+ - **Language(s) (NLP):** English-German
33
+ - **License:** Unknown
34
+ - **Related Models:** [xlm-clm-enfr-1024](https://huggingface.co/xlm-clm-enfr-1024), [xlm-mlm-ende-1024](https://huggingface.co/xlm-mlm-ende-1024), [xlm-mlm-enfr-1024](https://huggingface.co/xlm-mlm-enfr-1024), [xlm-mlm-enro-1024](https://huggingface.co/xlm-mlm-enro-1024)
35
+ - **Resources for more information:**
36
+ - [Associated paper](https://arxiv.org/abs/1901.07291)
37
+ - [GitHub Repo](https://github.com/facebookresearch/XLM)
38
+ - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
39
+
40
+ # Uses
41
+
42
+ ## Direct Use
43
+
44
+ The model is a language model. The model can be used for causal language modeling.
45
+
46
+ ## Downstream Use
47
+
48
+ To learn more about this task and potential downstream uses, see the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs.
49
+
50
+ ## Out-of-Scope Use
51
+
52
+ The model should not be used to intentionally create hostile or alienating environments for people.
53
+
54
+ # Bias, Risks, and Limitations
55
+
56
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
57
+
58
+ ## Recommendations
59
+
60
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
61
+
62
+ # Training
63
+
64
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for details on the training data and training procedure.
65
+
66
+ # Evaluation
67
+
68
+ ## Testing Data, Factors & Metrics
69
+
70
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for details on the testing data, factors and metrics.
71
+
72
+ ## Results
73
+
74
+ For xlm-clm-ende-1024 results, see Table 2 of the [associated paper](https://arxiv.org/pdf/1901.07291.pdf).
75
+
76
+ # Environmental Impact
77
+
78
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
79
+
80
+ - **Hardware Type:** More information needed
81
+ - **Hours used:** More information needed
82
+ - **Cloud Provider:** More information needed
83
+ - **Compute Region:** More information needed
84
+ - **Carbon Emitted:** More information needed
85
+
86
+ # Technical Specifications
87
+
88
+ The model developers write:
89
+
90
+ > We implement all our models in PyTorch (Paszke et al., 2017), and train them on 64 Volta GPUs for the language modeling tasks, and 8 GPUs for the MT tasks. We use float16 operations to speed up training and to reduce the memory usage of our models.
91
+
92
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for further details.
93
+
94
+ # Citation
95
+
96
+ **BibTeX:**
97
+
98
+ ```bibtex
99
+ @article{lample2019cross,
100
+ title={Cross-lingual language model pretraining},
101
+ author={Lample, Guillaume and Conneau, Alexis},
102
+ journal={arXiv preprint arXiv:1901.07291},
103
+ year={2019}
104
+ }
105
+ ```
106
+
107
+ **APA:**
108
+ - Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
109
+
110
+ # Model Card Authors
111
+
112
+ This model card was written by the team at Hugging Face.
113
+
114
+ # How to Get Started with the Model
115
+
116
+ Use the code below to get started with the model.
117
+
118
+ <details>
119
+ <summary> Click to expand </summary>
120
+
121
+ ```python
122
+ import torch
123
+ from transformers import XLMTokenizer, XLMWithLMHeadModel
124
+
125
+ tokenizer = XLMTokenizer.from_pretrained("xlm-clm-ende-1024")
126
+ model = XLMWithLMHeadModel.from_pretrained("xlm-clm-ende-1024")
127
+
128
+ input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")]) # batch size of 1
129
+
130
+ language_id = tokenizer.lang2id["en"] # 0
131
+ langs = torch.tensor([language_id] * input_ids.shape[1]) # torch.tensor([0, 0, 0, ..., 0])
132
+
133
+ # We reshape it to be of size (batch_size, sequence_length)
134
+ langs = langs.view(1, -1) # is now of shape [1, sequence_length] (we have a batch size of 1)
135
+
136
+ outputs = model(input_ids, langs=langs)
137
+ ```
138
+
139
+ </details>