gchhablani
commited on
Commit
•
66503d3
1
Parent(s):
b25e52a
Update README.md
Browse files
README.md
CHANGED
@@ -17,7 +17,7 @@ Disclaimer: This model card has been written by [gchhablani](https://huggingface
|
|
17 |
|
18 |
## Model description
|
19 |
|
20 |
-
FNet is a transformers model with attention replaced with fourier transforms. It is pretrained on a large corpus of
|
21 |
English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling
|
22 |
them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and
|
23 |
labels from those texts. More precisely, it was pretrained with two objectives:
|
@@ -46,9 +46,10 @@ to make decisions, such as sequence classification, token classification or ques
|
|
46 |
generation you should look at model like GPT2.
|
47 |
|
48 |
### How to use
|
49 |
-
|
50 |
You can use this model directly with a pipeline for masked language modeling:
|
51 |
|
|
|
|
|
52 |
```python
|
53 |
>>> from transformers import FNetForMaskedLM, FNetTokenizer, pipeline
|
54 |
>>> tokenizer = FNetTokenizer.from_pretrained("google/fnet-base")
|
@@ -68,12 +69,14 @@ You can use this model directly with a pipeline for masked language modeling:
|
|
68 |
|
69 |
Here is how to use this model to get the features of a given text in PyTorch:
|
70 |
|
|
|
|
|
71 |
```python
|
72 |
from transformers import FNetTokenizer, FNetModel
|
73 |
tokenizer = FNetTokenizer.from_pretrained("google/fnet-base")
|
74 |
model = FNetModel.from_pretrained("google/fnet-base")
|
75 |
text = "Replace me by any text you'd like."
|
76 |
-
encoded_input = tokenizer(text, return_tensors='pt')
|
77 |
output = model(**encoded_input)
|
78 |
```
|
79 |
|
@@ -172,4 +175,7 @@ Glue test results:
|
|
172 |
biburl = {https://dblp.org/rec/journals/corr/abs-2105-03824.bib},
|
173 |
bibsource = {dblp computer science bibliography, https://dblp.org}
|
174 |
}
|
175 |
-
```
|
|
|
|
|
|
|
|
17 |
|
18 |
## Model description
|
19 |
|
20 |
+
FNet is a transformers model with attention replaced with fourier transforms. Hence, the inputs do not contain an `attention_mask`. It is pretrained on a large corpus of
|
21 |
English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling
|
22 |
them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and
|
23 |
labels from those texts. More precisely, it was pretrained with two objectives:
|
|
|
46 |
generation you should look at model like GPT2.
|
47 |
|
48 |
### How to use
|
|
|
49 |
You can use this model directly with a pipeline for masked language modeling:
|
50 |
|
51 |
+
**Note: The mask filling pipeline doesn't work exactly as the original model performs masking after converting to tokens. In masking pipeline an additional space is added after the [MASK].**
|
52 |
+
|
53 |
```python
|
54 |
>>> from transformers import FNetForMaskedLM, FNetTokenizer, pipeline
|
55 |
>>> tokenizer = FNetTokenizer.from_pretrained("google/fnet-base")
|
|
|
69 |
|
70 |
Here is how to use this model to get the features of a given text in PyTorch:
|
71 |
|
72 |
+
**Note: You must specify the maximum sequence length to be 512 and truncate/pad to the same length because the original model has no attention mask and considers all the hidden states during forward pass.**
|
73 |
+
|
74 |
```python
|
75 |
from transformers import FNetTokenizer, FNetModel
|
76 |
tokenizer = FNetTokenizer.from_pretrained("google/fnet-base")
|
77 |
model = FNetModel.from_pretrained("google/fnet-base")
|
78 |
text = "Replace me by any text you'd like."
|
79 |
+
encoded_input = tokenizer(text, return_tensors='pt', padding='max_length', truncation=True, max_length=512)
|
80 |
output = model(**encoded_input)
|
81 |
```
|
82 |
|
|
|
175 |
biburl = {https://dblp.org/rec/journals/corr/abs-2105-03824.bib},
|
176 |
bibsource = {dblp computer science bibliography, https://dblp.org}
|
177 |
}
|
178 |
+
```
|
179 |
+
|
180 |
+
## Contributions
|
181 |
+
Thanks to @gchhablani for adding this model.
|