Fix spelling mistakes
Browse files
README.md
CHANGED
@@ -19,17 +19,17 @@ ProtT5-XL-UniRef50 is based on the `t5-3b` model and was pretrained on a large c
|
|
19 |
This means it was pretrained on the raw protein sequences only, with no humans labelling them in any way (which is why it can use lots of
|
20 |
publicly available data) with an automatic process to generate inputs and labels from those protein sequences.
|
21 |
|
22 |
-
One important difference between this T5 model and the original T5 version is the
|
23 |
-
The original T5-3B model was pretrained using a span
|
24 |
The masking probability is consistent with the original T5 training by randomly masking 15% of the amino acids in the input.
|
25 |
|
26 |
This model only contains the encoder portion of the original ProtT5-XL-UniRef50 model using half precision (float16).
|
27 |
-
As such this model can efficiently be used to create protein/ amino acid representations. When used for training downstream networks/ feature extraction, these embeddings
|
28 |
|
29 |
|
30 |
## Intended uses & limitations
|
31 |
|
32 |
-
This version of the original ProtT5-XL-UniRef50 is mostly meant for conveniently creating amino-acid or protein embeddings with a low GPU-memory footprint
|
33 |
|
34 |
### How to use
|
35 |
|
|
|
19 |
This means it was pretrained on the raw protein sequences only, with no humans labelling them in any way (which is why it can use lots of
|
20 |
publicly available data) with an automatic process to generate inputs and labels from those protein sequences.
|
21 |
|
22 |
+
One important difference between this T5 model and the original T5 version is the denoising objective.
|
23 |
+
The original T5-3B model was pretrained using a span denoising objective, while this model was pretrained with a Bart-like MLM denoising objective.
|
24 |
The masking probability is consistent with the original T5 training by randomly masking 15% of the amino acids in the input.
|
25 |
|
26 |
This model only contains the encoder portion of the original ProtT5-XL-UniRef50 model using half precision (float16).
|
27 |
+
As such, this model can efficiently be used to create protein/ amino acid representations. When used for training downstream networks/ feature extraction, these embeddings produced the same performance (established empirically by comparing on several downstream tasks).
|
28 |
|
29 |
|
30 |
## Intended uses & limitations
|
31 |
|
32 |
+
This version of the original ProtT5-XL-UniRef50 is mostly meant for conveniently creating amino-acid or protein embeddings with a low GPU-memory footprint without any measurable performance-decrease in our experiments. This model is fully usable on 8 GB of video RAM.
|
33 |
|
34 |
### How to use
|
35 |
|