yarongef commited on
Commit
7c034ff
·
1 Parent(s): a8f0280

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -5
README.md CHANGED
@@ -9,7 +9,7 @@ datasets:
9
 
10
  # DistilProtBert
11
 
12
- Distilled version of [ProtBert](https://huggingface.co/Rostlab/prot_bert) model.
13
  In addition to cross entropy and cosine teacher-student losses, DistilProtBert was pretrained on a masked language modeling (MLM) objective and it only works with capital letter amino acids.
14
 
15
  # Model description
@@ -17,9 +17,9 @@ In addition to cross entropy and cosine teacher-student losses, DistilProtBert w
17
  DistilProtBert was pretrained on millions of proteins sequences.
18
 
19
  Few important differences between DistilProtBert model and the original ProtBert version are:
20
- 1. Size of the model
21
- 2. Size of the pretraining dataset
22
- 3. Hardware used for pretraining
23
 
24
  ## Intended uses & limitations
25
 
@@ -51,6 +51,5 @@ When fine-tuned on downstream tasks, this model achieves the following results:
51
  | CB513 | 79 | |
52
  | DeepLoc | | 86 |
53
 
54
- Distinguish between:
55
 
56
  ### BibTeX entry and citation info
 
9
 
10
  # DistilProtBert
11
 
12
+ Distilled version of [ProtBert-UniRef100](https://huggingface.co/Rostlab/prot_bert) model.
13
  In addition to cross entropy and cosine teacher-student losses, DistilProtBert was pretrained on a masked language modeling (MLM) objective and it only works with capital letter amino acids.
14
 
15
  # Model description
 
17
  DistilProtBert was pretrained on millions of proteins sequences.
18
 
19
  Few important differences between DistilProtBert model and the original ProtBert version are:
20
+ 1. Size of the model: 230M parameters (ProtBert has 420M parameters)
21
+ 2. Size of the pretraining dataset: ~43M proteins (ProtBert was pretrained on 216M proteins)
22
+ 3. Hardware used for pretraining: five v100 32GB Nvidia GPUs (ProtBert was pretrained on 512 16GB TPUs)
23
 
24
  ## Intended uses & limitations
25
 
 
51
  | CB513 | 79 | |
52
  | DeepLoc | | 86 |
53
 
 
54
 
55
  ### BibTeX entry and citation info