tnaumann commited on
Commit
9f030fc
1 Parent(s): eaa409b

Renames PubMedBERT to BiomedBERT

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -7,15 +7,17 @@ widget:
7
  - text: "[MASK] is a tumor suppressor gene."
8
  ---
9
 
10
- ## PubMedBERT (abstracts + full text)
 
 
11
 
12
  Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. [Recent work](https://arxiv.org/abs/2007.15779) shows that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models.
13
 
14
- PubMedBERT is pretrained from scratch using _abstracts_ from [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and _full-text_ articles from [PubMedCentral](https://www.ncbi.nlm.nih.gov/pmc/). This model achieves state-of-the-art performance on many biomedical NLP tasks, and currently holds the top score on the [Biomedical Language Understanding and Reasoning Benchmark](https://aka.ms/BLURB).
15
 
16
  ## Citation
17
 
18
- If you find PubMedBERT useful in your research, please cite the following paper:
19
 
20
  ```latex
21
  @misc{pubmedbert,
 
7
  - text: "[MASK] is a tumor suppressor gene."
8
  ---
9
 
10
+ ## MSR BiomedBERT (abstracts + full text)
11
+
12
+ *NOTE: This model was previously named "PubMedBERT (abstracts + full text)".*
13
 
14
  Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. [Recent work](https://arxiv.org/abs/2007.15779) shows that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models.
15
 
16
+ BiomedBERT is pretrained from scratch using _abstracts_ from [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and _full-text_ articles from [PubMedCentral](https://www.ncbi.nlm.nih.gov/pmc/). This model achieves state-of-the-art performance on many biomedical NLP tasks, and currently holds the top score on the [Biomedical Language Understanding and Reasoning Benchmark](https://aka.ms/BLURB).
17
 
18
  ## Citation
19
 
20
+ If you find BiomedBERT useful in your research, please cite the following paper:
21
 
22
  ```latex
23
  @misc{pubmedbert,