improve readme
Browse files
readme.md
CHANGED
|
@@ -1,6 +1,51 @@
|
|
| 1 |
# Material SciBERT (TPU)
|
| 2 |
|
|
|
|
| 3 |
**Work in progress**
|
| 4 |
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
|
|
|
| 1 |
# Material SciBERT (TPU)
|
| 2 |
|
| 3 |
+
Goal: Improving language understanding in materials science
|
| 4 |
**Work in progress**
|
| 5 |
|
| 6 |
+
## Related work
|
| 7 |
+
|
| 8 |
+
### BERT Implementations
|
| 9 |
+
-
|
| 10 |
+
- BERT (the original) https://arxiv.org/abs/1810.04805
|
| 11 |
+
- RoBERTa (Re-implementation by Facebook) https://arxiv.org/abs/1907.11692
|
| 12 |
+
|
| 13 |
+
### Relevant models
|
| 14 |
+
|
| 15 |
+
- SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert
|
| 16 |
+
- MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K paper limited to 4 MS families http://github.com/m3rg-iitd/matscibert
|
| 17 |
+
- MaterialBERT: Not yet published
|
| 18 |
+
- MatBERT (CEDER): BERT, from scratch, 2M documents on materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT
|
| 19 |
+
- BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
### Results
|
| 23 |
+
|
| 24 |
+
Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft)
|
| 25 |
+
|
| 26 |
+
#### NER Superconductors
|
| 27 |
+
|
| 28 |
+
| Model | Precision | Recall | F1 |
|
| 29 |
+
|----------------------|-----------|---------|--------|
|
| 30 |
+
| SciBERT (baseline) | 81.62% | 84.23% | 82.90% |
|
| 31 |
+
| MatSciBERT (Gupta) | 81.45% | 84.36% | 82.88% |
|
| 32 |
+
| MatTPUSciBERT | 82.13% | 85.15% | 83.61% |
|
| 33 |
+
| MatBERT (Ceder) | 81.25% | 83.99% | 82.60% |
|
| 34 |
+
| BatteryScibert-cased | 81.09% | 84.14% | 82.59% |
|
| 35 |
+
|
| 36 |
+
#### NER Quantities
|
| 37 |
+
|
| 38 |
+
| Model | Precision | Recall | F1 |
|
| 39 |
+
|----------------------|-----------|---------|----------|
|
| 40 |
+
| SciBERT (baseline) | 88.73% | 86.76% | 87.73% |
|
| 41 |
+
| MatSciBERT (Gupta) | 84.98% | 90.12% | 87.47% |
|
| 42 |
+
| MatTPUSciBERT | 88.62% | 86.33% | 87.46% |
|
| 43 |
+
| MatBERT (Ceder) | 85.08% | 89.93% | 87.44% |
|
| 44 |
+
| BatteryScibert-cased | 85.02% | 89.30% | 87.11% |
|
| 45 |
+
| BatteryScibert-cased | 81.09% | 84.14% | 82.59% |
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
## References
|
| 49 |
+
|
| 50 |
+
This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers
|
| 51 |
|