File size: 1,776 Bytes
2d62678 8d08d2f 7e55a0d 8d08d2f 9129738 8d08d2f 3c47826 8d08d2f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
---
license: mit
metrics:
- accuracy
tags:
- chemistry
---
# Molecular BERT Pretrained Using ChEMBL Database
This model has been pretrained based on the methodology outlined in the paper [Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration](https://spj.science.org/doi/10.34133/research.0004). While the original model was initially trained using custom code, it has been adapted for use within the Hugging Face Transformers framework in this project.
## Model Details
The model architecture utilized is based on BERT. Here are the key configuration details:
```
BertConfig(
vocab_size=70,
hidden_size=256,
num_hidden_layers=8,
num_attention_heads=8,
intermediate_size=1024,
hidden_act="gelu",
hidden_dropout_prob=0.1,
attention_probs_dropout_prob=0.1,
max_position_embeddings=max_seq_len,
type_vocab_size=1,
pad_token_id=tokenizer_pretrained.vocab["[PAD]"],
position_embedding_type="absolute"
)
```
- Optimizer: AdamW
- Learning rate: 1e-4
- Learning rate scheduler: False
- Epochs: 50
- AMP: True
- GPU: Single Nvidia RTX 3090
## Pretraining Database
The model was pretrained using data from the ChEMBL database, specifically version 33. You can download the database from [ChEMBL](https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/).
Additionally, the dataset is available on the Hugging Face Datasets Hub and can be accessed at [Hugging Face Datasets - ChEMBL_v33_pretraining](https://huggingface.co/datasets/jonghyunlee/ChEMBL_v33_pretraining/viewer/default/train).
## Performance
The accuracy score achieved by the pretrained model is 0.9672. The testing dataset used for evaluation constitutes 10% of the ChEMBL dataset.
|