File size: 1,399 Bytes
a049c3f c20ef98 a049c3f c20ef98 45e45d6 c20ef98 4d0ba55 c20ef98 45e45d6 0b6b91e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
---
license: mit
language:
- en
library_name: peft
tags:
- ESM-2
- QLoRA
- Binding Sites
- biology
---
# ESM-2 QLoRA
These are the checkpoints for the first ever QLoRA for ESM-2! They haven't been checked for overfitting yet, so use with caution!
You can load and use them similarly to the LoRA models. This is the smallest `esm2_t6_8M_UR50D` model, so the metrics aren't great.
Scaling to larger models for better metrics is in progress. These checkpoints were trained using [the 600K dataset](https://huggingface.co/datasets/AmelieSchreiber/600K_data).
## QLoRA Info
Note, we are only training 0.58% of the parameters, using only the query, key, and value weight matrices.
```
trainable params: 23682 || all params: 4075265 || trainable%: 0.5811155838945443
```
## Testing for Overfitting
### Checkpoint 1
### Checkpoint 2
### Checkpoint 3
### Checkpoint 4
```python
Train metrics:
{'eval_loss': 0.24070295691490173,
'eval_accuracy': 0.9018779246397052,
'eval_precision': 0.16624103834249204,
'eval_recall': 0.8651772818812425,
'eval_f1': 0.27889357183237473,
'eval_auc': 0.8839390799308487,
'eval_mcc': 0.3536803490333407}
Test metrics:
{'eval_loss': 0.26776671409606934,
'eval_accuracy': 0.8902711124906878,
'eval_precision': 0.13008662855482372,
'eval_recall': 0.7084623832213568,
'eval_f1': 0.219811797752809,
'eval_auc': 0.8013943890942485,
'eval_mcc': 0.2721459410994918}
```
|