End of training

README.md

@@ -0,0 +1,89 @@
+---
+base_model: AIRI-Institute/gena-lm-bert-base-t2t
+tags:
+- generated_from_trainer
+metrics:
+- precision
+- recall
+- accuracy
+model-index:
+- name: gena-lm-bert-base-t2t_ft_BioS74_1kbpHG19_DHSs_H3K27AC
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# gena-lm-bert-base-t2t_ft_BioS74_1kbpHG19_DHSs_H3K27AC
+
+This model is a fine-tuned version of [AIRI-Institute/gena-lm-bert-base-t2t](https://huggingface.co/AIRI-Institute/gena-lm-bert-base-t2t) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.4989
+- F1 Score: 0.8378
+- Precision: 0.8054
+- Recall: 0.8729
+- Accuracy: 0.8230
+- Auc: 0.8768
+- Prc: 0.8333
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 20
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss | F1 Score | Precision | Recall | Accuracy | Auc    | Prc    |
+|:-------------:|:------:|:-----:|:---------------:|:--------:|:---------:|:------:|:--------:|:------:|:------:|
+| 0.6857        | 0.1314 | 500   | 0.6358          | 0.7270   | 0.7424    | 0.7122 | 0.7200   | 0.7586 | 0.7471 |
+| 0.6115        | 0.2629 | 1000  | 0.5765          | 0.7146   | 0.7998    | 0.6459 | 0.7300   | 0.7812 | 0.7800 |
+| 0.5477        | 0.3943 | 1500  | 0.4954          | 0.8091   | 0.7478    | 0.8815 | 0.7823   | 0.8401 | 0.8126 |
+| 0.487         | 0.5258 | 2000  | 0.4893          | 0.8185   | 0.7659    | 0.8790 | 0.7960   | 0.8423 | 0.7918 |
+| 0.4824        | 0.6572 | 2500  | 0.5362          | 0.8176   | 0.7241    | 0.9387 | 0.7807   | 0.8192 | 0.7493 |
+| 0.5023        | 0.7886 | 3000  | 0.4826          | 0.8279   | 0.7491    | 0.9252 | 0.7986   | 0.8656 | 0.8357 |
+| 0.4762        | 0.9201 | 3500  | 0.4686          | 0.8229   | 0.7869    | 0.8624 | 0.8057   | 0.8765 | 0.8499 |
+| 0.4796        | 1.0515 | 4000  | 0.4538          | 0.8201   | 0.7977    | 0.8438 | 0.8062   | 0.8835 | 0.8673 |
+| 0.4654        | 1.1830 | 4500  | 0.4534          | 0.8285   | 0.7531    | 0.9206 | 0.8004   | 0.8641 | 0.8296 |
+| 0.4583        | 1.3144 | 5000  | 0.4943          | 0.8285   | 0.7376    | 0.9448 | 0.7952   | 0.8512 | 0.7967 |
+| 0.4605        | 1.4458 | 5500  | 0.4470          | 0.8308   | 0.8040    | 0.8594 | 0.8167   | 0.8903 | 0.8718 |
+| 0.4316        | 1.5773 | 6000  | 0.4880          | 0.8338   | 0.7886    | 0.8845 | 0.8154   | 0.8626 | 0.8079 |
+| 0.4447        | 1.7087 | 6500  | 0.4371          | 0.8369   | 0.7814    | 0.9011 | 0.8162   | 0.8866 | 0.8626 |
+| 0.4355        | 1.8402 | 7000  | 0.4654          | 0.8273   | 0.8032    | 0.8528 | 0.8136   | 0.8713 | 0.8183 |
+| 0.4447        | 1.9716 | 7500  | 0.4687          | 0.8327   | 0.7729    | 0.9026 | 0.8101   | 0.8877 | 0.8643 |
+| 0.4238        | 2.1030 | 8000  | 0.4868          | 0.8350   | 0.7753    | 0.9046 | 0.8128   | 0.8889 | 0.8651 |
+| 0.439         | 2.2345 | 8500  | 0.4307          | 0.8329   | 0.8271    | 0.8388 | 0.8238   | 0.8997 | 0.8889 |
+| 0.4248        | 2.3659 | 9000  | 0.4533          | 0.8408   | 0.7991    | 0.8870 | 0.8241   | 0.8902 | 0.8637 |
+| 0.4175        | 2.4974 | 9500  | 0.5197          | 0.8376   | 0.7963    | 0.8835 | 0.8207   | 0.8802 | 0.8442 |
+| 0.4426        | 2.6288 | 10000 | 0.4643          | 0.8246   | 0.8311    | 0.8182 | 0.8178   | 0.8974 | 0.8796 |
+| 0.4538        | 2.7603 | 10500 | 0.5072          | 0.8345   | 0.8211    | 0.8483 | 0.8238   | 0.8717 | 0.8182 |
+| 0.4105        | 2.8917 | 11000 | 0.5272          | 0.8360   | 0.8085    | 0.8654 | 0.8222   | 0.8812 | 0.8447 |
+| 0.4234        | 3.0231 | 11500 | 0.4989          | 0.8378   | 0.8054    | 0.8729 | 0.8230   | 0.8768 | 0.8333 |
+
+
+### Framework versions
+
+- Transformers 4.42.3
+- Pytorch 2.3.0+cu121
+- Datasets 2.18.0
+- Tokenizers 0.19.0

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "AIRI-Institute/gena-lm-bert-base-t2t",
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoModel": "AIRI-Institute/gena-lm-bert-base-t2t--modeling_bert.BertForMaskedLM"
+  },
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "last_layer_norm": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 3,
+  "position_embedding_type": "absolute",
+  "pre_layer_norm": true,
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81499b8e3ec86eaf16f2b76ddfdf956ffab0f1fcf38450249b7feb422183239e
+size 442503040

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,60 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "-",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9307c2086be4ab6004778dc569b6a68f4a6e1ddc458840ff2499f1499c1e228
+size 5240