Upload folder using huggingface_hub

CRISPR_transformer_model/README.md

@@ -0,0 +1,88 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+datasets:
+- crispr_data
+model-index:
+- name: SX_spymac_CRISPR_transformer
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# SX_spymac_CRISPR_transformer
+
+This model is a fine-tuned version of [](https://huggingface.co/) on the crispr_data dataset.
+It achieves the following results on the evaluation set:
+- Loss: 554919.0625
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 100
+- eval_batch_size: 100
+- seed: 63036
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 30.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 8892548.3058  | 1.0   | 327  | 2231862.75      |
+| 1596845.9939  | 2.0   | 654  | 767051.1875     |
+| 850237.4557   | 3.0   | 981  | 605501.0625     |
+| 586577.419    | 4.0   | 1308 | 559650.875      |
+| 557455.315    | 5.0   | 1635 | 556497.6875     |
+| 549405.8471   | 6.0   | 1962 | 556590.5625     |
+| 546295.633    | 7.0   | 2289 | 556962.8125     |
+| 544300.4281   | 8.0   | 2616 | 556499.75       |
+| 544035.5719   | 9.0   | 2943 | 556551.1875     |
+| 542642.2018   | 10.0  | 3270 | 556249.25       |
+| 541842.3486   | 11.0  | 3597 | 557194.25       |
+| 541445.1376   | 12.0  | 3924 | 556227.0        |
+| 541168.7339   | 13.0  | 4251 | 555564.25       |
+| 540981.8716   | 14.0  | 4578 | 556040.8125     |
+| 540638.9235   | 15.0  | 4905 | 556208.25       |
+| 540627.6208   | 16.0  | 5232 | 555999.9375     |
+| 540306.104    | 17.0  | 5559 | 555283.125      |
+| 540060.3792   | 18.0  | 5886 | 555715.625      |
+| 540050.3976   | 19.0  | 6213 | 555535.9375     |
+| 539878.1162   | 20.0  | 6540 | 555504.625      |
+| 539819.8899   | 21.0  | 6867 | 555669.3125     |
+| 539598.0428   | 22.0  | 7194 | 555249.3125     |
+| 539608.8073   | 23.0  | 7521 | 555444.9375     |
+| 539408.9786   | 24.0  | 7848 | 555080.8125     |
+| 539380.5994   | 25.0  | 8175 | 555413.8125     |
+| 539192.318    | 26.0  | 8502 | 554968.75       |
+| 539192.8073   | 27.0  | 8829 | 555104.0625     |
+| 539042.6911   | 28.0  | 9156 | 555027.5625     |
+| 538919.8287   | 29.0  | 9483 | 554871.5625     |
+| 538892.9174   | 30.0  | 9810 | 554919.0625     |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.4.0+cu124
+- Datasets 2.21.0
+- Tokenizers 0.19.1

CRISPR_transformer_model/config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "/home/ljw/sdc1/CRISPR_results/CRISPR_transformer/SX_spymac_CRISPR_transformer",
   "architectures": [
     "CRISPRTransformerModel"
   ],

CRISPR_transformer_model/model.py

@@ -0,0 +1,85 @@
+from transformers import PreTrainedModel, RoFormerConfig, RoFormerModel
+import torch.nn as nn
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+class CRISPRTransformerConfig(RoFormerConfig):
+    model_type = "CRISPR_transformer"
+    label_names = ["observation"]
+
+    def __init__(
+        self,
+        vocab_size = 4, # ACGT
+        hidden_size = 256, # model embedding dimension
+        num_hidden_layers = 3, # number of EncoderLayer
+        num_attention_heads = 4, # number of attention heads
+        intermediate_size = 1024, # FeedForward intermediate dimension size
+        hidden_dropout_prob = 0.1, # The dropout probability for all fully connected layers in the embeddings, encoder, and pooler
+        attention_probs_dropout_prob = 0.1, # The dropout ratio for the attention probabilities
+        max_position_embeddings = 256, # The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 1536).
+        ref1len = 127, # length of reference 1
+        ref2len = 127, # length of reference 2
+        seed = 63036, # random seed for intialization
+        **kwargs
+    ):
+        self.ref1len = ref1len
+        self.ref2len = ref2len
+        self.seed = seed
+        super().__init__(
+            vocab_size = vocab_size,
+            hidden_size = hidden_size,
+            num_hidden_layers = num_hidden_layers,
+            num_attention_heads = num_attention_heads,
+            intermediate_size = intermediate_size,
+            hidden_dropout_prob = hidden_dropout_prob,
+            attention_probs_dropout_prob = attention_probs_dropout_prob,
+            max_position_embeddings = max_position_embeddings,
+            **kwargs
+        )
+
+class CRISPRTransformerModel(PreTrainedModel):
+    config_class = CRISPRTransformerConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.generator = torch.Generator().manual_seed(config.seed)
+        self.model = RoFormerModel(config)
+        self.mlp = nn.Linear(
+            in_features=config.hidden_size,
+            out_features=(config.ref1len + 1) * (config.ref2len + 1)
+        )
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, mean=0, std=1, generator=self.generator)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, refcode: torch.Tensor, observation: torch.Tensor=None):
+        # refcode (batch_size X sequence_length)
+        # model(refcode) (batch_size X sequence_length X hidden_size)
+        # model(refcode)[:, -1, :] arbitrary choose the last position to predict the logits
+        batch_size = refcode.shape[0]
+        logit = self.mlp(
+            self.model(
+                input_ids=refcode,
+                attention_mask=torch.ones(
+                    batch_size,
+                    self.config.ref1len + self.config.ref2len,
+                    dtype=torch.int64,
+                    device=self.model.device
+                )
+            ).last_hidden_state[:, -1, :]
+        ).view(batch_size, self.config.ref2len + 1, self.config.ref1len + 1)
+        if observation is not None:
+            return {
+                "logit": logit,
+                "loss": - (
+                    observation.flatten(start_dim=1) *
+                    F.log_softmax(logit.flatten(start_dim=1), dim=1)
+                ).sum()
+            }
+        return {"logit": logit}

CRISPR_transformer_model/runs/Nov18_12-31-05_ljw-System-Product-Name/events.out.tfevents.1731904281.ljw-System-Product-Name.9190.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4db034badeb9cf76c1ba08a62994dbb915fd112fbccd877207c8f04c2c115a26
+size 19848

CRISPR_transformer_model/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c5da2219d03b02c849205903bd41f349d7a9e1a973daa848ba5c34df4ef2b81
+size 5304