Upload LayoutFIDNetV3

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -1,70 +1,70 @@
 {
   "architectures": [
-    "LayoutDmFIDNetV3"
+    "LayoutFIDNetV3"
   ],
   "auto_map": {
-    "AutoConfig": "configuration_fidnet_v3.LayoutDmFIDNetV3Config",
-    "AutoModel": "modeling_fidnet_v3.LayoutDmFIDNetV3"
+    "AutoConfig": "configuration_layout_fidnet_v3.LayoutFIDNetV3Config",
+    "AutoModel": "modeling_layout_fidnet_v3.LayoutFIDNetV3"
   },
   "d_model": 256,
   "id2label": {
-    "0": "Text",
-    "1": "Image",
-    "2": "Icon",
-    "3": "Text Button",
-    "4": "List Item",
-    "5": "Input",
-    "6": "Background Image",
-    "7": "Card",
-    "8": "Web View",
-    "9": "Radio Button",
-    "10": "Drawer",
-    "11": "Checkbox",
-    "12": "Advertisement",
-    "13": "Modal",
-    "14": "Pager Indicator",
-    "15": "Slider",
-    "16": "On/Off Switch",
-    "17": "Button Bar",
-    "18": "Toolbar",
-    "19": "Number Stepper",
-    "20": "Multi-Tab",
-    "21": "Date Picker",
-    "22": "Map View",
-    "23": "Video",
-    "24": "Bottom Navigation"
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6",
+    "7": "LABEL_7",
+    "8": "LABEL_8",
+    "9": "LABEL_9",
+    "10": "LABEL_10",
+    "11": "LABEL_11",
+    "12": "LABEL_12",
+    "13": "LABEL_13",
+    "14": "LABEL_14",
+    "15": "LABEL_15",
+    "16": "LABEL_16",
+    "17": "LABEL_17",
+    "18": "LABEL_18",
+    "19": "LABEL_19",
+    "20": "LABEL_20",
+    "21": "LABEL_21",
+    "22": "LABEL_22",
+    "23": "LABEL_23",
+    "24": "LABEL_24"
   },
   "label2id": {
-    "Advertisement": 12,
-    "Background Image": 6,
-    "Bottom Navigation": 24,
-    "Button Bar": 17,
-    "Card": 7,
-    "Checkbox": 11,
-    "Date Picker": 21,
-    "Drawer": 10,
-    "Icon": 2,
-    "Image": 1,
-    "Input": 5,
-    "List Item": 4,
-    "Map View": 22,
-    "Modal": 13,
-    "Multi-Tab": 20,
-    "Number Stepper": 19,
-    "On/Off Switch": 16,
-    "Pager Indicator": 14,
-    "Radio Button": 9,
-    "Slider": 15,
-    "Text": 0,
-    "Text Button": 3,
-    "Toolbar": 18,
-    "Video": 23,
-    "Web View": 8
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_10": 10,
+    "LABEL_11": 11,
+    "LABEL_12": 12,
+    "LABEL_13": 13,
+    "LABEL_14": 14,
+    "LABEL_15": 15,
+    "LABEL_16": 16,
+    "LABEL_17": 17,
+    "LABEL_18": 18,
+    "LABEL_19": 19,
+    "LABEL_2": 2,
+    "LABEL_20": 20,
+    "LABEL_21": 21,
+    "LABEL_22": 22,
+    "LABEL_23": 23,
+    "LABEL_24": 24,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5,
+    "LABEL_6": 6,
+    "LABEL_7": 7,
+    "LABEL_8": 8,
+    "LABEL_9": 9
   },
   "max_bbox": 25,
   "model_type": "layoutdm_fidnet_v3",
   "nhead": 4,
   "num_layers": 4,
   "torch_dtype": "float32",
-  "transformers_version": "4.36.2"
+  "transformers_version": "4.43.3"
 }

configuration_layout_fidnet_v3.py

@@ -0,0 +1,23 @@
+from transformers.configuration_utils import PretrainedConfig
+
+
+class LayoutFIDNetV3Config(PretrainedConfig):
+    model_type = "layoutdm_fidnet_v3"
+
+    def __init__(
+        self,
+        num_labels: int = 1,
+        d_model: int = 256,
+        nhead: int = 4,
+        num_layers: int = 4,
+        max_bbox: int = 50,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            num_labels=num_labels,
+            **kwargs,
+        )
+        self.d_model = d_model
+        self.nhead = nhead
+        self.num_layers = num_layers
+        self.max_bbox = max_bbox

modeling_layout_fidnet_v3.py

@@ -0,0 +1,145 @@
+import logging
+from dataclasses import dataclass
+from typing import Optional
+
+import torch
+import torch.nn as nn
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import ModelOutput
+
+from .configuration_layout_fidnet_v3 import LayoutFIDNetV3Config
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class LayoutFIDNetV3Output(ModelOutput):
+    logit_disc: torch.Tensor
+    logit_cls: torch.Tensor
+    bbox_pred: torch.Tensor
+
+
+class TransformerWithToken(nn.Module):
+    def __init__(
+        self,
+        d_model: int,
+        nhead: int,
+        dim_feedforward: int,
+        num_layers: int,
+    ) -> None:
+        super().__init__()
+
+        self.token = nn.Parameter(torch.randn(1, 1, d_model))
+        token_mask = torch.zeros(1, 1, dtype=torch.bool)
+        self.register_buffer("token_mask", token_mask)
+
+        self.core = nn.TransformerEncoder(
+            nn.TransformerEncoderLayer(
+                d_model=d_model,
+                nhead=nhead,
+                dim_feedforward=dim_feedforward,
+            ),
+            num_layers=num_layers,
+        )
+
+    def forward(self, x, src_key_padding_mask):
+        # x: [N, B, E]
+        # padding_mask: [B, N]
+        #   `False` for valid values
+        #   `True` for padded values
+
+        B = x.size(1)
+
+        token = self.token.expand(-1, B, -1)
+        x = torch.cat([token, x], dim=0)
+
+        token_mask = self.token_mask.expand(B, -1)
+        padding_mask = torch.cat([token_mask, src_key_padding_mask], dim=1)
+
+        x = self.core(x, src_key_padding_mask=padding_mask)
+
+        return x
+
+
+class LayoutFIDNetV3(PreTrainedModel):
+    config_class = LayoutFIDNetV3Config
+
+    def __init__(self, config: LayoutFIDNetV3Config) -> None:
+        super().__init__(config)
+
+        # encoder
+        self.emb_label = nn.Embedding(config.num_labels, config.d_model)
+        self.fc_bbox = nn.Linear(4, config.d_model)
+        self.enc_fc_in = nn.Linear(config.d_model * 2, config.d_model)
+
+        self.enc_transformer = TransformerWithToken(
+            d_model=config.d_model,
+            dim_feedforward=config.d_model // 2,
+            nhead=config.nhead,
+            num_layers=config.num_layers,
+        )
+
+        self.fc_out_disc = nn.Linear(config.d_model, 1)
+
+        # decoder
+        self.pos_token = nn.Parameter(torch.rand(config.max_bbox, 1, config.d_model))
+        self.dec_fc_in = nn.Linear(config.d_model * 2, config.d_model)
+
+        te = nn.TransformerEncoderLayer(
+            d_model=config.d_model,
+            nhead=config.nhead,
+            dim_feedforward=config.d_model // 2,
+        )
+        self.dec_transformer = nn.TransformerEncoder(te, num_layers=config.num_layers)
+
+        self.fc_out_cls = nn.Linear(config.d_model, config.num_labels)
+        self.fc_out_bbox = nn.Linear(config.d_model, 4)
+
+    def extract_features(self, bbox, label, padding_mask):
+        b = self.fc_bbox(bbox)
+        l = self.emb_label(label)
+        x = self.enc_fc_in(torch.cat([b, l], dim=-1))
+        x = torch.relu(x).permute(1, 0, 2)
+        x = self.enc_transformer(x, padding_mask)
+        return x[0]
+
+    def forward(self, bbox, label, padding_mask):
+        B, N, _ = bbox.size()
+        x = self.extract_features(bbox, label, padding_mask)
+
+        logit_disc = self.fc_out_disc(x).squeeze(-1)
+
+        x = x.unsqueeze(0).expand(N, -1, -1)
+        t = self.pos_token[:N].expand(-1, B, -1)
+        x = torch.cat([x, t], dim=-1)
+        x = torch.relu(self.dec_fc_in(x))
+
+        x = self.dec_transformer(x, src_key_padding_mask=padding_mask)
+        # x = x.permute(1, 0, 2)[~padding_mask]
+        x = x.permute(1, 0, 2)
+
+        # logit_cls: [B, N, L]    bbox_pred: [B, N, 4]
+        logit_cls = self.fc_out_cls(x)
+        bbox_pred = torch.sigmoid(self.fc_out_bbox(x))
+
+        return LayoutFIDNetV3Output(
+            logit_disc=logit_disc, logit_cls=logit_cls, bbox_pred=bbox_pred
+        )
+
+
+def convert_from_checkpoint(
+    repo_id: str, filename: str, config: Optional[LayoutFIDNetV3Config] = None
+) -> LayoutFIDNetV3:
+    from huggingface_hub import hf_hub_download
+
+    checkpoint_path = hf_hub_download(repo_id=repo_id, filename=filename)
+    config = config or LayoutFIDNetV3Config()
+    model = LayoutFIDNetV3(config)
+
+    logger.info(f"Loading model from {checkpoint_path}")
+    state_dict = torch.load(checkpoint_path, map_location="cpu")["state_dict"]
+
+    model.load_state_dict(state_dict, strict=True)
+    model.eval()
+
+    return model