DeepChem

/

ChemBERTa-77M-MTR

like 9

Follow

DeepChem 39

Found the same issue. I'm guessing this might be associated with this discussion too: https://github.com/huggingface/transformers/issues/25362

Hello! 

The `RobertaForRegression` architecture does not exist natively in the Transformers library. It is likely a custom architecture that needs to be manually implemented for regression tasks.

Here’s how you can build a custom `RobertaForRegression` model using the `RobertaModel` as a base and adding a regression head:

```python
from transformers import RobertaModel, RobertaConfig
import torch.nn as nn

# Define a custom RobertaForRegression class
class RobertaForRegression(nn.Module):
    def __init__(self, config: RobertaConfig):
        super().__init__()
        self.roberta = RobertaModel(config)  # Load the base RoBERTa model
        self.regressor = nn.Linear(config.hidden_size, 1)  # Add a regression layer

    def forward(self, input_ids, attention_mask):
        # Forward pass through RoBERTa
        outputs = self.roberta(input_ids=input_ids, attention_mask=attention_mask)
        # Extract [CLS] token output and pass it through the regression head
        regression_output = self.regressor(outputs.last_hidden_state[:, 0])
        return regression_output

Steps to Use This Custom Model:

1. Load Pre-Trained Weights: You can initialize the model using the pre-trained RobertaModel weights:

   from transformers import RobertaConfig
   
   config = RobertaConfig.from_pretrained("roberta-base")
   model = RobertaForRegression(config)
   

2. Train the Model: Train this model with your regression dataset by defining a suitable loss function, such as Mean Squared Error (MSE).

3. Save and Upload: Once trained, you can save and upload the custom model to the Hugging Face Hub using push_to_hub.

Key Points to Clarify:

• RobertaForRegression Is Not a Default Model: Transformers provides general-purpose architectures like RobertaForSequenceClassification, but for tasks like regression, customization is required.
• Why Customize: Regression tasks often need outputs in the form of continuous values, unlike classification tasks that output probabilities over discrete categories.
• Implementation Flexibility: Customizing architectures allows users to fine-tune models for domain-specific tasks and datasets.

For additional help, you can explore the Transformers documentation or check out similar examples in the community forums.