|
--- |
|
metrics: |
|
- mse |
|
- r_squared |
|
- mae |
|
datasets: |
|
- google_wellformed_query |
|
inference: false |
|
model-index: |
|
- name: distilroberta-query-wellformedness |
|
results: |
|
- task: |
|
type: text-classification |
|
name: Text Classification |
|
metrics: |
|
- type: loss |
|
value: 0.061837393790483475 |
|
- type: mse |
|
value: 0.061837393790483475 |
|
name: Validation Mean Squared Error |
|
- type: r2 |
|
value: 0.5726782083511353 |
|
name: Validation R-Squared |
|
- type: mae |
|
value: 0.183049738407135 |
|
name: Validation Mean Absolute Error |
|
language: |
|
- en |
|
--- |
|
## DistilRoBERTa-query-wellformedness |
|
|
|
This model utilizes the [Distilroberta base](https://huggingface.co/distilroberta-base) architecture, which has been fine-tuned for a regression task on the [Google's query wellformedness](https://huggingface.co/datasets/google_wellformed_query) dataset encompassing 25,100 queries from the Paralex corpus. Each query received annotations from five raters, who provided a continuous rating indicating the degree to which the query is well-formed. |
|
|
|
## Model description |
|
|
|
A regression head has been appended to the DistilRoBERTa model to tailor it for a regression task. This additional component is crucial and needs to be loaded alongside the base model during inference to ensure accurate predictions. The model evaluates the query for completeness and grammatical correctness, providing a score between 0 and 1, where 1 indicates correctness. |
|
|
|
## Usage |
|
|
|
Inference API has been disabled as this is a regression task, not a text classification task, and HuggingFace does not provide a pipeline for regression tasks. |
|
Because of the dataset, it will perform better when handling queries in question form. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
tokenizer = AutoTokenizer.from_pretrained("AdamCodd/distilroberta-query-wellformedness") |
|
|
|
class RegressionModel(torch.nn.Module): |
|
def __init__(self): |
|
super().__init__() |
|
self.model = AutoModelForSequenceClassification.from_pretrained("AdamCodd/distilroberta-query-wellformedness") |
|
self.regression_head = torch.nn.Linear(self.model.config.hidden_size, 1) |
|
|
|
def forward(self, input_ids, attention_mask, **kwargs): |
|
outputs = self.model.base_model(input_ids=input_ids, attention_mask=attention_mask) |
|
rating = self.regression_head(outputs.last_hidden_state[:, 0, :]) |
|
rating = torch.sigmoid(rating) |
|
return rating.squeeze() |
|
|
|
regression_model = RegressionModel() |
|
# Do not forget to set the correct path to load the regression head |
|
regression_model.regression_head.load_state_dict(torch.load("path_to_the_regression_head.pth")) |
|
regression_model.eval() |
|
# Examples |
|
sentences = [ |
|
"The cat and dog in the yard.", |
|
"she don't like apples.", |
|
"Is rain sunny days sometimes?", |
|
"She enjoys reading books and playing chess.", |
|
"How many planets are there in our solar system?" |
|
] |
|
|
|
inputs = tokenizer(sentences, truncation=True, padding=True, return_tensors='pt') |
|
|
|
with torch.no_grad(): |
|
outputs = regression_model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask']) |
|
|
|
predictions = outputs.tolist() |
|
for i, rating in enumerate(predictions): |
|
print(f'Sentence: {sentences[i]}') |
|
print(f'Predicted Rating: {rating}\n') |
|
``` |
|
Output: |
|
``` |
|
Sentence: The cat and dog in the yard. |
|
Predicted Rating: 0.20430190861225128 |
|
|
|
Sentence: she don't like apples. |
|
Predicted Rating: 0.08289700001478195 |
|
|
|
Sentence: Is rain sunny days sometimes? |
|
Predicted Rating: 0.20011138916015625 |
|
|
|
Sentence: She enjoys reading books and playing chess. |
|
Predicted Rating: 0.8915354013442993 |
|
|
|
Sentence: How many planets are there in our solar system? |
|
Predicted Rating: 0.974799394607544 |
|
``` |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 16 |
|
- seed: 42 |
|
- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 400 |
|
- num_epochs: 5 |
|
|
|
### Training results |
|
|
|
Metrics: Mean Squared Error, R-Squared, Mean Absolute Error |
|
|
|
``` |
|
'test_loss': 0.061837393790483475, |
|
'test_mse': 0.061837393790483475, |
|
'test_r2': 0.5726782083511353, |
|
'test_mae': 0.183049738407135 |
|
``` |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.34.1 |
|
- Pytorch lightning 2.1.0 |
|
- Tokenizers 0.14.1 |
|
|
|
If you want to support me, you can [here](https://ko-fi.com/adamcodd). |
