Model Card for mdeberta-v3-base-re-ct

This relation extraction model extracts intervention-associated relationships, temporal relations, negation/speculation and others relevant for clinical trials.

The model achieves the following results on the test set (when trained with the training and development set; results are averaged over 5 evaluation rounds):

Precision: 0.886 (±0.003)
Recall: 0.857 (±0.007)
F1: 0.869 (±0.005)
Accuracy: 0.911 (±0.003)

Model description

This model adapts the pre-trained model mdeberta-v3-base. It is fine-tuned to conduct relation extraction on Spanish texts about clinical trials. The model is fine-tuned on the Clinical Trials for Evidence-Based-Medicine in Spanish corpus.

If you use this model, please, cite as follows:

@article{campillosetal2025,
        title = {{Benchmarking Transformer Models for Relation Extraction and Concept Normalization in a Clinical Trials Corpus}},
        author = {Campillos-Llanos, Leonardo and Valverde-Mateos, Ana and Capllonch-Carri{\'o}n, Adri{\'a}n and Zakhir-Puig, Sof{\'i}a and Heras-Vicente, J{\'o}nathan},
        journal = {(Under review)},
        year={2025}
}

Intended uses & limitations

Disclosure: This model is under development and needs to be improved. It should not be used for medical decision making without human assistance and supervision

This model is intended for a generalist purpose, and may have bias and/or any other undesirable distortions.

Third parties who deploy or provide systems and/or services using any of these models (or using systems based on these models) should note that it is their responsibility to mitigate the risks arising from their use. Third parties, in any event, need to comply with applicable regulations, including regulations concerning the use of artificial intelligence.

The owner or creator of the models will in no event be liable for any results arising from the use made by third parties of these models.

Descargo de responsabilidad: Esta herramienta se encuentra en desarrollo y no debe ser empleada para la toma de decisiones médicas

La finalidad de este modelo es generalista, y se advierte que puede tener sesgos y/u otro tipo de distorsiones indeseables.

Terceras partes que desplieguen o proporcionen sistemas y/o servicios usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) han tener presente que es su responsabilidad abordar y minimizar los riesgos derivados de su uso. Las terceras partes, en cualquier circunstancia, deben cumplir con la normativa aplicable, incluyendo la normativa que concierne al uso de la inteligencia artificial.

El propietario o creador de los modelos de ningún modo será responsable de los resultados derivados del uso que las terceras partes hagan de estos modelos.

Training and evaluation data

The data used for fine-tuning are the Clinical Trials for Evidence-Based-Medicine in Spanish corpus version 3 (annotated with semantic relationships). It is a collection of 1200 texts about clinical trials studies and clinical trials announcements:

500 abstracts from journals published under a Creative Commons license, e.g. available in PubMed or the Scientific Electronic Library Online (SciELO)
700 clinical trials announcements published in the European Clinical Trials Register and Repositorio Español de Estudios Clínicos

The CT-EBM-ES resource (version 1) can be cited as follows:

@article{campillosetal-midm2021,
        title = {A clinical trials corpus annotated with UMLS© entities to enhance the access to Evidence-Based Medicine},
        author = {Campillos-Llanos, Leonardo and Valverde-Mateos, Ana and Capllonch-Carri{\'o}n, Adri{\'a}n and Moreno-Sandoval, Antonio},
        journal = {BMC Medical Informatics and Decision Making},
        volume={21},
        number={1},
        pages={1--19},
        year={2021},
        publisher={BioMed Central}
}

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 16
eval_batch_size: 16
seed: we used different seeds for 5 evaluation rounds, and uploaded the model with the best results
optimizer: AdamW
weight decay: 1e-2
lr_scheduler_type: linear
num_epochs: 5 epochs.

Training results (test set; average and standard deviation of 5 rounds with different seeds)

Precision	Recall	F1	Accuracy
0.886 (±0.003)	0.857 (±0.007)	0.869 (±0.005)	0.911 (±0.003)

Results per class (test set; best model)

Class	Precision	Recall	F1	Support
Experiences	0.96	0.97	0.97	2003
Has_Age	0.93	0.84	0.88	152
Has_Dose_or_Strength	0.84	0.81	0.83	189
Has_Drug_Form	0.90	0.73	0.81	64
Has_Duration_or_Interval	0.83	0.84	0.84	365
Has_Frequency	0.79	0.86	0.82	84
Has_Quantifier_or_Qualifier	0.91	0.89	0.90	1040
Has_Result_or_Value	0.92	0.87	0.89	384
Has_Route_or_Mode	0.91	0.87	0.89	221
Has_Time_Data	0.83	0.91	0.86	589
Location_of	0.96	0.96	0.96	1119
Used_for	0.89	0.88	0.89	731

Usage

To use this model you need to install the datasets library.

pip install datasets

Then you can define the necessary functions and classes to load the model.

from transformers import (
    DebertaV2Model, PreTrainedModel,
    DataCollatorWithPadding,AutoTokenizer
)
from transformers.modeling_outputs import SequenceClassifierOutput
import torch
import torch.nn as nn
from datasets import Dataset
from torch.utils.data import DataLoader


class DebertaV2ForRelationExtraction(PreTrainedModel):
  def __init__(self, config, num_labels):
    super(DebertaV2ForRelationExtraction, self).__init__(config)
    self.num_labels = num_labels
    # body
    self.deberta = DebertaV2Model(config)
    # head
    self.dropout = nn.Dropout(config.hidden_dropout_prob)
    self.layer_norm = nn.LayerNorm(config.hidden_size * 2)
    self.linear = nn.Linear(config.hidden_size * 2, self.num_labels)
    self.init_weights()

  def forward(self, input_ids, token_type_ids, attention_mask,
              span_idxs, labels=None):
    outputs = (
        self.deberta(input_ids, token_type_ids=token_type_ids,
                  attention_mask=attention_mask,
                  output_hidden_states=False)
            .last_hidden_state)

    sub_maxpool, obj_maxpool = [], []
    for bid in range(outputs.size(0)):
      # span includes entity markers, maxpool across span
      sub_span = torch.max(outputs[bid, span_idxs[bid, 0]:span_idxs[bid, 1]+1, :],
                           dim=0, keepdim=True).values
      obj_span = torch.max(outputs[bid, span_idxs[bid, 2]:span_idxs[bid, 3]+1, :],
                           dim=0, keepdim=True).values
      sub_maxpool.append(sub_span)
      obj_maxpool.append(obj_span)

    sub_emb = torch.cat(sub_maxpool, dim=0)
    obj_emb = torch.cat(obj_maxpool, dim=0)
    rel_input = torch.cat((sub_emb, obj_emb), dim=-1)

    rel_input = self.layer_norm(rel_input)
    rel_input = self.dropout(rel_input)
    logits = self.linear(rel_input)

    if labels is not None:
      loss_fn = nn.CrossEntropyLoss()
      loss = loss_fn(logits.view(-1, self.num_labels), labels.view(-1))
      return SequenceClassifierOutput(loss, logits)
    else:
      return SequenceClassifierOutput(None, logits)

id2label = {0: 'Experiences',
 1: 'Has_Age',
 2: 'Has_Dose_or_Strength',
 3: 'Has_Duration_or_Interval',
 4: 'Has_Frequency',
 5: 'Has_Route_or_Mode',
 6: 'Location_of',
 7: 'Used_for'}

def encode_data_inference(token_list,tokenizer):
  tokenized_inputs = tokenizer(token_list,
                               is_split_into_words=True,
                               truncation=True)
  span_idxs = []
  for input_id in tokenized_inputs.input_ids:
    tokens = tokenizer.convert_ids_to_tokens(input_id)
    span_idxs.append([
      [idx for idx, token in enumerate(tokens) if token.startswith("<S:")][0],
      [idx for idx, token in enumerate(tokens) if token.startswith("</S:")][0],
      [idx for idx, token in enumerate(tokens) if token.startswith("<O:")][0],
      [idx for idx, token in enumerate(tokens) if token.startswith("</O:")][0]
    ])
  tokenized_inputs["span_idxs"] = span_idxs
  # tokenized_inputs["labels"] = [label2id[label] for label in examples["label"]]
  return tokenized_inputs

def predict_example(example,model,tokenizer):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    collate_fn = DataCollatorWithPadding(tokenizer, padding="longest", return_tensors="pt")


    encoded_data = encode_data_inference(example,tokenizer)

    inferenceds = Dataset.from_dict(encoded_data)

    inference_dl = DataLoader(inferenceds, 
                         shuffle=False,
                        #  sampler=SubsetRandomSampler(np.random.randint(0, encoded_nyt_dataset["test"].num_rows, 100).tolist()),
                         batch_size=1, 
                         collate_fn=collate_fn)
    for batch in inference_dl:
        batch = {k: v.to(device) for k, v in batch.items()}
        with torch.no_grad():
          outputs = model(**batch)
          predictions = torch.argmax(outputs.logits, dim=-1).cpu().numpy()
    return [id2label[p] for p in predictions]

Finally, you can use it to make predictions:

example = [['Título',
  'público:',
  'Estudio',
  'multicéntrico,',
  'aleatorizado,',
  'doble',
  'ciego,',
  'controlado',
  'con',
  'placebo',
  'del',
  'anticuerpo',
  'monoclonal',
  'humano',
  'anti-TNF',
  'Adalimumab',
  'en',
  '<S:LIV>',
  'sujetos',
  'pediátricos',
  '</S:LIV>',
  'con',
  'colitis',
  'ulcerosa',
  'moderada',
  'o',
  'grav<O:CHE>',
  'Adalimumab',
  '</O:CHE>blico:',
  'Estudio',
  'multicéntrico,',
  'aleatorizado,',
  'doble',
  'ciego,',
  'controlado',
  'con',
  'placebo',
  'del',
  'anticuerpo',
  'monoclonal',
  'humano',
  'anti-TNF',
  'Adalimumab',
  'en',
  'sujetos',
  'pediátricos',
  'con',
  'colitis',
  'ulcerosa',
  'moderada',
  'o',
  'grave']]

model = DebertaV2ForRelationExtraction.from_pretrained("medspaner/mdeberta-v3-base-re-ct-v2",8)  
tokenizer = AutoTokenizer.from_pretrained("medspaner/mdeberta-v3-base-re-ct-v2")
predict_example(example,model,tokenizer)

Framework versions

Transformers 4.42.4
Pytorch 2.0.1+cu117
Datasets 2.15.0
Tokenizers 0.19.1

medspaner
/

mdeberta-v3-base-re-ct-v2