README.md

metadata

license: unknown
base_model: microsoft/deberta-v3-base
tags:
  - medical
  - biology
  - NER
  - Biomedical
  - deberta
  - dataset
pipeline_tag: token-classification
language:
  - en

BIOMed_NER: Named Entity Recognition for Biomedical Entities

Model Overview: BIOMed_NER is a Named Entity Recognition (NER) model which identifies biomedical entities using DeBERTaV3. This model is useful for extracting structured information from clinical text, such as diseases, procedures, medications, and anatomical terms. Here's a more detailed and enthusiastic introduction emphasizing the strengths of DeBERTa and why it's an excellent choice for your BIOMed_NER model:

---

Why DeBERTa for Biomedical NER?

DeBERTa (Decoding-enhanced BERT with Disentangled Attention) represents a significant leap forward in NLP model architecture, particularly for nuanced tasks like Named Entity Recognition (NER) in complex domains such as biomedical texts. Here’s why DeBERTa was the ideal choice for BIOMed_NER:

1. Advanced Disentangled Attention Mechanism:

   • DeBERTa goes beyond traditional transformers by using a unique disentangled attention mechanism that separately encodes word content and word position. This allows DeBERTa to capture the contextual meaning of biomedical terms and understand complex sentence structures, which is essential for accurately tagging biomedical entities that often have overlapping or highly specific terms.

2. Enhanced Embedding for Richer Contextual Understanding:

   • Biomedical text often contains long sentences, specialized terminology, and hierarchical relationships between entities (e.g., "diabetes" vs. "Type 1 diabetes"). DeBERTa’s improved embedding layer allows it to capture these nuanced relationships better than traditional transformer models, making it especially effective in understanding context-rich medical documents.

3. Superior Performance on Downstream NLP Tasks:

   • DeBERTa consistently ranks among the top models on NLP benchmarks like GLUE and SQuAD, which is a testament to its ability to generalize across tasks. This high performance is especially beneficial for BIOMed_NER, where accurate recognition of subtle differences between biomedical entities can significantly enhance the quality of structured data extracted from unstructured clinical notes.

4. Pre-trained for Optimal Transfer Learning:

   • Leveraging the "base" DeBERTaV3 variant allows us to tap into a model pre-trained on vast amounts of text, thus providing an excellent foundation for fine-tuning on domain-specific biomedical data. This pre-training, combined with the fine-tuning on the dataset, allows BIOMed_NER to accurately distinct biomedical entities, from diseases and medications to clinical events and anatomical structures.

5. Efficient Fine-Tuning for Large Biomedical Datasets:

   • DeBERTa is optimized for both accuracy and efficiency, making it easier to train on large and complex datasets without needing excessive computational resources. This means faster iterations during model development and a more accessible deployment pipeline.

By selecting DeBERTa for BIOMed_NER, we've built a model that excels in understanding the intricate language of medicine, providing high accuracy and contextual depth essential for healthcare applications. Whether for researchers analyzing clinical data or applications structuring patient records, DeBERTa enables BIOMed_NER to extract, tag, and organize critical medical information effectively. Hyperparameters:

• Base Model: microsoft/deberta-v3-base
• Learning Rate: 3e-5
• Batch Size: 8
• Gradient Accumulation Steps: 2
• Scheduler: Cosine schedule with warmup
• Epochs: 30
• Optimizer: AdamW with betas (0.9, 0.999) and epsilon 1e-8

How to Use the Model for Inference:

You can use the Hugging Face pipeline for easy inference:

from transformers import pipeline

# Load the model
model_path = "venkatd/BIOMed_NER"
pipe = pipeline(
    task="token-classification",
    model=model_path,
    tokenizer=model_path,
    aggregation_strategy="simple"
)

# Test the pipeline
text = ("A 48-year-old female presented with vaginal bleeding and abnormal Pap smears. "
        "Upon diagnosis of invasive non-keratinizing SCC of the cervix, she underwent a radical "
        "hysterectomy with salpingo-oophorectomy which demonstrated positive spread to the pelvic "
        "lymph nodes and the parametrium.")
result = pipe(text)
print(result)

Output Example:

The output will be a list of recognized entities with their entity type, score, and start/end positions in the text. Here’s a sample output format:

[
    {
        "entity_group": "Disease_disorder",
        "score": 0.98,
        "word": "SCC of the cervix",
        "start": 63,
        "end": 80
    },
    ...
]

Use Cases:

• Extracting clinical information from unstructured text in medical records.
• Structuring data for downstream biomedical research or applications.
• Assisting healthcare professionals by highlighting relevant biomedical entities.

This model is publicly available on Hugging Face and can be easily integrated into applications for medical text analysis.