PhilBERT: Phishing Detection with DistilBERT
PhilBERT is a fine-tuned DistilBERT model optimized for detecting phishing threats across multiple communication channels, including emails, SMS, URLs, and websites. It is trained on a diverse dataset sourced from Kaggle, Mendeley, Phishing.Database, and Bancolombia, ensuring high adaptability and real-world applicability.
Key Features
- Multi-Channel Detection – Analyzes text, URLs, and web content to detect phishing patterns.
- Fine-Tuned on Real-World Data – Includes recent three months of financial institution data (Bancolombia).
- Lightweight & Efficient – Based on DistilBERT, providing high performance with reduced computational costs.
- High Accuracy – Achieves 85.22% precision, 93.81% recall, and 88.77% accuracy on unseen data.
- Self-Adaptive Learning – Continuously evolves using real-time phishing simulations generated with GPT-4o.
- Scalability – Designed to support 7,000–25,000 simultaneous users in production environments.
Model Architecture
PhilBERT leverages DistilBERT, a distilled version of BERT, maintaining the same architecture but with 40% fewer parameters, making it lightweight while preserving high accuracy. The final model includes:
- Tokenizer: Trained to recognize phishing-specific patterns (URLs, obfuscation, domain misspellings).
- Custom Classifier: A fully connected dense layer added for binary classification (phishing vs. benign).
- Risk Scoring Mechanism: A weighted confidence score applied to enhance detection reliability.
Data Preprocessing
Before fine-tuning, the dataset underwent extensive preprocessing to ensure balance and quality:
- Duplicate Removal & Balancing: Maintained a near 50-50 phishing-to-benign ratio to prevent model bias.
- Feature Extraction: Applied to URLs, HTML, email bodies, and SMS content to enrich input representations.
- Dataset Split: Final dataset included:
- 427,028 benign URLs & 381,014 phishing URLs
- 17,536 unique email samples
- 5,949 SMS samples
- Web entries filtered for efficiency (removing entries >100KB).
- Export Format: Data transformed and stored in JSON for efficient training.
Training & Evaluation
PhilBERT was fine-tuned on multi-modal phishing datasets using transfer learning, achieving:
| Metric | Value |
|---|---|
| Accuracy | 88.77% |
| Precision | 85.22% |
| Recall | 93.81% |
| F1-Score | 89.31% |
| Evaluation Runtime | 130.46s |
| Samples/sec | 58.701 |
- False Positive Reduction: Multi-layered filtering minimized false positives while maintaining high recall.
- Scalability: Successfully stress-tested for up to 25,000 simultaneous users.
- Compliance: Meets ISO 27001 and GDPR standards for security and privacy.
Usage
Installation
pip install transformers torch
Inference
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
model_name = "your_username/PhilBERT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
text = "Click this link to update your bank details: http://fakebank.com"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
print(f"Phishing probability: {predictions[0][1].item():.4f}")
License
This model is proprietary and protected under a custom license. Please refer to the LICENSE file for terms of use.