Model Card for Model ID
IRIISNEPAL/RoBERTa_Nepali_110M is a RoBERTa-based transformer model developed specifically for the Nepali language. This 110-million-parameter model is intended for tasks in natural language understanding (NLU), such as sentiment analysis, text classification, and named entity recognition in Nepali.
Model Details
Model Description
- Developed by: Institute of Research and Innovation in Intelligent Systems (IRIIS)
- Model type: RoBERTa-based transformer model specifically trained on Nepali language data
- Model Size: 110 million parameters
- Language (NLP): Nepali
- Training Objective: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)
The IRIISNEPAL/RoBERTa_Nepali_110M model aims to provide a robust tool for NLP tasks specific to the Nepali language, supporting NLP research and applications within low-resource languages.
Uses
Direct Use
The model provides contextual embeddings for each token in an input sequence (last_hidden_state
) and a pooled representation of the entire input (pooler_output
). These outputs can be used for:
- Text Classification: Using
pooler_output
to classify the overall sentiment, intent, or category of a sentence. - Token-Level Tasks: Leveraging
last_hidden_state
to perform tasks like named entity recognition (NER) or part-of-speech tagging by predicting labels for individual tokens. - Sentence Embeddings: Using
pooler_output
as an embedding for the entire input text for similarity search or clustering tasks.
Downstream Use
The model was evaluated on the Nepali Language Understanding Evaluation (Nep-gLUE) benchmark, demonstrating strong performance across various natural language understanding (NLU) tasks:
- Named Entity Recognition (NER): 93.74
- Part-of-Speech (POS) Tagging: 97.52
- Categorical Classification (CC): 94.68
- Categorical Pair Similarity (CPS): 96.49
These results indicate the model’s effectiveness in capturing language nuances for multiple NLU tasks in Nepali.
Bias, Risks, and Limitations
The model may exhibit biases present in its training data, especially regarding social, cultural, and regional aspects of the Nepali language. Users should exercise caution when deploying it in applications that might perpetuate stereotypes or cultural biases.
Recommendations
It’s advisable for users to monitor model outputs for fairness and avoid high-stakes applications without thorough testing. Fine-tuning or retraining may be necessary for sensitive applications.
How to Get Started with the Model
Use the code below to get started with the model.
# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")
model = AutoModel.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")
text = "नेपालमा पर्यटनको विकास गर्नुपर्ने आवश्यकता छ।"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
Training Details
Training Data
The model was trained on a 27.5 GB Nepali language corpus compiled from 99 Nepali news websites. This dataset represents the largest Nepali language corpus to date, providing a significant expansion in training resources for the language. The preprocessing involved deduplication, translation/removal of non-Nepali content, and noise reduction.
You can find detailed information about the dataset in the dataset card on Hugging Face.
Training Procedure
- Training Regime: Mixed precision (fp16) on TPU v4-8 hardware
- Batch Size: 256
- Learning Rate: 1e-4 with a warmup over the first 10,000 steps followed by linear decay
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Max Sequence Length: 512 tokens
- Learning Rate Scheduler: Linear with warmup
- Optimizer: AdamW with β1 = 0.9, β2 = 0.999, and L2 weight decay of 0.01
- Dropout Probability: 0.1 across all layers
- Activation Function: GELU
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on the Nepali Language Evaluation Benchmark (Nep-gLUE), which includes tasks like Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity.
Metrics
- Macro-F1 for the Nepali Language Understanding Evaluation (Nep-gLUE) benchmark.
Results
On Nep-gLUE, the model outperformed existing state-of-the-art models with an overall score of 95.60, reflecting its strong language understanding capabilities.
Model Examination
Performance analysis indicates robustness in capturing grammatical and syntactical features of Nepali. However, the model may have limited effectiveness in handling dialect-specific content or informal language.
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications
Model Architecture and Objective
RoBERTa architecture with 12 transformer layers, hidden size of 768, 12 attention heads, and 110 million parameters. This architecture facilitates strong bidirectional attention for accurate language understanding.
Compute Infrastructure
- Hardware: TPU v4-8 and Nvidia GeForce RTX 3090 GPUs
- Software: Python, PyTorch, Hugging Face Transformers
Citation
BibTeX:
@misc{IRIISNEPAL_RoBERTa_Nepali_110M,
title = {Development of Pre-trained Transformer-based Models for the Nepali Language},
author = {Thapa, Prajwal and Nyachhyon, Jinu and Sharma, Mridul and Bal, Bal Krishna},
year = {2024},
note = {Submitted to COLING 2025}
}
APA:
Thapa, P., Nyachhyon, J., Sharma, M., & Bal, B. K. (2024). Development of pre-trained transformer-based models for the Nepali language. Manuscript submitted for publication to COLING 2025.
Model Card Contact
For questions and support, contact IRIIS Nepal.
- Downloads last month
- 15
Dataset used to train IRIISNEPAL/RoBERTa_Nepali_110M
Collection including IRIISNEPAL/RoBERTa_Nepali_110M
Evaluation results
- Macro F1 Score on Nep-gLUE - Named Entity Recognition (NER)self-reported93.740
- Macro F1 Score on Nep-gLUE - Part-of-Speech (POS) Taggingself-reported97.520
- Macro F1 Score on Nep-gLUE - Categorical Classification (CC)self-reported94.680
- Macro F1 Score on Nep-gLUE - Categorical Pair Similarity (CPS)self-reported96.490
- Overall Score on Nep-gLUEself-reported95.600