w11wo commited on
Commit
0b5b7a9
1 Parent(s): c1caa78

Created README.md; Added essential model information.

Browse files
Files changed (1) hide show
  1. README.md +63 -0
README.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: ms
3
+ tags:
4
+ - malaysian-distilbert-small
5
+ license: mit
6
+ datasets:
7
+ - oscar
8
+ widget:
9
+ - text: "Hari ini adalah hari yang [MASK]!"
10
+ ---
11
+
12
+ ## Malaysian DistilBERT Small
13
+ Malaysian DistilBERT Small is a masked language model based on the [DistilBERT model](https://arxiv.org/abs/1910.01108). It was trained on the OSCAR dataset, specifically the `unshuffled_original_ms` subset.
14
+
15
+ The model was originally HuggingFace's pretrained [English DistilBERT model](https://huggingface.co/distilbert-base-uncased) and is later fine-tuned on the Malaysian dataset. It achieved a perplexity of 10.33 on the validation dataset (20% of the dataset). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).
16
+
17
+ Hugging Face's [Transformers]((https://huggingface.co/transformers)) library was used to train the model -- utilizing the base DistilBERT model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.
18
+
19
+ ## Model
20
+ | Model | #params | Arch. | Training/Validation data (text) |
21
+ |------------------------------|---------|------------------|----------------------------------------|
22
+ | `malaysian-distilbert-small` | 66M | DistilBERT Small | OSCAR `unshuffled_original_ms` Dataset |
23
+
24
+ ## Evaluation Results
25
+ The model was trained for 1 epoch and the following is the final result once the training ended.
26
+
27
+ | train loss | valid loss | perplexity | total time |
28
+ |------------|------------|------------|------------|
29
+ | 2.476 | 2.336 | 10.33 | 0:40:05 |
30
+
31
+ ## How to Use
32
+ ### As Masked Language Model
33
+ ```python
34
+ from transformers import pipeline
35
+
36
+ pretrained_name = "w11wo/malaysian-distilbert-small"
37
+
38
+ fill_mask = pipeline(
39
+ "fill-mask",
40
+ model=pretrained_name,
41
+ tokenizer=pretrained_name
42
+ )
43
+
44
+ fill_mask("Henry adalah seorang lelaki yang tinggal di [MASK].")
45
+ ```
46
+ ### Feature Extraction in PyTorch
47
+ ```python
48
+ from transformers import DistilBertModel, DistilBertTokenizerFast
49
+
50
+ pretrained_name = "w11wo/malaysian-distilbert-small"
51
+ model = DistilBertModel.from_pretrained(pretrained_name)
52
+ tokenizer = DistilBertTokenizerFast.from_pretrained(pretrained_name)
53
+
54
+ prompt = "Bolehkah anda [MASK] Bahasa Melayu?"
55
+ encoded_input = tokenizer(prompt, return_tensors='pt')
56
+ output = model(**encoded_input)
57
+ ```
58
+
59
+ ## Disclaimer
60
+ Do consider the biases which came from the OSCAR dataset that may be carried over into the results of this model.
61
+
62
+ ## Author
63
+ Malaysian DistilBERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.