Hugging Face's logo

yonigo
/
distilbert-base-multilingual-cased-pii

Token Classification
Transformers
Safetensors
distilbert
Generated from Trainer
Inference Endpoints
Model card Files Community
yonigo commited on
Commit
93a6a1f
•
1 Parent(s): bc1486f

Training in progress, step 1000

Browse files
Files changed (8) hide show
  1. README.md +115 -0
  2. config.json +143 -0
  3. model.safetensors +3 -0
  4. special_tokens_map.json +7 -0
  5. tokenizer.json +0 -0
  6. tokenizer_config.json +55 -0
  7. training_args.bin +3 -0
  8. vocab.txt +0 -0
README.md ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: distilbert-base-cased
4
+ tags:
5
+ - generated_from_trainer
6
+ metrics:
7
+ - precision
8
+ - recall
9
+ - f1
10
+ - accuracy
11
+ model-index:
12
+ - name: distilbert-base-cased-pii-en
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # distilbert-base-cased-pii-en
20
+
21
+ This model is a fine-tuned version of [distilbert-base-cased](https://huggingface.co/distilbert-base-cased) on an unknown dataset.
22
+ It achieves the following results on the evaluation set:
23
+ - Loss: 0.0635
24
+ - Bod F1: 0.9622
25
+ - Building F1: 0.9808
26
+ - Cardissuer F1: 0.0
27
+ - City F1: 0.9589
28
+ - Country F1: 0.9741
29
+ - Date F1: 0.9176
30
+ - Driverlicense F1: 0.9355
31
+ - Email F1: 0.9855
32
+ - Geocoord F1: 0.9814
33
+ - Givenname1 F1: 0.8496
34
+ - Givenname2 F1: 0.7261
35
+ - Idcard F1: 0.9306
36
+ - Ip F1: 0.9850
37
+ - Lastname1 F1: 0.7974
38
+ - Lastname2 F1: 0.6672
39
+ - Lastname3 F1: 0.6393
40
+ - Pass F1: 0.9005
41
+ - Passport F1: 0.9405
42
+ - Postcode F1: 0.9794
43
+ - Secaddress F1: 0.9745
44
+ - Sex F1: 0.9738
45
+ - Socialnumber F1: 0.9437
46
+ - State F1: 0.9794
47
+ - Street F1: 0.9651
48
+ - Tel F1: 0.9674
49
+ - Time F1: 0.9638
50
+ - Title F1: 0.9587
51
+ - Username F1: 0.9469
52
+ - Precision: 0.9318
53
+ - Recall: 0.9460
54
+ - F1: 0.9388
55
+ - Accuracy: 0.9913
56
+
57
+ ## Model description
58
+
59
+ More information needed
60
+
61
+ ## Intended uses & limitations
62
+
63
+ More information needed
64
+
65
+ ## Training and evaluation data
66
+
67
+ More information needed
68
+
69
+ ## Training procedure
70
+
71
+ ### Training hyperparameters
72
+
73
+ The following hyperparameters were used during training:
74
+ - learning_rate: 2e-05
75
+ - train_batch_size: 64
76
+ - eval_batch_size: 128
77
+ - seed: 42
78
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
79
+ - lr_scheduler_type: cosine
80
+ - lr_scheduler_warmup_ratio: 0.2
81
+ - lr_scheduler_warmup_steps: 3000
82
+ - training_steps: 20000
83
+
84
+ ### Training results
85
+
86
+ | Training Loss | Epoch | Step | Validation Loss | Bod F1 | Building F1 | Cardissuer F1 | City F1 | Country F1 | Date F1 | Driverlicense F1 | Email F1 | Geocoord F1 | Givenname1 F1 | Givenname2 F1 | Idcard F1 | Ip F1 | Lastname1 F1 | Lastname2 F1 | Lastname3 F1 | Pass F1 | Passport F1 | Postcode F1 | Secaddress F1 | Sex F1 | Socialnumber F1 | State F1 | Street F1 | Tel F1 | Time F1 | Title F1 | Username F1 | Precision | Recall | F1 | Accuracy |
87
+ |:-------------:|:-------:|:-----:|:---------------:|:------:|:-----------:|:-------------:|:-------:|:----------:|:-------:|:----------------:|:--------:|:-----------:|:-------------:|:-------------:|:---------:|:------:|:------------:|:------------:|:------------:|:-------:|:-----------:|:-----------:|:-------------:|:------:|:---------------:|:--------:|:---------:|:------:|:-------:|:--------:|:-----------:|:---------:|:------:|:------:|:--------:|
88
+ | 0.23 | 2.1368 | 1000 | 0.1111 | 0.8804 | 0.9218 | 0.0 | 0.5854 | 0.8890 | 0.8009 | 0.5896 | 0.9469 | 0.7002 | 0.4149 | 0.0 | 0.5740 | 0.9056 | 0.5230 | 0.0 | 0.0 | 0.6551 | 0.6301 | 0.7819 | 0.5871 | 0.8396 | 0.7137 | 0.8771 | 0.6466 | 0.8567 | 0.9185 | 0.7411 | 0.8202 | 0.7127 | 0.7640 | 0.7374 | 0.9723 |
89
+ | 0.0687 | 4.2735 | 2000 | 0.0541 | 0.9365 | 0.9688 | 0.0 | 0.9115 | 0.9434 | 0.8841 | 0.8530 | 0.9768 | 0.9698 | 0.7276 | 0.3073 | 0.8720 | 0.9701 | 0.6430 | 0.1854 | 0.0 | 0.8169 | 0.8857 | 0.9638 | 0.9617 | 0.9312 | 0.8927 | 0.9418 | 0.9231 | 0.9438 | 0.9461 | 0.9232 | 0.9138 | 0.8698 | 0.9020 | 0.8856 | 0.9862 |
90
+ | 0.0452 | 6.4103 | 3000 | 0.0469 | 0.9550 | 0.9753 | 0.0 | 0.9404 | 0.9620 | 0.9014 | 0.9004 | 0.9831 | 0.9721 | 0.7664 | 0.4606 | 0.9099 | 0.97 | 0.6838 | 0.3818 | 0.0 | 0.8611 | 0.9079 | 0.9757 | 0.9715 | 0.9661 | 0.9197 | 0.9674 | 0.9435 | 0.9448 | 0.96 | 0.9354 | 0.9357 | 0.8975 | 0.9212 | 0.9092 | 0.9883 |
91
+ | 0.0311 | 8.5470 | 4000 | 0.0434 | 0.9581 | 0.9753 | 0.0 | 0.9481 | 0.9641 | 0.8973 | 0.9217 | 0.9833 | 0.9745 | 0.8085 | 0.6085 | 0.9232 | 0.9781 | 0.7519 | 0.4891 | 0.0849 | 0.8685 | 0.9280 | 0.9771 | 0.9704 | 0.9659 | 0.9278 | 0.9729 | 0.9549 | 0.9563 | 0.9594 | 0.9564 | 0.9407 | 0.9117 | 0.9344 | 0.9229 | 0.9897 |
92
+ | 0.023 | 10.6838 | 5000 | 0.0416 | 0.9578 | 0.9778 | 0.0 | 0.9503 | 0.9639 | 0.9043 | 0.9245 | 0.9804 | 0.9767 | 0.8277 | 0.6607 | 0.9031 | 0.9836 | 0.7790 | 0.5779 | 0.25 | 0.8791 | 0.9276 | 0.9767 | 0.9660 | 0.9708 | 0.9353 | 0.9749 | 0.9610 | 0.9681 | 0.9611 | 0.9545 | 0.9382 | 0.9185 | 0.9363 | 0.9273 | 0.9903 |
93
+ | 0.016 | 12.8205 | 6000 | 0.0436 | 0.9607 | 0.9778 | 0.0 | 0.9516 | 0.9698 | 0.9027 | 0.9274 | 0.9781 | 0.9677 | 0.8472 | 0.6913 | 0.9259 | 0.9877 | 0.7857 | 0.5925 | 0.3819 | 0.8853 | 0.9377 | 0.9731 | 0.9688 | 0.9752 | 0.9461 | 0.9756 | 0.9594 | 0.9613 | 0.9568 | 0.9545 | 0.9447 | 0.9247 | 0.9396 | 0.9321 | 0.9907 |
94
+ | 0.012 | 14.9573 | 7000 | 0.0447 | 0.9598 | 0.9808 | 0.0 | 0.9558 | 0.9737 | 0.9070 | 0.9339 | 0.9719 | 0.9654 | 0.8482 | 0.7009 | 0.9280 | 0.9859 | 0.7879 | 0.6027 | 0.4729 | 0.8909 | 0.9413 | 0.9781 | 0.9750 | 0.9740 | 0.9431 | 0.9778 | 0.9624 | 0.9623 | 0.9665 | 0.9555 | 0.9403 | 0.9275 | 0.9413 | 0.9344 | 0.9911 |
95
+ | 0.0086 | 17.0940 | 8000 | 0.0487 | 0.9584 | 0.9814 | 0.0 | 0.9529 | 0.9733 | 0.9114 | 0.9350 | 0.9857 | 0.9767 | 0.8493 | 0.7059 | 0.9306 | 0.9829 | 0.7918 | 0.6216 | 0.5292 | 0.8934 | 0.9330 | 0.9790 | 0.9766 | 0.9750 | 0.9395 | 0.9747 | 0.9624 | 0.9605 | 0.9631 | 0.9580 | 0.9399 | 0.9282 | 0.9423 | 0.9352 | 0.9908 |
96
+ | 0.0062 | 19.2308 | 9000 | 0.0509 | 0.9596 | 0.9795 | 0.0 | 0.9594 | 0.9708 | 0.9128 | 0.9299 | 0.9872 | 0.9837 | 0.8491 | 0.7188 | 0.9270 | 0.9859 | 0.7923 | 0.6468 | 0.5371 | 0.8919 | 0.9369 | 0.9783 | 0.9756 | 0.9749 | 0.9382 | 0.9778 | 0.9650 | 0.9647 | 0.9653 | 0.9559 | 0.9461 | 0.9337 | 0.9394 | 0.9365 | 0.9911 |
97
+ | 0.0045 | 21.3675 | 10000 | 0.0548 | 0.9559 | 0.9774 | 0.0 | 0.9524 | 0.9720 | 0.9080 | 0.9348 | 0.9827 | 0.9814 | 0.8446 | 0.7117 | 0.9271 | 0.9800 | 0.7977 | 0.6428 | 0.5266 | 0.8964 | 0.9351 | 0.9754 | 0.9716 | 0.9728 | 0.9478 | 0.9757 | 0.9584 | 0.9698 | 0.9587 | 0.9548 | 0.9423 | 0.9242 | 0.9457 | 0.9348 | 0.9907 |
98
+ | 0.0036 | 23.5043 | 11000 | 0.0560 | 0.9594 | 0.9781 | 0.0 | 0.9575 | 0.9720 | 0.9121 | 0.9367 | 0.9814 | 0.9814 | 0.8504 | 0.7209 | 0.9317 | 0.9807 | 0.7922 | 0.6507 | 0.5918 | 0.8864 | 0.9380 | 0.9769 | 0.9722 | 0.9745 | 0.9399 | 0.9771 | 0.9628 | 0.9675 | 0.9618 | 0.9581 | 0.9446 | 0.9288 | 0.9434 | 0.9361 | 0.9910 |
99
+ | 0.0026 | 25.6410 | 12000 | 0.0576 | 0.9596 | 0.9798 | 0.0 | 0.9575 | 0.9732 | 0.9130 | 0.9308 | 0.9831 | 0.9791 | 0.8471 | 0.7104 | 0.9268 | 0.9836 | 0.7967 | 0.6563 | 0.6222 | 0.8982 | 0.9345 | 0.9771 | 0.9739 | 0.9733 | 0.9402 | 0.9771 | 0.9673 | 0.9656 | 0.9642 | 0.9576 | 0.9480 | 0.9288 | 0.9446 | 0.9366 | 0.9910 |
100
+ | 0.002 | 27.7778 | 13000 | 0.0608 | 0.9555 | 0.9796 | 0.0 | 0.9561 | 0.9717 | 0.9045 | 0.9319 | 0.9848 | 0.9791 | 0.8488 | 0.7157 | 0.9268 | 0.9852 | 0.7909 | 0.6580 | 0.6039 | 0.8900 | 0.9360 | 0.9802 | 0.9717 | 0.9750 | 0.9361 | 0.9778 | 0.9646 | 0.9683 | 0.9615 | 0.9565 | 0.9465 | 0.9279 | 0.9433 | 0.9355 | 0.9909 |
101
+ | 0.0016 | 29.9145 | 14000 | 0.0601 | 0.9573 | 0.9801 | 0.0 | 0.9589 | 0.9722 | 0.9135 | 0.9353 | 0.9848 | 0.9837 | 0.8499 | 0.7202 | 0.9316 | 0.9871 | 0.7942 | 0.6677 | 0.6432 | 0.9017 | 0.9402 | 0.9799 | 0.9744 | 0.9740 | 0.9455 | 0.9783 | 0.9631 | 0.9719 | 0.9645 | 0.9600 | 0.9482 | 0.9331 | 0.9443 | 0.9387 | 0.9913 |
102
+ | 0.0013 | 32.0513 | 15000 | 0.0613 | 0.9606 | 0.9798 | 0.0 | 0.9571 | 0.9739 | 0.9155 | 0.9365 | 0.9840 | 0.9791 | 0.8466 | 0.7221 | 0.9321 | 0.9861 | 0.7948 | 0.6622 | 0.6281 | 0.9017 | 0.9407 | 0.9787 | 0.9739 | 0.9740 | 0.9475 | 0.9774 | 0.9632 | 0.9693 | 0.9644 | 0.9594 | 0.9469 | 0.9313 | 0.9457 | 0.9384 | 0.9912 |
103
+ | 0.001 | 34.1880 | 16000 | 0.0639 | 0.9611 | 0.9808 | 0.0 | 0.9601 | 0.9729 | 0.9149 | 0.9337 | 0.9867 | 0.9814 | 0.8483 | 0.7269 | 0.9307 | 0.9838 | 0.7956 | 0.6627 | 0.6154 | 0.9025 | 0.9397 | 0.9797 | 0.9733 | 0.9738 | 0.9383 | 0.9776 | 0.9636 | 0.9674 | 0.9637 | 0.9576 | 0.9480 | 0.9304 | 0.9457 | 0.9380 | 0.9911 |
104
+ | 0.0009 | 36.3248 | 17000 | 0.0621 | 0.9622 | 0.9811 | 0.0 | 0.9604 | 0.9741 | 0.9156 | 0.9359 | 0.9855 | 0.9814 | 0.8510 | 0.7273 | 0.9319 | 0.9859 | 0.7991 | 0.6646 | 0.6413 | 0.8999 | 0.9393 | 0.9789 | 0.9739 | 0.9740 | 0.9427 | 0.9789 | 0.9653 | 0.9687 | 0.9637 | 0.9597 | 0.9460 | 0.9324 | 0.9456 | 0.9390 | 0.9913 |
105
+ | 0.0008 | 38.4615 | 18000 | 0.0631 | 0.9620 | 0.9801 | 0.0 | 0.9582 | 0.9744 | 0.9190 | 0.9350 | 0.9853 | 0.9814 | 0.8514 | 0.7253 | 0.9298 | 0.9848 | 0.7992 | 0.6677 | 0.6434 | 0.8992 | 0.9401 | 0.9797 | 0.9739 | 0.9731 | 0.9421 | 0.9789 | 0.9653 | 0.9681 | 0.9643 | 0.9592 | 0.9462 | 0.9319 | 0.9457 | 0.9388 | 0.9913 |
106
+ | 0.0007 | 40.5983 | 19000 | 0.0633 | 0.9615 | 0.9806 | 0.0 | 0.9589 | 0.9741 | 0.9170 | 0.9358 | 0.9861 | 0.9814 | 0.8501 | 0.7268 | 0.9316 | 0.9857 | 0.7973 | 0.6662 | 0.6393 | 0.9002 | 0.9405 | 0.9797 | 0.9745 | 0.9738 | 0.9442 | 0.9796 | 0.9658 | 0.9679 | 0.9644 | 0.9586 | 0.9469 | 0.9323 | 0.9459 | 0.9391 | 0.9913 |
107
+ | 0.0008 | 42.7350 | 20000 | 0.0635 | 0.9622 | 0.9808 | 0.0 | 0.9589 | 0.9741 | 0.9176 | 0.9355 | 0.9855 | 0.9814 | 0.8496 | 0.7261 | 0.9306 | 0.9850 | 0.7974 | 0.6672 | 0.6393 | 0.9005 | 0.9405 | 0.9794 | 0.9745 | 0.9738 | 0.9437 | 0.9794 | 0.9651 | 0.9674 | 0.9638 | 0.9587 | 0.9469 | 0.9318 | 0.9460 | 0.9388 | 0.9913 |
108
+
109
+
110
+ ### Framework versions
111
+
112
+ - Transformers 4.41.2
113
+ - Pytorch 2.3.1+cu121
114
+ - Datasets 2.20.0
115
+ - Tokenizers 0.19.1
config.json ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "distilbert-base-multilingual-cased",
3
+ "activation": "gelu",
4
+ "architectures": [
5
+ "DistilBertForTokenClassification"
6
+ ],
7
+ "attention_dropout": 0.1,
8
+ "dim": 768,
9
+ "dropout": 0.1,
10
+ "hidden_dim": 3072,
11
+ "id2label": {
12
+ "0": "I-BOD",
13
+ "1": "I-BUILDING",
14
+ "2": "I-CARDISSUER",
15
+ "3": "I-CITY",
16
+ "4": "I-COUNTRY",
17
+ "5": "I-DATE",
18
+ "6": "I-DRIVERLICENSE",
19
+ "7": "I-EMAIL",
20
+ "8": "I-GEOCOORD",
21
+ "9": "I-GIVENNAME1",
22
+ "10": "I-GIVENNAME2",
23
+ "11": "I-IDCARD",
24
+ "12": "I-IP",
25
+ "13": "I-LASTNAME1",
26
+ "14": "I-LASTNAME2",
27
+ "15": "I-LASTNAME3",
28
+ "16": "I-PASS",
29
+ "17": "I-PASSPORT",
30
+ "18": "I-POSTCODE",
31
+ "19": "I-SECADDRESS",
32
+ "20": "I-SEX",
33
+ "21": "I-SOCIALNUMBER",
34
+ "22": "I-STATE",
35
+ "23": "I-STREET",
36
+ "24": "I-TEL",
37
+ "25": "I-TIME",
38
+ "26": "I-TITLE",
39
+ "27": "I-USERNAME",
40
+ "28": "B-BOD",
41
+ "29": "B-BUILDING",
42
+ "30": "B-CARDISSUER",
43
+ "31": "B-CITY",
44
+ "32": "B-COUNTRY",
45
+ "33": "B-DATE",
46
+ "34": "B-DRIVERLICENSE",
47
+ "35": "B-EMAIL",
48
+ "36": "B-GEOCOORD",
49
+ "37": "B-GIVENNAME1",
50
+ "38": "B-GIVENNAME2",
51
+ "39": "B-IDCARD",
52
+ "40": "B-IP",
53
+ "41": "B-LASTNAME1",
54
+ "42": "B-LASTNAME2",
55
+ "43": "B-LASTNAME3",
56
+ "44": "B-PASS",
57
+ "45": "B-PASSPORT",
58
+ "46": "B-POSTCODE",
59
+ "47": "B-SECADDRESS",
60
+ "48": "B-SEX",
61
+ "49": "B-SOCIALNUMBER",
62
+ "50": "B-STATE",
63
+ "51": "B-STREET",
64
+ "52": "B-TEL",
65
+ "53": "B-TIME",
66
+ "54": "B-TITLE",
67
+ "55": "B-USERNAME",
68
+ "56": "O"
69
+ },
70
+ "initializer_range": 0.02,
71
+ "label2id": {
72
+ "LABEL_0": 0,
73
+ "LABEL_1": 1,
74
+ "LABEL_10": 10,
75
+ "LABEL_11": 11,
76
+ "LABEL_12": 12,
77
+ "LABEL_13": 13,
78
+ "LABEL_14": 14,
79
+ "LABEL_15": 15,
80
+ "LABEL_16": 16,
81
+ "LABEL_17": 17,
82
+ "LABEL_18": 18,
83
+ "LABEL_19": 19,
84
+ "LABEL_2": 2,
85
+ "LABEL_20": 20,
86
+ "LABEL_21": 21,
87
+ "LABEL_22": 22,
88
+ "LABEL_23": 23,
89
+ "LABEL_24": 24,
90
+ "LABEL_25": 25,
91
+ "LABEL_26": 26,
92
+ "LABEL_27": 27,
93
+ "LABEL_28": 28,
94
+ "LABEL_29": 29,
95
+ "LABEL_3": 3,
96
+ "LABEL_30": 30,
97
+ "LABEL_31": 31,
98
+ "LABEL_32": 32,
99
+ "LABEL_33": 33,
100
+ "LABEL_34": 34,
101
+ "LABEL_35": 35,
102
+ "LABEL_36": 36,
103
+ "LABEL_37": 37,
104
+ "LABEL_38": 38,
105
+ "LABEL_39": 39,
106
+ "LABEL_4": 4,
107
+ "LABEL_40": 40,
108
+ "LABEL_41": 41,
109
+ "LABEL_42": 42,
110
+ "LABEL_43": 43,
111
+ "LABEL_44": 44,
112
+ "LABEL_45": 45,
113
+ "LABEL_46": 46,
114
+ "LABEL_47": 47,
115
+ "LABEL_48": 48,
116
+ "LABEL_49": 49,
117
+ "LABEL_5": 5,
118
+ "LABEL_50": 50,
119
+ "LABEL_51": 51,
120
+ "LABEL_52": 52,
121
+ "LABEL_53": 53,
122
+ "LABEL_54": 54,
123
+ "LABEL_55": 55,
124
+ "LABEL_56": 56,
125
+ "LABEL_6": 6,
126
+ "LABEL_7": 7,
127
+ "LABEL_8": 8,
128
+ "LABEL_9": 9
129
+ },
130
+ "max_position_embeddings": 512,
131
+ "model_type": "distilbert",
132
+ "n_heads": 12,
133
+ "n_layers": 6,
134
+ "output_past": true,
135
+ "pad_token_id": 0,
136
+ "qa_dropout": 0.1,
137
+ "seq_classif_dropout": 0.2,
138
+ "sinusoidal_pos_embds": false,
139
+ "tie_weights_": true,
140
+ "torch_dtype": "float32",
141
+ "transformers_version": "4.41.2",
142
+ "vocab_size": 119547
143
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f49bdab483fed57b2c26d4cec3876636fe61dd6f907cef1346c9ab793a0e213
3
+ size 539124012
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "pad_token": "[PAD]",
5
+ "sep_token": "[SEP]",
6
+ "unk_token": "[UNK]"
7
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "100": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "101": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "102": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "103": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_lower_case": false,
47
+ "mask_token": "[MASK]",
48
+ "model_max_length": 512,
49
+ "pad_token": "[PAD]",
50
+ "sep_token": "[SEP]",
51
+ "strip_accents": null,
52
+ "tokenize_chinese_chars": true,
53
+ "tokenizer_class": "DistilBertTokenizer",
54
+ "unk_token": "[UNK]"
55
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06f96619c3ed5a76a444b39332b2b28724911e2463eb2b02a1bfd9bee2c9bc89
3
+ size 5176
vocab.txt ADDED
The diff for this file is too large to render. See raw diff