--- library_name: transformers license: mit base_model: roberta-base tags: - generated_from_trainer model-index: - name: BERiT_2.0 results: [] --- # BERiT_2.0 This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 4.3491 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0008131878854370431 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 75 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-------:|:-----:|:---------------:| | 6.7675 | 0.3873 | 500 | 6.5915 | | 6.5119 | 0.7746 | 1000 | 6.3912 | | 6.378 | 1.1619 | 1500 | 6.3244 | | 6.3103 | 1.5492 | 2000 | 6.2670 | | 6.3097 | 1.9365 | 2500 | 6.2378 | | 6.2799 | 2.3238 | 3000 | 6.2406 | | 6.2717 | 2.7111 | 3500 | 6.2725 | | 6.2694 | 3.0984 | 4000 | 6.2770 | | 6.2595 | 3.4857 | 4500 | 6.2695 | | 6.2386 | 3.8730 | 5000 | 6.2952 | | 6.2393 | 4.2603 | 5500 | 6.2538 | | 6.2442 | 4.6476 | 6000 | 6.2138 | | 6.2322 | 5.0349 | 6500 | 6.2639 | | 6.2028 | 5.4222 | 7000 | 6.3072 | | 6.2306 | 5.8095 | 7500 | 6.2069 | | 6.2027 | 6.1967 | 8000 | 6.2065 | | 6.2 | 6.5840 | 8500 | 6.2331 | | 6.1877 | 6.9713 | 9000 | 6.2548 | | 6.181 | 7.3586 | 9500 | 6.2235 | | 6.1608 | 7.7459 | 10000 | 6.2316 | | 6.1933 | 8.1332 | 10500 | 6.2513 | | 6.187 | 8.5205 | 11000 | 6.1930 | | 6.1734 | 8.9078 | 11500 | 6.2105 | | 6.167 | 9.2951 | 12000 | 6.2032 | | 6.148 | 9.6824 | 12500 | 6.2129 | | 6.1411 | 10.0697 | 13000 | 6.2228 | | 6.131 | 10.4570 | 13500 | 6.2537 | | 6.1321 | 10.8443 | 14000 | 6.2499 | | 6.1412 | 11.2316 | 14500 | 6.2424 | | 6.1389 | 11.6189 | 15000 | 6.1810 | | 6.1265 | 12.0062 | 15500 | 6.1974 | | 6.1345 | 12.3935 | 16000 | 6.2404 | | 6.1072 | 12.7808 | 16500 | 6.1847 | | 6.1213 | 13.1681 | 17000 | 6.1444 | | 6.1016 | 13.5554 | 17500 | 6.1880 | | 6.1306 | 13.9427 | 18000 | 6.1328 | | 6.108 | 14.3300 | 18500 | 6.1761 | | 6.1137 | 14.7173 | 19000 | 6.1504 | | 6.0994 | 15.1046 | 19500 | 6.1977 | | 6.1026 | 15.4919 | 20000 | 6.1830 | | 6.0999 | 15.8792 | 20500 | 6.1811 | | 6.0978 | 16.2665 | 21000 | 6.1263 | | 6.0877 | 16.6538 | 21500 | 6.1645 | | 6.0917 | 17.0411 | 22000 | 6.1742 | | 6.0973 | 17.4284 | 22500 | 6.1636 | | 6.1098 | 17.8156 | 23000 | 6.2017 | | 6.0828 | 18.2029 | 23500 | 6.1007 | | 6.0999 | 18.5902 | 24000 | 6.0879 | | 6.0953 | 18.9775 | 24500 | 6.1521 | | 6.079 | 19.3648 | 25000 | 6.1391 | | 6.0682 | 19.7521 | 25500 | 6.0962 | | 6.058 | 20.1394 | 26000 | 6.0719 | | 6.0643 | 20.5267 | 26500 | 6.1114 | | 6.0498 | 20.9140 | 27000 | 6.1111 | | 6.0665 | 21.3013 | 27500 | 6.1200 | | 6.0825 | 21.6886 | 28000 | 6.0961 | | 6.0369 | 22.0759 | 28500 | 6.1578 | | 6.0512 | 22.4632 | 29000 | 6.0876 | | 6.026 | 22.8505 | 29500 | 6.1211 | | 6.0558 | 23.2378 | 30000 | 6.0837 | | 6.0466 | 23.6251 | 30500 | 6.0552 | | 6.0202 | 24.0124 | 31000 | 6.0906 | | 6.0019 | 24.3997 | 31500 | 6.0580 | | 6.0352 | 24.7870 | 32000 | 6.0521 | | 5.9983 | 25.1743 | 32500 | 6.0701 | | 6.0367 | 25.5616 | 33000 | 6.0859 | | 6.0183 | 25.9489 | 33500 | 6.1353 | | 5.9726 | 26.3362 | 34000 | 6.0918 | | 5.982 | 26.7235 | 34500 | 6.0434 | | 6.0261 | 27.1108 | 35000 | 6.0038 | | 5.9818 | 27.4981 | 35500 | 6.0328 | | 5.9659 | 27.8854 | 36000 | 6.0672 | | 5.9835 | 28.2727 | 36500 | 6.0334 | | 5.98 | 28.6600 | 37000 | 6.0673 | | 5.9756 | 29.0473 | 37500 | 5.9969 | | 5.979 | 29.4345 | 38000 | 6.0067 | | 5.9728 | 29.8218 | 38500 | 6.0297 | | 5.9596 | 30.2091 | 39000 | 5.9682 | | 5.9866 | 30.5964 | 39500 | 6.0026 | | 5.975 | 30.9837 | 40000 | 5.9987 | | 5.9678 | 31.3710 | 40500 | 5.9919 | | 5.9676 | 31.7583 | 41000 | 5.9807 | | 5.9294 | 32.1456 | 41500 | 5.9629 | | 5.9465 | 32.5329 | 42000 | 5.9608 | | 5.9554 | 32.9202 | 42500 | 5.9522 | | 5.9042 | 33.3075 | 43000 | 5.9674 | | 5.9359 | 33.6948 | 43500 | 5.9959 | | 5.9339 | 34.0821 | 44000 | 5.9914 | | 5.9215 | 34.4694 | 44500 | 5.9134 | | 5.8901 | 34.8567 | 45000 | 5.9219 | | 5.9134 | 35.2440 | 45500 | 5.9305 | | 5.9086 | 35.6313 | 46000 | 5.9433 | | 5.9051 | 36.0186 | 46500 | 5.8672 | | 5.8991 | 36.4059 | 47000 | 5.8599 | | 5.8789 | 36.7932 | 47500 | 5.8966 | | 5.892 | 37.1805 | 48000 | 5.8956 | | 5.8591 | 37.5678 | 48500 | 5.8597 | | 5.8855 | 37.9551 | 49000 | 5.8776 | | 5.856 | 38.3424 | 49500 | 5.9281 | | 5.838 | 38.7297 | 50000 | 5.8091 | | 5.8556 | 39.1170 | 50500 | 5.7789 | | 5.8527 | 39.5043 | 51000 | 5.7454 | | 5.8172 | 39.8916 | 51500 | 5.7894 | | 5.8249 | 40.2789 | 52000 | 5.7938 | | 5.809 | 40.6662 | 52500 | 5.7688 | | 5.8152 | 41.0534 | 53000 | 5.7233 | | 5.7961 | 41.4407 | 53500 | 5.6899 | | 5.7767 | 41.8280 | 54000 | 5.7165 | | 5.7582 | 42.2153 | 54500 | 5.7639 | | 5.7893 | 42.6026 | 55000 | 5.6803 | | 5.7365 | 42.9899 | 55500 | 5.6790 | | 5.7365 | 43.3772 | 56000 | 5.6499 | | 5.7569 | 43.7645 | 56500 | 5.6215 | | 5.7321 | 44.1518 | 57000 | 5.6148 | | 5.7186 | 44.5391 | 57500 | 5.5600 | | 5.701 | 44.9264 | 58000 | 5.5373 | | 5.7009 | 45.3137 | 58500 | 5.5664 | | 5.7 | 45.7010 | 59000 | 5.5163 | | 5.677 | 46.0883 | 59500 | 5.4210 | | 5.6673 | 46.4756 | 60000 | 5.3903 | | 5.6297 | 46.8629 | 60500 | 5.3785 | | 5.6222 | 47.2502 | 61000 | 5.3162 | | 5.6181 | 47.6375 | 61500 | 5.2644 | | 5.5784 | 48.0248 | 62000 | 5.2543 | | 5.5799 | 48.4121 | 62500 | 5.2034 | | 5.5509 | 48.7994 | 63000 | 5.1793 | | 5.5665 | 49.1867 | 63500 | 5.1611 | | 5.5418 | 49.5740 | 64000 | 5.1162 | | 5.5094 | 49.9613 | 64500 | 5.0998 | | 5.4983 | 50.3486 | 65000 | 5.0847 | | 5.488 | 50.7359 | 65500 | 5.0962 | | 5.4842 | 51.1232 | 66000 | 5.0385 | | 5.4456 | 51.5105 | 66500 | 5.0509 | | 5.4167 | 51.8978 | 67000 | 4.9671 | | 5.4094 | 52.2851 | 67500 | 4.9199 | | 5.4044 | 52.6723 | 68000 | 4.9520 | | 5.3853 | 53.0596 | 68500 | 4.9233 | | 5.388 | 53.4469 | 69000 | 4.8602 | | 5.3735 | 53.8342 | 69500 | 4.8504 | | 5.3755 | 54.2215 | 70000 | 4.8019 | | 5.3352 | 54.6088 | 70500 | 4.8239 | | 5.3469 | 54.9961 | 71000 | 4.8391 | | 5.3198 | 55.3834 | 71500 | 4.7593 | | 5.2901 | 55.7707 | 72000 | 4.7801 | | 5.2921 | 56.1580 | 72500 | 4.7699 | | 5.2942 | 56.5453 | 73000 | 4.7290 | | 5.2615 | 56.9326 | 73500 | 4.7631 | | 5.2719 | 57.3199 | 74000 | 4.7217 | | 5.2724 | 57.7072 | 74500 | 4.7176 | | 5.256 | 58.0945 | 75000 | 4.6826 | | 5.2318 | 58.4818 | 75500 | 4.6467 | | 5.2152 | 58.8691 | 76000 | 4.6603 | | 5.2301 | 59.2564 | 76500 | 4.6464 | | 5.2057 | 59.6437 | 77000 | 4.7027 | | 5.2073 | 60.0310 | 77500 | 4.6243 | | 5.1845 | 60.4183 | 78000 | 4.6194 | | 5.1919 | 60.8056 | 78500 | 4.5877 | | 5.1805 | 61.1929 | 79000 | 4.5641 | | 5.1723 | 61.5802 | 79500 | 4.5644 | | 5.1805 | 61.9675 | 80000 | 4.5706 | | 5.1738 | 62.3548 | 80500 | 4.5631 | | 5.1586 | 62.7421 | 81000 | 4.5652 | | 5.1666 | 63.1294 | 81500 | 4.5658 | | 5.1412 | 63.5167 | 82000 | 4.5540 | | 5.1336 | 63.9040 | 82500 | 4.4969 | | 5.1561 | 64.2912 | 83000 | 4.5313 | | 5.1223 | 64.6785 | 83500 | 4.5508 | | 5.1426 | 65.0658 | 84000 | 4.6013 | | 5.1124 | 65.4531 | 84500 | 4.4723 | | 5.1187 | 65.8404 | 85000 | 4.5116 | | 5.1162 | 66.2277 | 85500 | 4.4991 | | 5.0953 | 66.6150 | 86000 | 4.4395 | | 5.1166 | 67.0023 | 86500 | 4.4828 | | 5.0869 | 67.3896 | 87000 | 4.4650 | | 5.0936 | 67.7769 | 87500 | 4.4560 | | 5.0963 | 68.1642 | 88000 | 4.4724 | | 5.1117 | 68.5515 | 88500 | 4.4756 | | 5.0679 | 68.9388 | 89000 | 4.4347 | | 5.0803 | 69.3261 | 89500 | 4.4484 | | 5.0786 | 69.7134 | 90000 | 4.4027 | | 5.0483 | 70.1007 | 90500 | 4.3975 | | 5.0678 | 70.4880 | 91000 | 4.4025 | | 5.053 | 70.8753 | 91500 | 4.4227 | | 5.068 | 71.2626 | 92000 | 4.4204 | | 5.0643 | 71.6499 | 92500 | 4.3634 | | 5.0812 | 72.0372 | 93000 | 4.4053 | | 5.0543 | 72.4245 | 93500 | 4.4241 | | 5.0564 | 72.8118 | 94000 | 4.3448 | | 5.0486 | 73.1991 | 94500 | 4.4480 | | 5.0508 | 73.5864 | 95000 | 4.3254 | | 5.0298 | 73.9737 | 95500 | 4.4201 | | 5.0585 | 74.3610 | 96000 | 4.4283 | | 5.0454 | 74.7483 | 96500 | 4.3491 | ### Framework versions - Transformers 4.47.1 - Pytorch 2.5.1+cu121 - Datasets 3.2.0 - Tokenizers 0.21.0