--- library_name: peft license: mit base_model: FacebookAI/xlm-roberta-large tags: - generated_from_trainer datasets: - conll2002 metrics: - precision - recall - f1 - accuracy model-index: - name: roberta-large-ner-qlorafinetune-runs-colab results: [] --- # roberta-large-ner-qlorafinetune-runs-colab This model is a fine-tuned version of [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) on the conll2002 dataset. It achieves the following results on the evaluation set: - Loss: 0.0818 - Precision: 0.8756 - Recall: 0.8830 - F1: 0.8793 - Accuracy: 0.9821 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0004 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - training_steps: 1820 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy | |:-------------:|:------:|:----:|:---------------:|:---------:|:------:|:------:|:--------:| | 1.1988 | 0.0766 | 20 | 0.5183 | 0.0 | 0.0 | 0.0 | 0.8570 | | 0.346 | 0.1533 | 40 | 0.3115 | 0.3631 | 0.4614 | 0.4064 | 0.9159 | | 0.2248 | 0.2299 | 60 | 0.1851 | 0.6607 | 0.6994 | 0.6795 | 0.9504 | | 0.1411 | 0.3065 | 80 | 0.1653 | 0.6161 | 0.6687 | 0.6413 | 0.9568 | | 0.1164 | 0.3831 | 100 | 0.1017 | 0.7897 | 0.8196 | 0.8044 | 0.9744 | | 0.0937 | 0.4598 | 120 | 0.1144 | 0.7908 | 0.8224 | 0.8063 | 0.9732 | | 0.0846 | 0.5364 | 140 | 0.0809 | 0.8454 | 0.8667 | 0.8559 | 0.9804 | | 0.0852 | 0.6130 | 160 | 0.1207 | 0.7424 | 0.7727 | 0.7573 | 0.9702 | | 0.0798 | 0.6897 | 180 | 0.0878 | 0.8264 | 0.8564 | 0.8411 | 0.9775 | | 0.0668 | 0.7663 | 200 | 0.0832 | 0.8362 | 0.8539 | 0.8449 | 0.9781 | | 0.062 | 0.8429 | 220 | 0.0766 | 0.8411 | 0.8564 | 0.8487 | 0.9796 | | 0.0624 | 0.9195 | 240 | 0.0750 | 0.8478 | 0.8624 | 0.8550 | 0.9792 | | 0.0577 | 0.9962 | 260 | 0.1152 | 0.7684 | 0.7934 | 0.7807 | 0.9707 | | 0.0519 | 1.0728 | 280 | 0.0889 | 0.8155 | 0.8440 | 0.8295 | 0.9766 | | 0.0516 | 1.1494 | 300 | 0.0855 | 0.8339 | 0.8614 | 0.8474 | 0.9780 | | 0.0584 | 1.2261 | 320 | 0.0794 | 0.8418 | 0.8633 | 0.8524 | 0.9789 | | 0.0518 | 1.3027 | 340 | 0.0879 | 0.8240 | 0.8458 | 0.8348 | 0.9766 | | 0.06 | 1.3793 | 360 | 0.0922 | 0.8100 | 0.8277 | 0.8187 | 0.9756 | | 0.0556 | 1.4559 | 380 | 0.1141 | 0.7741 | 0.7927 | 0.7833 | 0.9704 | | 0.0524 | 1.5326 | 400 | 0.0863 | 0.8247 | 0.8412 | 0.8329 | 0.9765 | | 0.0526 | 1.6092 | 420 | 0.0779 | 0.8391 | 0.8518 | 0.8454 | 0.9781 | | 0.0699 | 1.6858 | 440 | 0.0903 | 0.8099 | 0.8290 | 0.8193 | 0.9750 | | 0.0626 | 1.7625 | 460 | 0.0746 | 0.8569 | 0.8571 | 0.8570 | 0.9794 | | 0.05 | 1.8391 | 480 | 0.0725 | 0.8443 | 0.8709 | 0.8574 | 0.9798 | | 0.0521 | 1.9157 | 500 | 0.0694 | 0.8558 | 0.8771 | 0.8663 | 0.9813 | | 0.0463 | 1.9923 | 520 | 0.0798 | 0.8521 | 0.8672 | 0.8596 | 0.9796 | | 0.0748 | 2.0690 | 540 | 0.1192 | 0.7639 | 0.7868 | 0.7752 | 0.9717 | | 0.0419 | 2.1456 | 560 | 0.0905 | 0.7932 | 0.8327 | 0.8125 | 0.9758 | | 0.0329 | 2.2222 | 580 | 0.0771 | 0.8443 | 0.8663 | 0.8552 | 0.9801 | | 0.0434 | 2.2989 | 600 | 0.0838 | 0.8204 | 0.8472 | 0.8336 | 0.9772 | | 0.0378 | 2.3755 | 620 | 0.0770 | 0.8220 | 0.8382 | 0.8300 | 0.9773 | | 0.0378 | 2.4521 | 640 | 0.0784 | 0.8288 | 0.8575 | 0.8429 | 0.9796 | | 0.0417 | 2.5287 | 660 | 0.0863 | 0.8510 | 0.8571 | 0.8540 | 0.9794 | | 0.0423 | 2.6054 | 680 | 0.0912 | 0.8394 | 0.8504 | 0.8449 | 0.9777 | | 0.0484 | 2.6820 | 700 | 0.0837 | 0.8215 | 0.8472 | 0.8342 | 0.9759 | | 0.0394 | 2.7586 | 720 | 0.0777 | 0.8522 | 0.8585 | 0.8553 | 0.9800 | | 0.0383 | 2.8352 | 740 | 0.0724 | 0.8663 | 0.8713 | 0.8688 | 0.9814 | | 0.0446 | 2.9119 | 760 | 0.0774 | 0.8636 | 0.8828 | 0.8731 | 0.9804 | | 0.0424 | 2.9885 | 780 | 0.0750 | 0.8641 | 0.875 | 0.8695 | 0.9815 | | 0.0518 | 3.0651 | 800 | 0.0784 | 0.8359 | 0.8683 | 0.8518 | 0.9799 | | 0.0282 | 3.1418 | 820 | 0.0752 | 0.8649 | 0.8720 | 0.8684 | 0.9808 | | 0.0292 | 3.2184 | 840 | 0.0820 | 0.8562 | 0.8690 | 0.8626 | 0.9795 | | 0.0304 | 3.2950 | 860 | 0.0847 | 0.8528 | 0.8679 | 0.8603 | 0.9796 | | 0.0288 | 3.3716 | 880 | 0.0784 | 0.8583 | 0.8683 | 0.8633 | 0.9800 | | 0.0281 | 3.4483 | 900 | 0.0753 | 0.8546 | 0.8619 | 0.8583 | 0.9795 | | 0.0338 | 3.5249 | 920 | 0.0710 | 0.8589 | 0.8686 | 0.8637 | 0.9807 | | 0.0263 | 3.6015 | 940 | 0.0752 | 0.8635 | 0.8693 | 0.8664 | 0.9806 | | 0.0317 | 3.6782 | 960 | 0.0732 | 0.8649 | 0.8722 | 0.8686 | 0.9798 | | 0.0264 | 3.7548 | 980 | 0.0711 | 0.8650 | 0.8851 | 0.8750 | 0.9808 | | 0.0342 | 3.8314 | 1000 | 0.0694 | 0.8729 | 0.8821 | 0.8775 | 0.9821 | | 0.0294 | 3.9080 | 1020 | 0.0726 | 0.8662 | 0.8794 | 0.8727 | 0.9802 | | 0.0338 | 3.9847 | 1040 | 0.0749 | 0.8747 | 0.8787 | 0.8767 | 0.9812 | | 0.0203 | 4.0613 | 1060 | 0.0777 | 0.8711 | 0.8761 | 0.8736 | 0.9803 | | 0.0221 | 4.1379 | 1080 | 0.0836 | 0.8629 | 0.8736 | 0.8682 | 0.9801 | | 0.0186 | 4.2146 | 1100 | 0.0800 | 0.8644 | 0.8805 | 0.8724 | 0.9806 | | 0.02 | 4.2912 | 1120 | 0.0844 | 0.8683 | 0.8817 | 0.8749 | 0.9811 | | 0.0172 | 4.3678 | 1140 | 0.0797 | 0.8701 | 0.8851 | 0.8775 | 0.9810 | | 0.0227 | 4.4444 | 1160 | 0.0806 | 0.8755 | 0.8824 | 0.8789 | 0.9810 | | 0.0198 | 4.5211 | 1180 | 0.0809 | 0.8658 | 0.8778 | 0.8717 | 0.9803 | | 0.022 | 4.5977 | 1200 | 0.0826 | 0.8748 | 0.8798 | 0.8773 | 0.9812 | | 0.0226 | 4.6743 | 1220 | 0.0765 | 0.8668 | 0.8849 | 0.8757 | 0.9815 | | 0.0248 | 4.7510 | 1240 | 0.0799 | 0.8598 | 0.8722 | 0.8660 | 0.9803 | | 0.0229 | 4.8276 | 1260 | 0.0803 | 0.8646 | 0.8727 | 0.8686 | 0.9809 | | 0.0229 | 4.9042 | 1280 | 0.0773 | 0.8639 | 0.875 | 0.8694 | 0.9811 | | 0.0202 | 4.9808 | 1300 | 0.0756 | 0.8765 | 0.8824 | 0.8794 | 0.9822 | | 0.0156 | 5.0575 | 1320 | 0.0764 | 0.8646 | 0.8835 | 0.8740 | 0.9817 | | 0.0139 | 5.1341 | 1340 | 0.0818 | 0.8673 | 0.8801 | 0.8736 | 0.9813 | | 0.0181 | 5.2107 | 1360 | 0.0792 | 0.8732 | 0.8844 | 0.8788 | 0.9812 | | 0.018 | 5.2874 | 1380 | 0.0778 | 0.8750 | 0.8801 | 0.8775 | 0.9819 | | 0.0139 | 5.3640 | 1400 | 0.0762 | 0.8704 | 0.8824 | 0.8763 | 0.9820 | | 0.0154 | 5.4406 | 1420 | 0.0791 | 0.8753 | 0.8791 | 0.8772 | 0.9819 | | 0.0151 | 5.5172 | 1440 | 0.0816 | 0.8779 | 0.8803 | 0.8791 | 0.9818 | | 0.0177 | 5.5939 | 1460 | 0.0807 | 0.8721 | 0.8803 | 0.8762 | 0.9816 | | 0.0164 | 5.6705 | 1480 | 0.0762 | 0.8701 | 0.8837 | 0.8769 | 0.9817 | | 0.0131 | 5.7471 | 1500 | 0.0783 | 0.8775 | 0.8872 | 0.8823 | 0.9822 | | 0.0174 | 5.8238 | 1520 | 0.0774 | 0.8759 | 0.8824 | 0.8791 | 0.9816 | | 0.0168 | 5.9004 | 1540 | 0.0822 | 0.8619 | 0.8718 | 0.8668 | 0.9808 | | 0.0131 | 5.9770 | 1560 | 0.0822 | 0.8696 | 0.8778 | 0.8736 | 0.9815 | | 0.009 | 6.0536 | 1580 | 0.0848 | 0.8704 | 0.8794 | 0.8748 | 0.9816 | | 0.0108 | 6.1303 | 1600 | 0.0824 | 0.8773 | 0.8821 | 0.8797 | 0.9820 | | 0.0132 | 6.2069 | 1620 | 0.0842 | 0.8762 | 0.8801 | 0.8781 | 0.9816 | | 0.0136 | 6.2835 | 1640 | 0.0814 | 0.8764 | 0.8867 | 0.8816 | 0.9824 | | 0.0144 | 6.3602 | 1660 | 0.0798 | 0.8764 | 0.8863 | 0.8813 | 0.9825 | | 0.0113 | 6.4368 | 1680 | 0.0812 | 0.8790 | 0.8849 | 0.8819 | 0.9823 | | 0.0102 | 6.5134 | 1700 | 0.0826 | 0.8742 | 0.8828 | 0.8785 | 0.9821 | | 0.014 | 6.5900 | 1720 | 0.0792 | 0.8754 | 0.8847 | 0.88 | 0.9825 | | 0.0111 | 6.6667 | 1740 | 0.0815 | 0.8734 | 0.8817 | 0.8775 | 0.9821 | | 0.0098 | 6.7433 | 1760 | 0.0818 | 0.8740 | 0.8828 | 0.8784 | 0.9821 | | 0.012 | 6.8199 | 1780 | 0.0822 | 0.8749 | 0.8840 | 0.8794 | 0.9820 | | 0.01 | 6.8966 | 1800 | 0.0819 | 0.8742 | 0.8833 | 0.8787 | 0.9820 | | 0.0093 | 6.9732 | 1820 | 0.0818 | 0.8756 | 0.8830 | 0.8793 | 0.9821 | ### Framework versions - PEFT 0.13.2 - Transformers 4.46.2 - Pytorch 2.5.1+cu121 - Datasets 3.1.0 - Tokenizers 0.20.3