diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,4 +1,5 @@ --- +language: en library_name: setfit tags: - setfit @@ -178,22 +179,23 @@ widget: pipeline_tag: text-classification inference: true model-index: -- name: SetFit with sentence-transformers/multi-qa-mpnet-base-cos-v1 +- name: SetFit with sentence-transformers/multi-qa-mpnet-base-cos-v1 on Wiki Labeled + Articles results: - task: type: text-classification name: Text Classification dataset: - name: Unknown + name: Wiki Labeled Articles type: unknown split: test metrics: - type: f1 - value: 0.7835469477082652 + value: 0.7897499102401195 name: F1 --- -# SetFit with sentence-transformers/multi-qa-mpnet-base-cos-v1 +# SetFit with sentence-transformers/multi-qa-mpnet-base-cos-v1 on Wiki Labeled Articles This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/multi-qa-mpnet-base-cos-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-cos-v1) as the Sentence Transformer embedding model. A [SetFitHead](huggingface.co/docs/setfit/reference/main#setfit.SetFitHead) instance is used for classification. @@ -211,7 +213,7 @@ The model has been trained using an efficient few-shot learning technique that i - **Maximum Sequence Length:** 512 tokens - **Number of Classes:** 43 classes - +- **Language:** en ### Model Sources @@ -221,58 +223,58 @@ The model has been trained using an efficient few-shot learning technique that i - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ### Model Labels -| Label | Examples | -|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 9 | | -| 1 | | -| 42 | | -| 26 | | -| 16 | | -| 21 | | -| 24 | | -| 2 | | -| 23 | | -| 14 | | -| 7 | | -| 18 | | -| 32 | | -| 22 | | -| 3 | | -| 33 | | -| 39 | | -| 27 | | -| 17 | | -| 10 | | -| 6 | | -| 8 | | -| 11 | | -| 0 | | -| 19 | | -| 13 | | -| 4 | | -| 40 | | -| 37 | | -| 15 | | -| 34 | | -| 38 | | -| 12 | | -| 5 | | -| 31 | | -| 30 | | -| 36 | | -| 20 | | -| 35 | | -| 41 | | -| 29 | | -| 25 | | -| 28 | | +| Label | Examples | +|:------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 27 | | +| 22 | | +| 3 | | +| 1 | | +| 9 | | +| 13 | | +| 17 | | +| 31 | | +| 37 | | +| 23 | | +| 24 | | +| 38 | | +| 36 | | +| 2 | | +| 0 | | +| 10 | | +| 4 | | +| 39 | | +| 41 | | +| 42 | | +| 6 | | +| 29 | | +| 34 | | +| 32 | | +| 21 | | +| 8 | | +| 33 | | +| 25 | | +| 16 | | +| 28 | | +| 5 | | +| 15 | | +| 12 | | +| 30 | | +| 14 | | +| 40 | | +| 7 | | +| 26 | | +| 35 | | +| 19 | | +| 20 | | +| 11 | | +| 18 | | ## Evaluation ### Metrics | Label | F1 | |:--------|:-------| -| **all** | 0.7835 | +| **all** | 0.7897 | ## Uses @@ -324,61 +326,61 @@ preds = model("##rch procedure that evaluates the objective function p x display ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:---------|:----| -| Word count | 1 | 369.7167 | 509 | +| Word count | 1 | 369.5217 | 509 | | Label | Training Sample Count | |:------|:----------------------| -| 0 | 400 | -| 1 | 400 | -| 2 | 400 | -| 3 | 400 | +| 0 | 830 | +| 1 | 584 | +| 2 | 420 | +| 3 | 927 | | 4 | 356 | | 5 | 374 | -| 6 | 400 | +| 6 | 520 | | 7 | 364 | -| 8 | 400 | +| 8 | 422 | | 9 | 372 | -| 10 | 400 | +| 10 | 494 | | 11 | 295 | -| 12 | 400 | +| 12 | 558 | | 13 | 278 | | 14 | 314 | -| 15 | 400 | -| 16 | 400 | +| 15 | 721 | +| 16 | 417 | | 17 | 379 | | 18 | 357 | | 19 | 370 | | 20 | 337 | | 21 | 373 | -| 22 | 400 | -| 23 | 400 | +| 22 | 661 | +| 23 | 754 | | 24 | 312 | -| 25 | 400 | +| 25 | 481 | | 26 | 386 | -| 27 | 400 | -| 28 | 400 | -| 29 | 400 | -| 30 | 400 | -| 31 | 400 | +| 27 | 556 | +| 28 | 551 | +| 29 | 840 | +| 30 | 574 | +| 31 | 470 | | 32 | 284 | | 33 | 311 | -| 34 | 400 | +| 34 | 633 | | 35 | 318 | -| 36 | 400 | -| 37 | 400 | -| 38 | 400 | -| 39 | 400 | -| 40 | 400 | -| 41 | 400 | +| 36 | 687 | +| 37 | 848 | +| 38 | 668 | +| 39 | 721 | +| 40 | 603 | +| 41 | 747 | | 42 | 336 | ### Training Hyperparameters - batch_size: (32, 32) -- num_epochs: (5, 16) +- num_epochs: (4, 8) - max_steps: -1 - sampling_strategy: oversampling - num_iterations: 20 -- body_learning_rate: (2.6042641306580816e-05, 0.01) +- body_learning_rate: (2.7e-05, 0.01) - head_learning_rate: 0.01 - loss: SupConLoss - distance_metric: cosine_distance @@ -394,17 +396,17 @@ preds = model("##rch procedure that evaluates the objective function p x display ### Training Results | Epoch | Step | Training Loss | Validation Loss | |:----------:|:--------:|:-------------:|:---------------:| -| 0.0020 | 1 | 2.1502 | - | -| 0.4970 | 250 | 1.8042 | - | -| 0.9940 | 500 | 0.5241 | 0.9686 | -| 1.4911 | 750 | 0.7032 | - | -| 1.9881 | 1000 | 0.6512 | 0.9839 | -| 2.4851 | 1250 | 0.5568 | - | -| **2.9821** | **1500** | **0.359** | **0.8957** | -| 3.4791 | 1750 | 0.2935 | - | -| 3.9761 | 2000 | 0.7332 | 0.9742 | -| 4.4732 | 2250 | 0.1874 | - | -| 4.9702 | 2500 | 0.0619 | 0.9543 | +| 0.0015 | 1 | 2.182 | - | +| 0.3671 | 250 | 1.0321 | - | +| 0.7342 | 500 | 1.01 | 0.9291 | +| 1.1013 | 750 | 0.7586 | - | +| 1.4684 | 1000 | 0.2408 | 0.9875 | +| 1.8355 | 1250 | 0.8995 | - | +| 2.2026 | 1500 | 0.3702 | 0.9411 | +| 2.5698 | 1750 | 0.669 | - | +| 2.9369 | 2000 | 0.2361 | 0.9538 | +| 3.3040 | 2250 | 0.1108 | - | +| **3.6711** | **2500** | **0.5895** | **0.9276** | * The bold row denotes the saved checkpoint. ### Framework Versions