5roop commited on
Commit
0355aa3
1 Parent(s): 7758e25

Update README.md

Browse files

Add metrics on ROG test data

Files changed (1) hide show
  1. README.md +26 -1
README.md CHANGED
@@ -16,7 +16,32 @@ metrics:
16
 
17
  This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
18
 
19
- It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.952868 on the test split of the same dataset.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
 
21
  Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
22
 
 
16
 
17
  This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
18
 
19
+ It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.95 for the positive class on
20
+ te test split of the same dataset.
21
+
22
+
23
+ # Evaluation
24
+
25
+ Although the output of the model is a series 0 or 1, describing their 20ms frames, the evaluation was done on
26
+ event level; spans of consecutive outputs 1 were bundled together into one event. When the true and predicted
27
+ events partially overlap, this is counted as a true positive.
28
+
29
+ ## Evaluation on ROG corpus
30
+
31
+ The train and test data were obtained by resegmenting ROG corpus and using only segments with filled pauses. As a result,
32
+ no true negatives are present in the data and the behaviour of the negative class (i.e. no filled pause detected) is unpredictable.
33
+ ```
34
+ precision recall f1-score support
35
+
36
+ 0 0.531 0.123 0.200 211
37
+ 1 0.907 0.987 0.946 1834
38
+
39
+ accuracy 0.898 2045
40
+ macro avg 0.719 0.555 0.573 2045
41
+ weighted avg 0.868 0.898 0.869 2045
42
+ ```
43
+
44
+ ## Evaluation on ParlaSpeech [HR](https://huggingface.co/datasets/classla/ParlaSpeech-HR) and [RS](https://huggingface.co/datasets/classla/ParlaSpeech-RS) corpora
45
 
46
  Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
47