Update README.md
Browse filesAdd metrics on ROG test data
README.md
CHANGED
@@ -16,7 +16,32 @@ metrics:
|
|
16 |
|
17 |
This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
|
18 |
|
19 |
-
It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
|
21 |
Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
|
22 |
|
|
|
16 |
|
17 |
This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
|
18 |
|
19 |
+
It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.95 for the positive class on
|
20 |
+
te test split of the same dataset.
|
21 |
+
|
22 |
+
|
23 |
+
# Evaluation
|
24 |
+
|
25 |
+
Although the output of the model is a series 0 or 1, describing their 20ms frames, the evaluation was done on
|
26 |
+
event level; spans of consecutive outputs 1 were bundled together into one event. When the true and predicted
|
27 |
+
events partially overlap, this is counted as a true positive.
|
28 |
+
|
29 |
+
## Evaluation on ROG corpus
|
30 |
+
|
31 |
+
The train and test data were obtained by resegmenting ROG corpus and using only segments with filled pauses. As a result,
|
32 |
+
no true negatives are present in the data and the behaviour of the negative class (i.e. no filled pause detected) is unpredictable.
|
33 |
+
```
|
34 |
+
precision recall f1-score support
|
35 |
+
|
36 |
+
0 0.531 0.123 0.200 211
|
37 |
+
1 0.907 0.987 0.946 1834
|
38 |
+
|
39 |
+
accuracy 0.898 2045
|
40 |
+
macro avg 0.719 0.555 0.573 2045
|
41 |
+
weighted avg 0.868 0.898 0.869 2045
|
42 |
+
```
|
43 |
+
|
44 |
+
## Evaluation on ParlaSpeech [HR](https://huggingface.co/datasets/classla/ParlaSpeech-HR) and [RS](https://huggingface.co/datasets/classla/ParlaSpeech-RS) corpora
|
45 |
|
46 |
Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
|
47 |
|