ClinicalMetaScience
commited on
Commit
•
38694cd
1
Parent(s):
fe42a04
Update README.md
Browse files
README.md
CHANGED
@@ -70,6 +70,81 @@ predict_test=trainer.predict(tokenized_data["inference"])
|
|
70 |
Further information on analyzing your own or our example data can be found in this [script](https://github.com/PsyCapsLock/PubBiasDetect/blob/main/Scripts/example_folder/Predict_Example_Abstracts_using_NegativeResultDetector.ipynb)
|
71 |
from our [GitHub repository](https://github.com/PsyCapsLock/PubBiasDetect).
|
72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
73 |
## Disclaimer
|
74 |
This tool is developed to analyze and predict the prevalence of positive and negative results in scientific abstracts based on the SciBERT model. While publication bias is a plausible explanation for certain patterns of results observed in scientific literature, the analyses conducted by this tool do not conclusively establish the presence of publication bias or any other underlying factors. It's essential to understand that this tool evaluates data but does not delve into the underlying reasons for the observed trends.
|
75 |
The validation of this tool has been conducted on primary studies from the field of clinical psychology and psychotherapy. While it might yield insights when applied to abstracts of other fields or other types of studies (such as meta-analyses), its applicability and accuracy in such contexts have not been thoroughly tested yet. The developers of this tool are not responsible for any misinterpretation or misuse of the tool's results, and encourage users to have a comprehensive understanding of the limitations inherent in statistical analysis and prediction models.
|
|
|
70 |
Further information on analyzing your own or our example data can be found in this [script](https://github.com/PsyCapsLock/PubBiasDetect/blob/main/Scripts/example_folder/Predict_Example_Abstracts_using_NegativeResultDetector.ipynb)
|
71 |
from our [GitHub repository](https://github.com/PsyCapsLock/PubBiasDetect).
|
72 |
|
73 |
+
## Results
|
74 |
+
**Table 1** <br>
|
75 |
+
*Different metric scores for model evaluation of test data from the annotated `MAIN` corpus, consisting of *n* = 198 abstracts authored by researchers affiliated with German clinical psychology departments and published between 2012 and 2022*
|
76 |
+
<table>
|
77 |
+
<thead>
|
78 |
+
<tr>
|
79 |
+
<th rowspan="2"></th>
|
80 |
+
<th rowspan="2">Accuracy</th>
|
81 |
+
<th colspan="3">Mixed & Negative Results</th>
|
82 |
+
<th colspan="3">Positive Results Only</th>
|
83 |
+
</tr>
|
84 |
+
<tr>
|
85 |
+
<th>F1</th>
|
86 |
+
<th>Recall</th>
|
87 |
+
<th>Precision</th>
|
88 |
+
<th>F1</th>
|
89 |
+
<th>Recall</th>
|
90 |
+
<th>Precision</th>
|
91 |
+
</tr>
|
92 |
+
</thead>
|
93 |
+
<tbody>
|
94 |
+
<tr>
|
95 |
+
<td>SciBERT</td>
|
96 |
+
<td><strong>0.864</strong></td>
|
97 |
+
<td><strong>0.867</strong></td>
|
98 |
+
<td><strong>0.907</strong></td>
|
99 |
+
<td><strong>0.830</strong></td>
|
100 |
+
<td><strong>0.860</strong></td>
|
101 |
+
<td><strong>0.822</strong></td>
|
102 |
+
<td><strong>0.902</strong></td>
|
103 |
+
</tr>
|
104 |
+
<tr>
|
105 |
+
<td>Random Forest</td>
|
106 |
+
<td>0.803</td>
|
107 |
+
<td>0.810</td>
|
108 |
+
<td>0.856</td>
|
109 |
+
<td>0.769</td>
|
110 |
+
<td>0.796</td>
|
111 |
+
<td>0.752</td>
|
112 |
+
<td>0.844</td>
|
113 |
+
</tr>
|
114 |
+
<tr>
|
115 |
+
<td>Extracted <em>p</em>-values</td>
|
116 |
+
<td>0.515</td>
|
117 |
+
<td>0.495</td>
|
118 |
+
<td>0.485</td>
|
119 |
+
<td>0.505</td>
|
120 |
+
<td>0.534</td>
|
121 |
+
<td>0.545</td>
|
122 |
+
<td>0.524</td>
|
123 |
+
</tr>
|
124 |
+
<tr>
|
125 |
+
<td>Extracted NL Indicators</td>
|
126 |
+
<td>0.530</td>
|
127 |
+
<td>0.497</td>
|
128 |
+
<td>0.474</td>
|
129 |
+
<td>0.523</td>
|
130 |
+
<td>0.559</td>
|
131 |
+
<td>0.584</td>
|
132 |
+
<td>0.536</td>
|
133 |
+
</tr>
|
134 |
+
<tr>
|
135 |
+
<td>Number of Words</td>
|
136 |
+
<td>0.475</td>
|
137 |
+
<td>0.441</td>
|
138 |
+
<td>0.423</td>
|
139 |
+
<td>0.461</td>
|
140 |
+
<td>0.505</td>
|
141 |
+
<td>0.525</td>
|
142 |
+
<td>0.486</td>
|
143 |
+
</tr>
|
144 |
+
</tbody>
|
145 |
+
</table>
|
146 |
+
|
147 |
+
|
148 |
## Disclaimer
|
149 |
This tool is developed to analyze and predict the prevalence of positive and negative results in scientific abstracts based on the SciBERT model. While publication bias is a plausible explanation for certain patterns of results observed in scientific literature, the analyses conducted by this tool do not conclusively establish the presence of publication bias or any other underlying factors. It's essential to understand that this tool evaluates data but does not delve into the underlying reasons for the observed trends.
|
150 |
The validation of this tool has been conducted on primary studies from the field of clinical psychology and psychotherapy. While it might yield insights when applied to abstracts of other fields or other types of studies (such as meta-analyses), its applicability and accuracy in such contexts have not been thoroughly tested yet. The developers of this tool are not responsible for any misinterpretation or misuse of the tool's results, and encourage users to have a comprehensive understanding of the limitations inherent in statistical analysis and prediction models.
|