Update README.md
Browse files
README.md
CHANGED
|
@@ -1,219 +1,221 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
<
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
<sup>
|
| 15 |
-
<sup>
|
| 16 |
-
|
| 17 |
-
<sup>
|
| 18 |
-
<sup>
|
| 19 |
-
|
| 20 |
-
<sup
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
<sup>
|
| 39 |
-
<sup>
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
<
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
|
| 108 |
-
|
| 109 |
-
|
|
| 110 |
-
|
|
| 111 |
-
|
|
| 112 |
-
|
|
| 113 |
-
|
|
| 114 |
-
|
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
</
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
"
|
| 133 |
-
"
|
| 134 |
-
"SurgeNet
|
| 135 |
-
"SurgeNet-
|
| 136 |
-
"SurgeNet-
|
| 137 |
-
"SurgeNet-
|
| 138 |
-
"SurgeNet-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
<
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
|
| 169 |
-
|
| 170 |
-
| **
|
| 171 |
-
|
|
| 172 |
-
| |
|
| 173 |
-
|
|
| 174 |
-
| **
|
| 175 |
-
| |
|
| 176 |
-
| |
|
| 177 |
-
|
|
| 178 |
-
| |
|
| 179 |
-
| |
|
| 180 |
-
| |
|
| 181 |
-
|
| 182 |
-
|
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
|
| 186 |
-
|
| 187 |
-
| **
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
<div align="center">
|
| 7 |
+
<h1>Publications</h1>
|
| 8 |
+
|
| 9 |
+
<h3>Scaling up self-supervised learning for improved surgical foundation models</h3>
|
| 10 |
+
|
| 11 |
+
[Tim J.M. Jaspers](https://timjaspers0801.github.io/)<sup>1* :email:</sup>, [Ronald L.P.D. de Jong](https://scholar.google.com/citations?user=We226GgAAAAJ&hl=en)<sup>2*</sup>,
|
| 12 |
+
[Yiping Li](https://research.tue.nl/nl/persons/yiping-li/publications/)<sup>2</sup>, [Carolus H.J. Kusters](https://chjkusters.github.io/)<sup>1</sup>, Franciscus H.A. Bakker<sup>5</sup>,
|
| 13 |
+
Romy C. van Jaarsveld<sup>3</sup>, Gino M. Kuipers<sup>3</sup>, Richard<sup>3</sup>, Jelle P. Ruurda<sup>3</sup>,
|
| 14 |
+
Willem M. Brinkman<sup>4</sup>, Josien P.W. Pluim<sup>2</sup>,
|
| 15 |
+
Peter H.N. de With<sup>1</sup>, Marcel Breeuwer<sup>2</sup>, [Yasmina Al Khalil](https://scholar.google.com/citations?user=m6co7N0AAAAJ&hl=en)<sup>2</sup>, [Fons van der Sommen](https://scholar.google.com/citations?user=qFiLkCAAAAAJ&hl=en)<sup>1</sup>
|
| 16 |
+
|
| 17 |
+
<sup>1</sup> Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology \
|
| 18 |
+
<sup>2</sup> Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands \
|
| 19 |
+
<sup>3</sup> Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands \
|
| 20 |
+
<sup>4</sup> Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands \
|
| 21 |
+
<sup>5</sup> Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
|
| 22 |
+
|
| 23 |
+
<sup>*</sup> Both authors attributed equally \
|
| 24 |
+
(<sup>:email:</sup>) corresponding author
|
| 25 |
+
|
| 26 |
+
*arxiv* <br /> ([Article](https://doi.org/10.1016/j.media.2024.103348))
|
| 27 |
+
|
| 28 |
+
<h3> Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision</h3>
|
| 29 |
+
|
| 30 |
+
[Tim J.M. Jaspers](https://timjaspers0801.github.io/)<sup>1 :email:</sup>, [Ronald L.P.D. de Jong](https://scholar.google.com/citations?user=We226GgAAAAJ&hl=en)<sup>2</sup>,
|
| 31 |
+
[Yasmina Al Khalil](https://scholar.google.com/citations?user=m6co7N0AAAAJ&hl=en)<sup>2</sup>, Tijn Zeelenberg <sup>1</sup>, [Carolus H.J. Kusters](https://chjkusters.github.io/)<sup>1</sup>, Franciscus H.A. Bakker<sup>5</sup>,
|
| 32 |
+
[Yiping Li](https://research.tue.nl/nl/persons/yiping-li/publications/)<sup>2</sup>, Romy C. van Jaarsveld<sup>3</sup>, Jelle P. Ruurda<sup>3</sup>, Willem M. Brinkman<sup>4</sup>,
|
| 33 |
+
Peter H.N. de With<sup>1</sup>, [Fons van der Sommen](https://scholar.google.com/citations?user=qFiLkCAAAAAJ&hl=en)<sup>1</sup>,
|
| 34 |
+
|
| 35 |
+
*Second Workshop on Data Engineering in Medical Imaging (DEMI) - Satellite Event MICCAI 2024* <br /> ([Proceeding](https://link.springer.com/chapter/10.1007/978-3-031-73748-0_5))
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
<sup>1</sup> Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology \
|
| 39 |
+
<sup>2</sup> Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands \
|
| 40 |
+
<sup>3</sup> Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands \
|
| 41 |
+
<sup>4</sup> Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands \
|
| 42 |
+
<sup>5</sup> Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
|
| 43 |
+
|
| 44 |
+
(<sup>:email:</sup>) corresponding author
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
<h1>Abstract</h1>
|
| 49 |
+
|
| 50 |
+
Foundation models have revolutionized computer vision by achieving state-of-the-art
|
| 51 |
+
performance across diverse tasks through large-scale pretraining on extensive datasets.
|
| 52 |
+
However, their application in surgical computer vision has been limited. This study
|
| 53 |
+
addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that
|
| 54 |
+
sets a new benchmark in surgical computer vision. Trained on the largest reported surgical
|
| 55 |
+
dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves
|
| 56 |
+
consistent top-tier performance across six datasets spanning four surgical procedures
|
| 57 |
+
and three tasks, including semantic segmentation, phase recognition, and critical view
|
| 58 |
+
of safety (CVS) classification. Compared to the best-performing surgical foundation
|
| 59 |
+
models, SurgeNetXL shows mean improvements of 2.4, 8.95, and 12.6% for semantic segmentation,
|
| 60 |
+
phase recognition, and CVS classification, respectively. Additionally,
|
| 61 |
+
SurgeNetXL outperforms the best-performing ImageNet-based variants by 14.4, 4.0,
|
| 62 |
+
and 1.6% in the respective tasks. In addition to advancing model performance, this work
|
| 63 |
+
provides key insights into scaling pretraining datasets, extending training durations, and
|
| 64 |
+
optimizing model architectures specifically for surgical computer vision. These findings
|
| 65 |
+
pave the way for improved generalizability and robustness in data-scarce scenarios, offering
|
| 66 |
+
a comprehensive framework for future research in this domain.
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
<h1>Results</h1>
|
| 70 |
+
|
| 71 |
+
The following figures are from our publications, showcasing the performance of our introduced foundation model
|
| 72 |
+
across diverse surgical tasks and procedures. These results demonstrate the model’s state-of-the-art
|
| 73 |
+
performance on a variety of downstream tasks, reflecting its versatility and robustness in handling
|
| 74 |
+
datasets from multiple surgical procedures.
|
| 75 |
+
|
| 76 |
+
Figure 1 and Figure 2 illustrate comparative rankings of our model against existing foundation models,
|
| 77 |
+
highlighting its superior generalization capabilities across datasets. Figure 3 provides a t-SNE visualization,
|
| 78 |
+
showcasing the clear cluster separation per specific dataset achieved by the model’s feature embeddings,
|
| 79 |
+
further emphasizing its effectiveness in capturing meaningful representations.
|
| 80 |
+
|
| 81 |
+
<div style="display: flex; justify-content: space-around; align-items: center; gap: 20px;">
|
| 82 |
+
|
| 83 |
+
<div style="text-align: center;">
|
| 84 |
+
<img src="figures/radar_ranks.png" alt="Fig 2" width="400" height="300">
|
| 85 |
+
<p><strong>Fig 1:</strong> Radar chart showing model ranks across datasets.</p>
|
| 86 |
+
</div>
|
| 87 |
+
|
| 88 |
+
<div style="text-align: center;">
|
| 89 |
+
<img src="figures/ranking_blob_all.png" alt="Fig 3" width="400" height="300">
|
| 90 |
+
<p><strong>Fig 2:</strong> Blob chart representing ranking metrics for models.</p>
|
| 91 |
+
</div>
|
| 92 |
+
|
| 93 |
+
</div>
|
| 94 |
+
|
| 95 |
+
<div style="text-align: center; margin-top: 20px;">
|
| 96 |
+
<img src="figures/TSNE.png" alt="Fig 3" width="600">
|
| 97 |
+
<p><strong>Fig 3:</strong> t-SNE visualization of feature embeddings showing cluster separation across datasets.</p>
|
| 98 |
+
</div>
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
<h1>Models</h1>
|
| 102 |
+
|
| 103 |
+
The models used in this study are based on the [MetaFormer](https://arxiv.org/abs/2210.13452) architecture. The models are trained using a self-supervised learning
|
| 104 |
+
approach on the SurgeNetXL dataset and its variations,
|
| 105 |
+
introduced this in the following [paper](https://). All model weights can be downloaded from the table below.
|
| 106 |
+
|
| 107 |
+
| Model | Backbone | Epochs | Teacher Backbone | Full DINO checkpoint |
|
| 108 |
+
|-----------------|------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
|
| 109 |
+
| SurgeNetXL | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint0050.pth?download=true) |
|
| 110 |
+
| SurgeNetSmall | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint0050.pth?download=true) |
|
| 111 |
+
| SurgeNetCholec | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint0050.pth?download=true) |
|
| 112 |
+
| SurgeNetRAMIE | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint0050.pth?download=true) |
|
| 113 |
+
| SurgeNetRARP | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint0050.pth?download=true) |
|
| 114 |
+
| SurgeNetPublic | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint0050.pth?download=true) |
|
| 115 |
+
| SurgeNet | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint0050.pth?download=true) |
|
| 116 |
+
| SurgeNet | ConvNextv2 | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint0050.pth?download=true) |
|
| 117 |
+
| SurgeNet | PVTv2 | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint0050.pth?download=true) |
|
| 118 |
+
|
| 119 |
+
<h1>Loading Models</h1>
|
| 120 |
+
|
| 121 |
+
The weights from the teacher network can be used to initialize either your classification or segmentation model using the following code snippet:
|
| 122 |
+
</div>
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
```python
|
| 126 |
+
import torch
|
| 127 |
+
from metaformer import caformer_s18, MetaFormerFPN
|
| 128 |
+
from convnextv2 import convnextv2_tiny, ConvNextFPN
|
| 129 |
+
from pvtv2 import pvt_v2_b2, PVTV2FPN
|
| 130 |
+
|
| 131 |
+
urls = {
|
| 132 |
+
"ImageNet1k": "https://huggingface.co/sail/dl/resolve/main/caformer/caformer_s18.pth",
|
| 133 |
+
"SurgeNetXL": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint_epoch0050_teacher.pth?download=true",
|
| 134 |
+
"SurgeNet": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint_epoch0050_teacher.pth?download=true",
|
| 135 |
+
"SurgeNet-Small": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint_epoch0050_teacher.pth?download=true",
|
| 136 |
+
"SurgeNet-CHOLEC": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint_epoch0050_teacher.pth?download=true",
|
| 137 |
+
"SurgeNet-RAMIE": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint_epoch0050_teacher.pth?download=true",
|
| 138 |
+
"SurgeNet-RARP": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint_epoch0050_teacher.pth?download=true",
|
| 139 |
+
"SurgeNet-Public": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint0050.pth?download=true",
|
| 140 |
+
"SurgeNet-ConvNextv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint_epoch0050_teacher.pth?download=true",
|
| 141 |
+
"SurgeNet-PVTv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint_epoch0050_teacher.pth?download=true",
|
| 142 |
+
}
|
| 143 |
+
|
| 144 |
+
# Metaformer model
|
| 145 |
+
classification_model = caformer_s18(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])
|
| 146 |
+
segmentation_model = MetaFormerFPN(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])
|
| 147 |
+
|
| 148 |
+
# ConvNextv2 model
|
| 149 |
+
classification_model = convnextv2_tiny(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])
|
| 150 |
+
segmentation_model = ConvNextFPN(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])
|
| 151 |
+
|
| 152 |
+
# PVTv2 model
|
| 153 |
+
classification_model = pvt_v2_b2(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])
|
| 154 |
+
segmentation_model = PVTV2FPN(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])
|
| 155 |
+
|
| 156 |
+
```
|
| 157 |
+
|
| 158 |
+
<div align="center">
|
| 159 |
+
Note: If your want a different version of SurgeNet weights (e.g. SurgeNet-Small), you can replace the `pretrained_weights` argument with the desired url (leave the `pretrained` argument as it is).
|
| 160 |
+
|
| 161 |
+
<h1>Surgical Youtube Dataset</h1>
|
| 162 |
+
|
| 163 |
+
A key contribution of our research is the Surgical YouTube dataset, which enhanced our foundation model's
|
| 164 |
+
performance. This curated dataset contains 2,074,234 frames sampled from 23 distinct surgical procedures
|
| 165 |
+
and is publicly available at [huggingface datasets.](https://huggingface.co/datasets/TimJaspersTue/SurgeNetYoutube)
|
| 166 |
+
This datasets is a large part of our SurgeNetXL dataset, which also includes other opensource datasets.
|
| 167 |
+
|
| 168 |
+
| Procedure-specific subset | Dataset | Procedure | #videos | #frames | Public |
|
| 169 |
+
|---------------------------|----------------------------------------------------------------|-----------|---------|-----------|--------|
|
| 170 |
+
| **SurgeNetCholec** | Cholec80 ([Twinnanda et al., 2017b](https://arxiv.org/abs/1602.03012)) | Laparoscopic Cholecystectomy | 76 | 179,164 | Yes |
|
| 171 |
+
| | HeiChole ([Maier-Hein et al., 2021](https://www.synapse.org/Synapse:syn25101790/wiki/608802)) | Laparoscopic Cholecystectomy | 30 | 53,427 | Yes |
|
| 172 |
+
| | hSDB-Chole ([Yoon et al., 2021](https://arxiv.org/abs/2110.12555)) | Laparoscopic Cholecystectomy | 24 | 18,064 | Yes |
|
| 173 |
+
| **SurgeNetRAMIE** | RAMIE-UMCU | RA Esophagectomy | 28 | 377,287 | No |
|
| 174 |
+
| **SurgeNetRARP** | ESAD [Bawa et al., 2021](https://arxiv.org/abs/2006.07164) | RA Esophagectomy | 28 | 47,282 | Yes |
|
| 175 |
+
| | PSI-AVA [Valderrama et al., 2022](https://arxiv.org/abs/2212.04582) | RA Prostatectomy | 8 | 73,618 | Yes |
|
| 176 |
+
| | RARP-AvL | RA Prostatectomy | 8 | 261,516 | No |
|
| 177 |
+
| **Others** | DSAD ([Carstens et al., 2023](https://www.nature.com/articles/s41597-022-01719-2)) | RA Rectal Resection/Extirpation | 32 | 14,623 | Yes |
|
| 178 |
+
| | GLENDA ([Leibetseder et al., 2020](https://link.springer.com/chapter/10.1007/978-3-030-37734-2_36)) | Gynecologic Laparoscopy | 400 | 25,682 | Yes |
|
| 179 |
+
| | LapGyn4 ([Leibetseder et al., 2018](https://dl.acm.org/doi/10.1145/3204949.3208127)) | Gynecologic Laparoscopy | 500 | 59,616 | Yes |
|
| 180 |
+
| | MultiBypass140 ([Lavanchy et al., 2024](https://github.com/CAMMA-public/MultiBypass140)) | Laparoscopic Gastric Bypass Surgery | 140 | 749,419 | Yes |
|
| 181 |
+
| | hSDB-Gastric ([Yoon et al., 2021](https://arxiv.org/abs/2110.12555)) | RA Gastrectomy | 24 | 35,576 | Yes |
|
| 182 |
+
| | SurgToolLoc2022 ([Zia et al., 2023](https://arxiv.org/abs/2305.07152)) | 11 different RA porcine procedures | N/A | 741,516 | Yes |
|
| 183 |
+
| | YouTube [ours](https://huggingface.co/datasets/TimJaspersTue/SurgeNetYoutube) | 23 identified procedures | 3,253 | 2,074,234 | Yes |
|
| 184 |
+
|
| 185 |
+
| SurgeNetXL variations | Dataset | Procedure | #videos | #frames | Public |
|
| 186 |
+
|-----------------------|------------------------------------------------------------|---------------------------------------------------------|---------|---------|--------|
|
| 187 |
+
| **SurgeNetSmall** | 10% of the above (excluding YouTube) | All of the above (excluding YouTube) | \>1345 | 263,679 | Partly |
|
| 188 |
+
| **SurgeNetPublic** | All public datasets (excluding YouTube & private datasets) | All of the above (excluding YouTube & RA Esophagectomy) | \>1238 | 1,997,987 | Yes |
|
| 189 |
+
| **SurgeNet** | All of the above (excluding YouTube) | All of the above (excluding YouTube) | \>1345 | 2,636,790 | Partly |
|
| 190 |
+
| **SurgeNetXL** | All of the above | All of the above | \>4598 | 4,711,024 | Partly |
|
| 191 |
+
|
| 192 |
+
<h1>Acknowledgements</h1>
|
| 193 |
+
|
| 194 |
+
Our implementation of the feature pyramid network is based on the [pytorch segmentation models](https://segmentation-modelspytorch.readthedocs.io/en/latest/) library.
|
| 195 |
+
Pretraining on SurgeNet was performed using the code provided with the [DINO](https://github.com/facebookresearch/dino) publication.
|
| 196 |
+
We have used the code of Schmidgall et al. (2024) to obtain the youtube videos, this code can be found [here](https://github.com/SamuelSchmidgall/GSViT).
|
| 197 |
+
|
| 198 |
+
<h1>Citation</h1>
|
| 199 |
+
If you find our work useful in your research please consider citing our paper:
|
| 200 |
+
</div>
|
| 201 |
+
|
| 202 |
+
```bibtex
|
| 203 |
+
@msc{Jaspers2025,
|
| 204 |
+
title={Scaling up self-supervised learning for improved surgical foundation models},
|
| 205 |
+
year={2025}
|
| 206 |
+
}
|
| 207 |
+
```
|
| 208 |
+
|
| 209 |
+
```bibtex
|
| 210 |
+
@inbook{Jaspers2024,
|
| 211 |
+
title={Exploring the Effect of Dataset Diversity in Self-supervised Learning for Surgical Computer Vision},
|
| 212 |
+
ISBN={9783031737480},
|
| 213 |
+
ISSN={1611-3349},
|
| 214 |
+
url={http://dx.doi.org/10.1007/978-3-031-73748-0_5},
|
| 215 |
+
DOI={10.1007/978-3-031-73748-0_5},
|
| 216 |
+
booktitle={Data Engineering in Medical Imaging},
|
| 217 |
+
publisher={Springer Nature Switzerland},
|
| 218 |
+
author={Jaspers, Tim J. M. and de Jong, Ronald L. P. D. and Al Khalil, Yasmina and Zeelenberg, Tijn and Kusters, Carolus H. J. and Li, Yiping and van Jaarsveld, Romy C. and Bakker, Franciscus H. A. and Ruurda, Jelle P. and Brinkman, Willem M. and De With, Peter H. N. and van der Sommen, Fons},
|
| 219 |
+
year={2024},
|
| 220 |
+
month=oct, pages={43–53} }
|
| 221 |
+
```
|