Hugging Face's logo

TimJaspersTue
/
SurgeNetModels

Model card Files Community
TimJaspersTue commited on
Commit
1a9b51e
·
verified ·
1 Parent(s): 2ef13d0

Delete README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -215
README.md DELETED
@@ -1,215 +0,0 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
4
- ![FIG 1.](figures/SurgeNet.png)
5
-
6
- <div align="center">
7
- <h1>Publications</h1>
8
-
9
- <h3>Scaling up self-supervised learning for improved surgical foundation models</h3>
10
-
11
- [Tim J.M. Jaspers](https://timjaspers0801.github.io/)<sup>1* :email:</sup>, [Ronald L.P.D. de Jong](https://scholar.google.com/citations?user=We226GgAAAAJ&hl=en)<sup>2*</sup>,
12
- [Yiping Li](https://research.tue.nl/nl/persons/yiping-li/publications/)<sup>2</sup>, [Carolus H.J. Kusters](https://chjkusters.github.io/)<sup>1</sup>, Franciscus H.A. Bakker<sup>5</sup>,
13
- Romy C. van Jaarsveld<sup>3</sup>, Gino M. Kuipers<sup>3</sup>, Richard<sup>3</sup>, Jelle P. Ruurda<sup>3</sup>,
14
- Willem M. Brinkman<sup>4</sup>, Josien P.W. Pluim<sup>2</sup>,
15
- Peter H.N. de With<sup>1</sup>, Marcel Breeuwer<sup>2</sup>, [Yasmina Al Khalil](https://scholar.google.com/citations?user=m6co7N0AAAAJ&hl=en)<sup>2</sup>, [Fons van der Sommen](https://scholar.google.com/citations?user=qFiLkCAAAAAJ&hl=en)<sup>1</sup>
16
-
17
- <sup>1</sup> Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology \
18
- <sup>2</sup> Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands \
19
- <sup>3</sup> Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands \
20
- <sup>4</sup> Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands \
21
- <sup>5</sup> Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
22
-
23
- <sup>*</sup> Both authors attributed equally \
24
- (<sup>:email:</sup>) corresponding author
25
-
26
- *arxiv* <br /> ([Article](https://doi.org/10.1016/j.media.2024.103348))
27
-
28
- <h3> Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision</h3>
29
-
30
- [Tim J.M. Jaspers](https://timjaspers0801.github.io/)<sup>1 :email:</sup>, [Ronald L.P.D. de Jong](https://scholar.google.com/citations?user=We226GgAAAAJ&hl=en)<sup>2</sup>,
31
- [Yasmina Al Khalil](https://scholar.google.com/citations?user=m6co7N0AAAAJ&hl=en)<sup>2</sup>, Tijn Zeelenberg <sup>1</sup>, [Carolus H.J. Kusters](https://chjkusters.github.io/)<sup>1</sup>, Franciscus H.A. Bakker<sup>5</sup>,
32
- [Yiping Li](https://research.tue.nl/nl/persons/yiping-li/publications/)<sup>2</sup>, Romy C. van Jaarsveld<sup>3</sup>, Jelle P. Ruurda<sup>3</sup>, Willem M. Brinkman<sup>4</sup>,
33
- Peter H.N. de With<sup>1</sup>, [Fons van der Sommen](https://scholar.google.com/citations?user=qFiLkCAAAAAJ&hl=en)<sup>1</sup>,
34
-
35
- *Second Workshop on Data Engineering in Medical Imaging &#40;DEMI&#41; - Satellite Event MICCAI 2024* <br /> ([Proceeding](https://link.springer.com/chapter/10.1007/978-3-031-73748-0_5))
36
-
37
-
38
- <sup>1</sup> Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology \
39
- <sup>2</sup> Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands \
40
- <sup>3</sup> Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands \
41
- <sup>4</sup> Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands \
42
- <sup>5</sup> Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
43
-
44
- (<sup>:email:</sup>) corresponding author
45
-
46
-
47
-
48
- ## Abstract
49
- Foundation models have revolutionized computer vision by achieving state-of-the-art
50
- performance across diverse tasks through large-scale pretraining on extensive datasets.
51
- However, their application in surgical computer vision has been limited. This study
52
- addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that
53
- sets a new benchmark in surgical computer vision. Trained on the largest reported surgical
54
- dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves
55
- consistent top-tier performance across six datasets spanning four surgical procedures
56
- and three tasks, including semantic segmentation, phase recognition, and critical view
57
- of safety (CVS) classification. Compared to the best-performing surgical foundation
58
- models, SurgeNetXL shows mean improvements of 2.4, 8.95, and 12.6% for semantic segmentation,
59
- phase recognition, and CVS classification, respectively. Additionally,
60
- SurgeNetXL outperforms the best-performing ImageNet-based variants by 14.4, 4.0,
61
- and 1.6% in the respective tasks. In addition to advancing model performance, this work
62
- provides key insights into scaling pretraining datasets, extending training durations, and
63
- optimizing model architectures specifically for surgical computer vision. These findings
64
- pave the way for improved generalizability and robustness in data-scarce scenarios, offering
65
- a comprehensive framework for future research in this domain.
66
-
67
-
68
- ## Results
69
- The following figures are from our publications, showcasing the performance of our introduced foundation model
70
- across diverse surgical tasks and procedures. These results demonstrate the model’s state-of-the-art
71
- performance on a variety of downstream tasks, reflecting its versatility and robustness in handling
72
- datasets from multiple surgical procedures.
73
-
74
- Figure 1 and Figure 2 illustrate comparative rankings of our model against existing foundation models,
75
- highlighting its superior generalization capabilities across datasets. Figure 3 provides a t-SNE visualization,
76
- showcasing the clear cluster separation per specific dataset achieved by the model’s feature embeddings,
77
- further emphasizing its effectiveness in capturing meaningful representations.
78
-
79
- <div style="display: flex; justify-content: space-around; align-items: center; gap: 20px;">
80
-
81
- <div style="text-align: center;">
82
- <img src="figures/radar_ranks.png" alt="Fig 2" width="400" height="300">
83
- <p><strong>Fig 1:</strong> Radar chart showing model ranks across datasets.</p>
84
- </div>
85
-
86
- <div style="text-align: center;">
87
- <img src="figures/ranking_blob_all.png" alt="Fig 3" width="400" height="300">
88
- <p><strong>Fig 2:</strong> Blob chart representing ranking metrics for models.</p>
89
- </div>
90
-
91
- </div>
92
-
93
- <div style="text-align: center; margin-top: 20px;">
94
- <img src="figures/TSNE.png" alt="Fig 3" width="600">
95
- <p><strong>Fig 3:</strong> t-SNE visualization of feature embeddings showing cluster separation across datasets.</p>
96
- </div>
97
-
98
-
99
- ## Models
100
- The models used in this study are based on the [MetaFormer](https://arxiv.org/abs/2210.13452) architecture. The models are trained using a self-supervised learning
101
- approach on the SurgeNetXL dataset and its variations,
102
- introduced this in the following [paper](https://). All model weights can be downloaded from the table below.
103
-
104
- | Model | Backbone | Epochs | Teacher Backbone | Full DINO checkpoint |
105
- |-----------------|------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
106
- | SurgeNetXL | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint0050.pth?download=true) |
107
- | SurgeNetSmall | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint0050.pth?download=true) |
108
- | SurgeNetCholec | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint0050.pth?download=true) |
109
- | SurgeNetRAMIE | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint0050.pth?download=true) |
110
- | SurgeNetRARP | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint0050.pth?download=true) |
111
- | SurgeNetPublic | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint0050.pth?download=true) |
112
- | SurgeNet | CaFormer | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint0050.pth?download=true) |
113
- | SurgeNet | ConvNextv2 | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint0050.pth?download=true) |
114
- | SurgeNet | PVTv2 | 50 | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint_epoch0050_teacher.pth?download=true) | [Download](https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint0050.pth?download=true) |
115
-
116
- ## Loading Models
117
- The weights from the teacher network can be used to initialize either your classification or segmentation model using the following code snippet:
118
- </div>
119
-
120
-
121
- ```python
122
- import torch
123
- from metaformer import caformer_s18, MetaFormerFPN
124
- from convnextv2 import convnextv2_tiny, ConvNextFPN
125
- from pvtv2 import pvt_v2_b2, PVTV2FPN
126
-
127
- urls = {
128
- "ImageNet1k": "https://huggingface.co/sail/dl/resolve/main/caformer/caformer_s18.pth",
129
- "SurgeNetXL": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint_epoch0050_teacher.pth?download=true",
130
- "SurgeNet": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint_epoch0050_teacher.pth?download=true",
131
- "SurgeNet-Small": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint_epoch0050_teacher.pth?download=true",
132
- "SurgeNet-CHOLEC": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint_epoch0050_teacher.pth?download=true",
133
- "SurgeNet-RAMIE": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint_epoch0050_teacher.pth?download=true",
134
- "SurgeNet-RARP": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint_epoch0050_teacher.pth?download=true",
135
- "SurgeNet-Public": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint0050.pth?download=true",
136
- "SurgeNet-ConvNextv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint_epoch0050_teacher.pth?download=true",
137
- "SurgeNet-PVTv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint_epoch0050_teacher.pth?download=true",
138
- }
139
-
140
- # Metaformer model
141
- classification_model = caformer_s18(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])
142
- segmentation_model = MetaFormerFPN(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])
143
-
144
- # ConvNextv2 model
145
- classification_model = convnextv2_tiny(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])
146
- segmentation_model = ConvNextFPN(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])
147
-
148
- # PVTv2 model
149
- classification_model = pvt_v2_b2(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])
150
- segmentation_model = PVTV2FPN(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])
151
-
152
- ```
153
-
154
- <div align="center">
155
- Note: If your want a different version of SurgeNet weights (e.g. SurgeNet-Small), you can replace the `pretrained_weights` argument with the desired url (leave the `pretrained` argument as it is).
156
-
157
- ## Surgical Youtube Dataset
158
- A key contribution of our research is the Surgical YouTube dataset, which enhanced our foundation model's
159
- performance. This curated dataset contains 2,074,234 frames sampled from 23 distinct surgical procedures
160
- and is publicly available at [huggingface datasets.](https://huggingface.co/datasets/TimJaspersTue/SurgeNetYoutube)
161
- This datasets is a large part of our SurgeNetXL dataset, which also includes other opensource datasets.
162
-
163
- | Procedure-specific subset | Dataset | Procedure | #videos | #frames | Public |
164
- |---------------------------|----------------------------------------------------------------|-----------|---------|-----------|--------|
165
- | **SurgeNetCholec** | Cholec80 ([Twinnanda et al., 2017b](https://arxiv.org/abs/1602.03012)) | Laparoscopic Cholecystectomy | 76 | 179,164 | Yes |
166
- | | HeiChole ([Maier-Hein et al., 2021](https://www.synapse.org/Synapse:syn25101790/wiki/608802)) | Laparoscopic Cholecystectomy | 30 | 53,427 | Yes |
167
- | | hSDB-Chole ([Yoon et al., 2021](https://arxiv.org/abs/2110.12555)) | Laparoscopic Cholecystectomy | 24 | 18,064 | Yes |
168
- | **SurgeNetRAMIE** | RAMIE-UMCU | RA Esophagectomy | 28 | 377,287 | No |
169
- | **SurgeNetRARP** | ESAD [Bawa et al., 2021](https://arxiv.org/abs/2006.07164) | RA Esophagectomy | 28 | 47,282 | Yes |
170
- | | PSI-AVA [Valderrama et al., 2022](https://arxiv.org/abs/2212.04582) | RA Prostatectomy | 8 | 73,618 | Yes |
171
- | | RARP-AvL | RA Prostatectomy | 8 | 261,516 | No |
172
- | **Others** | DSAD ([Carstens et al., 2023](https://www.nature.com/articles/s41597-022-01719-2)) | RA Rectal Resection/Extirpation | 32 | 14,623 | Yes |
173
- | | GLENDA ([Leibetseder et al., 2020](https://link.springer.com/chapter/10.1007/978-3-030-37734-2_36)) | Gynecologic Laparoscopy | 400 | 25,682 | Yes |
174
- | | LapGyn4 ([Leibetseder et al., 2018](https://dl.acm.org/doi/10.1145/3204949.3208127)) | Gynecologic Laparoscopy | 500 | 59,616 | Yes |
175
- | | MultiBypass140 ([Lavanchy et al., 2024](https://github.com/CAMMA-public/MultiBypass140)) | Laparoscopic Gastric Bypass Surgery | 140 | 749,419 | Yes |
176
- | | hSDB-Gastric ([Yoon et al., 2021](https://arxiv.org/abs/2110.12555)) | RA Gastrectomy | 24 | 35,576 | Yes |
177
- | | SurgToolLoc2022 ([Zia et al., 2023](https://arxiv.org/abs/2305.07152)) | 11 different RA porcine procedures | N/A | 741,516 | Yes |
178
- | | YouTube [ours](https://huggingface.co/datasets/TimJaspersTue/SurgeNetYoutube) | 23 identified procedures | 3,253 | 2,074,234 | Yes |
179
-
180
- | SurgeNetXL variations | Dataset | Procedure | #videos | #frames | Public |
181
- |-----------------------|------------------------------------------------------------|---------------------------------------------------------|---------|---------|--------|
182
- | **SurgeNetSmall** | 10% of the above (excluding YouTube) | All of the above (excluding YouTube) | \>1345 | 263,679 | Partly |
183
- | **SurgeNetPublic** | All public datasets (excluding YouTube & private datasets) | All of the above (excluding YouTube & RA Esophagectomy) | \>1238 | 1,997,987 | Yes |
184
- | **SurgeNet** | All of the above (excluding YouTube) | All of the above (excluding YouTube) | \>1345 | 2,636,790 | Partly |
185
- | **SurgeNetXL** | All of the above | All of the above | \>4598 | 4,711,024 | Partly |
186
-
187
- ## Acknowledgements
188
- Our implementation of the feature pyramid network is based on the [pytorch segmentation models](https://segmentation-modelspytorch.readthedocs.io/en/latest/) library.
189
- Pretraining on SurgeNet was performed using the code provided with the [DINO](https://github.com/facebookresearch/dino) publication.
190
- We have used the code of Schmidgall et al. (2024) to obtain the youtube videos, this code can be found [here](https://github.com/SamuelSchmidgall/GSViT).
191
-
192
- ## Citation
193
- If you find our work useful in your research please consider citing our paper:
194
- </div>
195
-
196
- ```bibtex
197
- @msc{Jaspers2025,
198
- title={Scaling up self-supervised learning for improved surgical foundation models},
199
- year={2025}
200
- }
201
- ```
202
-
203
- ```bibtex
204
- @inbook{Jaspers2024,
205
- title={Exploring the Effect of Dataset Diversity in Self-supervised Learning for Surgical Computer Vision},
206
- ISBN={9783031737480},
207
- ISSN={1611-3349},
208
- url={http://dx.doi.org/10.1007/978-3-031-73748-0_5},
209
- DOI={10.1007/978-3-031-73748-0_5},
210
- booktitle={Data Engineering in Medical Imaging},
211
- publisher={Springer Nature Switzerland},
212
- author={Jaspers, Tim J. M. and de Jong, Ronald L. P. D. and Al Khalil, Yasmina and Zeelenberg, Tijn and Kusters, Carolus H. J. and Li, Yiping and van Jaarsveld, Romy C. and Bakker, Franciscus H. A. and Ruurda, Jelle P. and Brinkman, Willem M. and De With, Peter H. N. and van der Sommen, Fons},
213
- year={2024},
214
- month=oct, pages={43–53} }
215
- ```