File size: 10,914 Bytes
8faf222 f5da179 1de1054 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 04a070d 8faf222 f5da179 8faf222 f5da179 8faf222 04a070d 8faf222 f5da179 e0fe93e f5da179 e0fe93e f5da179 0cc37ec f5da179 e0fe93e f5da179 e0fe93e f5da179 04a070d f5da179 04a070d f5da179 04a070d f5da179 04a070d f5da179 04a070d 640d80e f5da179 d9aa212 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 f5da179 8faf222 04a070d 8faf222 f5da179 8faf222 f5da179 8faf222 1de1054 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 |
---
library_name: transformers
inference: false
license: other
---
# SuperGlue
The SuperGlue model was proposed
in [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://arxiv.org/abs/1911.11763) by Paul-Edouard Sarlin, Daniel
DeTone, Tomasz Malisiewicz and Andrew Rabinovich.
This model consists of matching two sets of interest points detected in an image. Paired with the
[SuperPoint model](https://huggingface.co/magic-leap-community/superpoint), it can be used to match two images and
estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.
The abstract from the paper is the following:
*This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences
and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs
are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling
SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics,
our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image
pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in
challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and
can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at this [URL](https://github.com/magicleap/SuperGluePretrainedNetwork).*
<img src="https://cdn-uploads.huggingface.co/production/uploads/632885ba1558dac67c440aa8/2I8QDRNoMhQCuL236CvdN.png" alt="drawing" width="500"/>
<!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/632885ba1558dac67c440aa8/2I8QDRNoMhQCuL236CvdN.png) -->
This model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille).
The original code can be found [here](https://github.com/magicleap/SuperGluePretrainedNetwork).
## Model Details
### Model Description
SuperGlue is a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points.
It introduces a flexible context aggregation mechanism based on attention, enabling it to reason about the underlying 3D scene and feature
assignments. The architecture consists of two main components: the Attentional Graph Neural Network and the Optimal Matching Layer.
<img src="https://cdn-uploads.huggingface.co/production/uploads/632885ba1558dac67c440aa8/zZGjSWQU2na5aPFRak5kp.png" alt="drawing" width="1000"/>
<!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/632885ba1558dac67c440aa8/zZGjSWQU2na5aPFRak5kp.png) -->
The Attentional Graph Neural Network uses a Keypoint Encoder to map keypoint positions and visual descriptors.
It employs self- and cross-attention layers to create powerful representations. The Optimal Matching Layer creates a
score matrix, augments it with dustbins, and finds the optimal partial assignment using the Sinkhorn algorithm.
- **Developed by:** MagicLeap
- **Model type:** Image Matching
- **License:** ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/magicleap/SuperGluePretrainedNetwork
- **Paper:** https://arxiv.org/pdf/1911.11763
- **Demo:** https://psarlin.com/superglue/
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
SuperGlue is designed for feature matching and pose estimation tasks in computer vision. It can be applied to a variety of multiple-view
geometry problems and can handle challenging real-world indoor and outdoor environments. However, it may not perform well on tasks that
require different types of visual understanding, such as object detection or image classification.
## How to Get Started with the Model
Here is a quick example of using the model. Since this model is an image matching model, it requires pairs of images to be matched:
```python
from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests
url = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg"
im1 = Image.open(requests.get(url, stream=True).raw)
url = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg"
im2 = Image.open(requests.get(url, stream=True).raw)
images = [im1, im2]
processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_outdoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_outdoor")
inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)
```
The outputs contain the list of keypoints detected by the keypoint detector as well as the list of matches with their corresponding matching scores.
Due to the nature of SuperGlue, to output a dynamic number of matches, you will need to use the mask attribute to retrieve the respective information:
```python
from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests
url_image_1 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg"
image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
url_image_2 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg"
image_2 = Image.open(requests.get(url_image_2, stream=True).raw)
images = [image_1, image_2]
processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_indoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_indoor")
inputs = processor(images, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Get the respective image masks
image0_mask, image1_mask = outputs_mask[0]
image0_indices = torch.nonzero(image0_mask).squeeze()
image1_indices = torch.nonzero(image1_mask).squeeze()
image0_matches = outputs.matches[0, 0][image0_indices]
image1_matches = outputs.matches[0, 1][image1_indices]
image0_matching_scores = outputs.matching_scores[0, 0][image0_indices]
image1_matching_scores = outputs.matching_scores[0, 1][image1_indices]
```
You can use the `post_process_keypoint_matching` method from the `SuperGlueImageProcessor` to get the keypoints and matches in a more readable format:
```python
image_sizes = [(image.height, image.width) for image in images]
outputs = processor.post_process_keypoint_matching(outputs, image_sizes)
for i, output in enumerate(outputs):
print("For the image pair", i)
for keypoint0, keypoint1, matching_score in zip(output["keypoints0"], output["keypoints1"],
output["matching_scores"]):
print(
f"Keypoint at coordinate {keypoint0.numpy()} in the first image matches with keypoint at coordinate {keypoint1.numpy()} in the second image with a score of {matching_score}."
)
```
From the outputs, you can visualize the matches between the two images using the following code:
```python
import matplotlib.pyplot as plt
import numpy as np
# Create side by side image
merged_image = np.zeros((max(image1.height, image2.height), image1.width + image2.width, 3))
merged_image[: image1.height, : image1.width] = np.array(image1) / 255.0
merged_image[: image2.height, image1.width :] = np.array(image2) / 255.0
plt.imshow(merged_image)
plt.axis("off")
# Retrieve the keypoints and matches
output = outputs[0]
keypoints0 = output["keypoints0"]
keypoints1 = output["keypoints1"]
matching_scores = output["matching_scores"]
keypoints0_x, keypoints0_y = keypoints0[:, 0].numpy(), keypoints0[:, 1].numpy()
keypoints1_x, keypoints1_y = keypoints1[:, 0].numpy(), keypoints1[:, 1].numpy()
# Plot the matches
for keypoint0_x, keypoint0_y, keypoint1_x, keypoint1_y, matching_score in zip(
keypoints0_x, keypoints0_y, keypoints1_x, keypoints1_y, matching_scores
):
plt.plot(
[keypoint0_x, keypoint1_x + image1.width],
[keypoint0_y, keypoint1_y],
color=plt.get_cmap("RdYlGn")(matching_score.item()),
alpha=0.9,
linewidth=0.5,
)
plt.scatter(keypoint0_x, keypoint0_y, c="black", s=2)
plt.scatter(keypoint1_x + image1.width, keypoint1_y, c="black", s=2)
# Save the plot
plt.savefig("matched_image.png", dpi=300, bbox_inches='tight')
plt.close()
```
![image/png](https://cdn-uploads.huggingface.co/production/uploads/632885ba1558dac67c440aa8/_2a_V28C1eEMY_Wy6sRZa.png)
## Training Details
### Training Data
SuperGlue is trained on large annotated datasets for pose estimation, enabling it to learn priors for pose estimation and reason about the 3D scene.
The training data consists of image pairs with ground truth correspondences and unmatched keypoints derived from ground truth poses and depth maps.
### Training Procedure
SuperGlue is trained in a supervised manner using ground truth matches and unmatched keypoints. The loss function maximizes
the negative log-likelihood of the assignment matrix, aiming to simultaneously maximize precision and recall.
#### Training Hyperparameters
- **Training regime:** fp32
#### Speeds, Sizes, Times
SuperGlue is designed to be efficient and runs in real-time on a modern GPU. A forward pass takes approximately 69 milliseconds (15 FPS) for an indoor image pair.
The model has 12 million parameters, making it relatively compact compared to some other deep learning models.
The inference speed of SuperGlue is suitable for real-time applications and can be readily integrated into
modern Simultaneous Localization and Mapping (SLAM) or Structure-from-Motion (SfM) systems.
## Citation
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
```bibtex
@inproceedings{sarlin2020superglue,
title={Superglue: Learning feature matching with graph neural networks},
author={Sarlin, Paul-Edouard and DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={4938--4947},
year={2020}
}
```
## Model Card Authors
[Steven Bucaille](https://github.com/sbucaille) |