SuperGlue

The SuperGlue model was proposed in SuperGlue: Learning Feature Matching with Graph Neural Networks by Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich.

This model consists of matching two sets of interest points detected in an image. Paired with the SuperPoint model, it can be used to match two images and estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.

The abstract from the paper is the following:

This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at this URL.

This model was contributed by stevenbucaille. The original code can be found here.

Model Details

Model Description

SuperGlue is a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. It introduces a flexible context aggregation mechanism based on attention, enabling it to reason about the underlying 3D scene and feature assignments. The architecture consists of two main components: the Attentional Graph Neural Network and the Optimal Matching Layer.

The Attentional Graph Neural Network uses a Keypoint Encoder to map keypoint positions and visual descriptors. It employs self- and cross-attention layers to create powerful representations. The Optimal Matching Layer creates a score matrix, augments it with dustbins, and finds the optimal partial assignment using the Sinkhorn algorithm.

Developed by: MagicLeap
Model type: Image Matching
License: ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY

Model Sources [optional]

Repository: https://github.com/magicleap/SuperGluePretrainedNetwork
Paper: https://arxiv.org/pdf/1911.11763
Demo: https://psarlin.com/superglue/

Uses

Direct Use

SuperGlue is designed for feature matching and pose estimation tasks in computer vision. It can be applied to a variety of multiple-view geometry problems and can handle challenging real-world indoor and outdoor environments. However, it may not perform well on tasks that require different types of visual understanding, such as object detection or image classification.

How to Get Started with the Model

Here is a quick example of using the model. Since this model is an image matching model, it requires pairs of images to be matched:

from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests

url_image1 = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/scannet_sample_images/scene0758_00_frame-000510.jpg?raw=true"
image1 = Image.open(requests.get(url_image1, stream=True).raw)
url_image2 = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/scannet_sample_images/scene0758_00_frame-000165.jpg?raw=true"
image2 = Image.open(requests.get(url_image2, stream=True).raw)
images = [im1, im2]

processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_indoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_indoor")

inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)

The outputs contain the list of keypoints detected by the keypoint detector as well as the list of matches with their corresponding matching scores. Due to the nature of SuperGlue, to output a dynamic number of matches, you will need to use the mask attribute to retrieve the respective information:

from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests

url_image1 = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/scannet_sample_images/scene0758_00_frame-000510.jpg?raw=true"
image1 = Image.open(requests.get(url_image1, stream=True).raw)
url_image2 = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/scannet_sample_images/scene0758_00_frame-000165.jpg?raw=true"
image2 = Image.open(requests.get(url_image2, stream=True).raw)

images = [image_1, image_2]

processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_indoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_indoor")

inputs = processor(images, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Get the respective image masks 
image0_mask, image1_mask = outputs_mask[0]

image0_indices = torch.nonzero(image0_mask).squeeze()
image1_indices = torch.nonzero(image1_mask).squeeze()

image0_matches = outputs.matches[0, 0][image0_indices]
image1_matches = outputs.matches[0, 1][image1_indices]

image0_matching_scores = outputs.matching_scores[0, 0][image0_indices]
image1_matching_scores = outputs.matching_scores[0, 1][image1_indices]

You can use the post_process_keypoint_matching method from the SuperGlueImageProcessor to get the keypoints and matches in a more readable format:

image_sizes = [[(image.height, image.width) for image in images]]
outputs = processor.post_process_keypoint_matching(outputs, image_sizes)
for i, output in enumerate(outputs):
    print("For the image pair", i)
    for keypoint0, keypoint1, matching_score in zip(output["keypoints0"], output["keypoints1"],
                                                    output["matching_scores"]):
        print(
            f"Keypoint at coordinate {keypoint0.numpy()} in the first image matches with keypoint at coordinate {keypoint1.numpy()} in the second image with a score of {matching_score}."
        )

From the outputs, you can visualize the matches between the two images using the following code:

import matplotlib.pyplot as plt
import numpy as np

# Create side by side image
merged_image = np.zeros((max(image1.height, image2.height), image1.width + image2.width, 3))
merged_image[: image1.height, : image1.width] = np.array(image1) / 255.0
merged_image[: image2.height, image1.width :] = np.array(image2) / 255.0
plt.imshow(merged_image)
plt.axis("off")

# Retrieve the keypoints and matches
output = outputs[0]
keypoints0 = output["keypoints0"]
keypoints1 = output["keypoints1"]
matching_scores = output["matching_scores"]
keypoints0_x, keypoints0_y = keypoints0[:, 0].numpy(), keypoints0[:, 1].numpy()
keypoints1_x, keypoints1_y = keypoints1[:, 0].numpy(), keypoints1[:, 1].numpy()

# Plot the matches
for keypoint0_x, keypoint0_y, keypoint1_x, keypoint1_y, matching_score in zip(
        keypoints0_x, keypoints0_y, keypoints1_x, keypoints1_y, matching_scores
):
    plt.plot(
        [keypoint0_x, keypoint1_x + image1.width],
        [keypoint0_y, keypoint1_y],
        color=plt.get_cmap("RdYlGn")(matching_score.item()),
        alpha=0.9,
        linewidth=0.5,
    )
    plt.scatter(keypoint0_x, keypoint0_y, c="black", s=2)
    plt.scatter(keypoint1_x + image1.width, keypoint1_y, c="black", s=2)

# Save the plot
plt.savefig("matched_image.png", dpi=300, bbox_inches='tight')
plt.close()

Training Details

Training Data

SuperGlue is trained on large annotated datasets for pose estimation, enabling it to learn priors for pose estimation and reason about the 3D scene. The training data consists of image pairs with ground truth correspondences and unmatched keypoints derived from ground truth poses and depth maps.

Training Procedure

SuperGlue is trained in a supervised manner using ground truth matches and unmatched keypoints. The loss function maximizes the negative log-likelihood of the assignment matrix, aiming to simultaneously maximize precision and recall.

Training Hyperparameters

Training regime: fp32

Speeds, Sizes, Times

SuperGlue is designed to be efficient and runs in real-time on a modern GPU. A forward pass takes approximately 69 milliseconds (15 FPS) for an indoor image pair. The model has 12 million parameters, making it relatively compact compared to some other deep learning models. The inference speed of SuperGlue is suitable for real-time applications and can be readily integrated into modern Simultaneous Localization and Mapping (SLAM) or Structure-from-Motion (SfM) systems.

Citation

BibTeX:

@inproceedings{sarlin2020superglue,
  title={Superglue: Learning feature matching with graph neural networks},
  author={Sarlin, Paul-Edouard and DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={4938--4947},
  year={2020}
}

Model Card Authors

Steven Bucaille