amyeroberts HF staff stevenbucaille commited on
Commit
783ea42
1 Parent(s): 0c13f95

Updated model card (#1)

Browse files

- Updated model card (655ca09461baa74bf9cf412a1e9a5e6b91ec9927)


Co-authored-by: Steven Bucaille <stevenbucaille@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +102 -1
README.md CHANGED
@@ -3,4 +3,105 @@ tags:
3
  - vision
4
  - image-matching
5
  inference: false
6
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  - vision
4
  - image-matching
5
  inference: false
6
+ ---
7
+
8
+
9
+ # SuperPoint
10
+
11
+ ## Overview
12
+
13
+ The SuperPoint model was proposed
14
+ in [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel
15
+ DeTone, Tomasz Malisiewicz and Andrew Rabinovich.
16
+
17
+ This model is the result of a self-supervised training of a fully-convolutional network for interest point detection and
18
+ description. The model is able to detect interest points that are repeatable under homographic transformations and
19
+ provide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature
20
+ extractor for other tasks such as homography estimation, image matching, etc.
21
+
22
+ The abstract from the paper is the following:
23
+
24
+ *This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a
25
+ large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our
26
+ fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and
27
+ associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography
28
+ approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
29
+ synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
30
+ to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
31
+ traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
32
+ when compared to LIFT, SIFT and ORB.*
33
+
34
+ ## How to use
35
+
36
+ Here is a quick example of using the model to detect interest points in an image:
37
+
38
+ ```python
39
+ from transformers import AutoImageProcessor, AutoModel
40
+ import torch
41
+ from PIL import Image
42
+ import requests
43
+
44
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
45
+ image = Image.open(requests.get(url, stream=True).raw)
46
+
47
+ processor = AutoImageProcessor.from_pretrained("stevenbucaille/superpoint")
48
+ model = AutoModel.from_pretrained("stevenbucaille/superpoint")
49
+
50
+ inputs = processor(image, return_tensors="pt")
51
+ outputs = model(**inputs)
52
+ ```
53
+
54
+ The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).
55
+
56
+ You can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,
57
+ you will need to use the mask attribute to retrieve the respective information :
58
+
59
+ ```python
60
+ from transformers import AutoImageProcessor, AutoModel
61
+ import torch
62
+ from PIL import Image
63
+ import requests
64
+
65
+ url_image_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
66
+ image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
67
+ url_image_2 = "http://images.cocodataset.org/test-stuff2017/000000000568.jpg"
68
+ image_2 = Image.open(requests.get(url_image_2, stream=True).raw)
69
+
70
+ images = [image_1, image_2]
71
+
72
+ processor = AutoImageProcessor.from_pretrained("stevenbucaille/superpoint")
73
+ model = AutoModel.from_pretrained("stevenbucaille/superpoint")
74
+
75
+ inputs = processor(images, return_tensors="pt")
76
+ outputs = model(**inputs)
77
+
78
+ for i in range(len(images)):
79
+ image_mask = outputs.mask[i]
80
+ image_indices = torch.nonzero(image_mask).squeeze()
81
+ image_keypoints = outputs.keypoints[i][image_indices]
82
+ image_scores = outputs.scores[i][image_indices]
83
+ image_descriptors = outputs.descriptors[i][image_indices]
84
+ ```
85
+
86
+ You can then print the keypoints on the image to visualize the result :
87
+ ```python
88
+ import cv2
89
+ for keypoint, score in zip(image_keypoints, image_scores):
90
+ keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())
91
+ color = tuple([score.item() * 255] * 3)
92
+ image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)
93
+ cv2.imwrite("output_image.png", image)
94
+ ```
95
+
96
+ This model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille).
97
+ The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).
98
+
99
+ ```bibtex
100
+ @inproceedings{detone2018superpoint,
101
+ title={Superpoint: Self-supervised interest point detection and description},
102
+ author={DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew},
103
+ booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition workshops},
104
+ pages={224--236},
105
+ year={2018}
106
+ }
107
+ ```