Transformers
MDSR
super-image
image-super-resolution
Inference Endpoints
Eugene Siow commited on
Commit
4609933
1 Parent(s): 5f15acf

Initial commit.

Browse files
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - super-image
5
+ - image-super-resolution
6
+ datasets:
7
+ - eugenesiow/Div2k
8
+ - eugenesiow/Set5
9
+ - eugenesiow/Set14
10
+ - eugenesiow/BSD100
11
+ - eugenesiow/Urban100
12
+ metrics:
13
+ - pnsr
14
+ - ssim
15
+ ---
16
+ # Multi-Scale Deep Super-Resolution System (MDSR)
17
+ MDSR model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/abs/1707.02921) by Lim et al. (2017) and first released in [this repository](https://github.com/sanghyun-son/EDSR-PyTorch).
18
+
19
+ The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling.
20
+
21
+ ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4](images/mdsr_4_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4")
22
+ ## Model description
23
+ The MDSR is a model that uses both deeper and wider architecture (32 ResBlocks and 256 channels) to improve performance. It uses both global and local skip connections, and up-scaling is done at the end of the network. It doesn't use batch normalization layers (input and output have similar distributions, normalizing intermediate features may not be desirable) instead it uses constant scaling layers to ensure stable training. An L1 loss function (absolute error) is used instead of L2 (MSE), the authors showed better performance empirically and it requires less computation.
24
+
25
+ This model also applies the balanced attention (BAM) method invented by [Wang et al. (2021)](https://arxiv.org/abs/2104.07566) to further improve the results.
26
+ ## Intended uses & limitations
27
+ You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
28
+ ### How to use
29
+ The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
30
+ ```bash
31
+ pip install super-image
32
+ ```
33
+ Here is how to use a pre-trained model to upscale your image:
34
+ ```python
35
+ from super_image import MdsrModel, ImageLoader
36
+ from PIL import Image
37
+ import requests
38
+
39
+ url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
40
+ image = Image.open(requests.get(url, stream=True).raw)
41
+
42
+ model = MdsrModel.from_pretrained('eugenesiow/mdsr-bam', scale=2) # scale 2, 3 and 4 models available
43
+ inputs = ImageLoader.load_image(image)
44
+ preds = model(inputs)
45
+
46
+ ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png`
47
+ ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling
48
+ ```
49
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Upscale_Images_with_Pretrained_super_image_Models.ipynb "Open in Colab")
50
+ ## Training data
51
+ The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://huggingface.co/datasets/eugenesiow/Div2k), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
52
+ ## Training procedure
53
+ ### Preprocessing
54
+ We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
55
+ Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
56
+ During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
57
+ Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
58
+
59
+ We need the huggingface [datasets](https://huggingface.co/datasets?filter=task_ids:other-other-image-super-resolution) library to download the data:
60
+ ```bash
61
+ pip install datasets
62
+ ```
63
+ The following code gets the data and preprocesses/augments the data.
64
+
65
+ ```python
66
+ from datasets import load_dataset
67
+ from super_image.data import EvalDataset, TrainDataset, augment_five_crop
68
+
69
+ augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\
70
+ .map(augment_five_crop, batched=True, desc="Augmenting Dataset") # download and augment the data with the five_crop method
71
+ train_dataset = TrainDataset(augmented_dataset) # prepare the train dataset for loading PyTorch DataLoader
72
+ eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation')) # prepare the eval dataset for the PyTorch DataLoader
73
+ ```
74
+ ### Pretraining
75
+ The model was trained on GPU. The training code is provided below:
76
+ ```python
77
+ from super_image import Trainer, TrainingArguments, MdsrModel, MdsrConfig
78
+
79
+ training_args = TrainingArguments(
80
+ output_dir='./results', # output directory
81
+ num_train_epochs=1000, # total number of training epochs
82
+ )
83
+
84
+ config = MdsrConfig(
85
+ scale=4, # train a model to upscale 4x
86
+ bam=True, # apply balanced attention to the network
87
+ )
88
+ model = MdsrModel(config)
89
+
90
+ trainer = Trainer(
91
+ model=model, # the instantiated model to be trained
92
+ args=training_args, # training arguments, defined above
93
+ train_dataset=train_dataset, # training dataset
94
+ eval_dataset=eval_dataset # evaluation dataset
95
+ )
96
+
97
+ trainer.train()
98
+ ```
99
+
100
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Train_super_image_Models.ipynb "Open in Colab")
101
+ ## Evaluation results
102
+ The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm).
103
+
104
+ Evaluation datasets include:
105
+ - Set5 - [Bevilacqua et al. (2012)](https://huggingface.co/datasets/eugenesiow/Set5)
106
+ - Set14 - [Zeyde et al. (2010)](https://huggingface.co/datasets/eugenesiow/Set14)
107
+ - BSD100 - [Martin et al. (2001)](https://huggingface.co/datasets/eugenesiow/BSD100)
108
+ - Urban100 - [Huang et al. (2015)](https://huggingface.co/datasets/eugenesiow/Urban100)
109
+
110
+ The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
111
+
112
+ |Dataset |Scale |Bicubic |mdsr-bam |
113
+ |--- |--- |--- |--- |
114
+ |Set5 |2x |33.64/0.9292 |**** |
115
+ |Set5 |3x |30.39/0.8678 |**** |
116
+ |Set5 |4x |28.42/0.8101 |**32.19/0.8949** |
117
+ |Set14 |2x |30.22/0.8683 |**** |
118
+ |Set14 |3x |27.53/0.7737 |**** |
119
+ |Set14 |4x |25.99/0.7023 |**28.73/0.7847** |
120
+ |BSD100 |2x |29.55/0.8425 |**** |
121
+ |BSD100 |3x |27.20/0.7382 |**** |
122
+ |BSD100 |4x |25.96/0.6672 |**28.5/0.7645** |
123
+ |Urban100 |2x |26.66/0.8408 |**** |
124
+ |Urban100 |3x | |**** |
125
+ |Urban100 |4x |23.14/0.6573 |**26.02/0.7834** |
126
+
127
+ ![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2](images/mdsr_2_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2")
128
+
129
+ You can find a notebook to easily run evaluation on pretrained models below:
130
+
131
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Evaluate_Pretrained_super_image_Models.ipynb "Open in Colab")
132
+
133
+ ## BibTeX entry and citation info
134
+ ```bibtex
135
+ @misc{wang2021bam,
136
+ title={BAM: A Lightweight and Efficient Balanced Attention Mechanism for Single Image Super Resolution},
137
+ author={Fanyi Wang and Haotian Hu and Cheng Shen},
138
+ year={2021},
139
+ eprint={2104.07566},
140
+ archivePrefix={arXiv},
141
+ primaryClass={eess.IV}
142
+ }
143
+ ```
144
+
145
+ ```bibtex
146
+ @article{ahn2018fast,
147
+ title={Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network},
148
+ author={Ahn, Namhyuk and Kang, Byungkon and Sohn, Kyung-Ah},
149
+ journal={arXiv preprint arXiv:1803.08664},
150
+ year={2018}
151
+ }
152
+ ```
config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bam": true,
3
+ "data_parallel": false,
4
+ "model_type": "MDSR"
5
+ }
images/mdsr_2_4_compare.png ADDED
images/mdsr_4_4_compare.png ADDED
pytorch_model_4x.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a7962c20d2fd3e0a96129c666468b91f31894cda88a0260b494fca83b1f8a5f
3
+ size 11581573