Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang
We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Super-Resolution | Set5 - 4x upscaling | PSNR | 30.49 | SRCNN |
| Super-Resolution | Set5 - 4x upscaling | SSIM | 0.8628 | SRCNN |
| Super-Resolution | Set14 - 4x upscaling | PSNR | 27.5 | SRCNN |
| Super-Resolution | Set14 - 4x upscaling | SSIM | 0.7513 | SRCNN |
| Super-Resolution | FFHQ 256 x 256 - 4x upscaling | FID | 147.21 | SRCNN |
| Super-Resolution | FFHQ 256 x 256 - 4x upscaling | MS-SSIM | 0.9 | SRCNN |
| Super-Resolution | FFHQ 256 x 256 - 4x upscaling | PSNR | 23.12 | SRCNN |
| Super-Resolution | FFHQ 256 x 256 - 4x upscaling | SSIM | 0.688 | SRCNN |
| Super-Resolution | IXI | PSNR 2x T2w | 37.32 | SRCNN |
| Super-Resolution | IXI | PSNR 4x T2w | 29.69 | SRCNN |
| Super-Resolution | IXI | SSIM 4x T2w | 0.9052 | SRCNN |
| Super-Resolution | IXI | SSIM for 2x T2w | 0.9796 | SRCNN |
| Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | FID | 31.84 | SRCNN |
| Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | MS-SSIM | 0.924 | SRCNN |
| Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | PSNR | 27.4 | SRCNN |
| Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | SSIM | 0.801 | SRCNN |
| Super-Resolution | Manga109 - 4x upscaling | PSNR | 27.58 | SRCNN |
| Super-Resolution | Manga109 - 4x upscaling | SSIM | 0.8555 | SRCNN |
| Super-Resolution | Urban100 - 4x upscaling | PSNR | 24.52 | SRCNN |
| Super-Resolution | Urban100 - 4x upscaling | SSIM | 0.7221 | SRCNN |
| Super-Resolution | BSD100 - 4x upscaling | PSNR | 26.9 | SRCNN |
| Super-Resolution | BSD100 - 4x upscaling | SSIM | 0.7101 | SRCNN |
| Super-Resolution | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| Super-Resolution | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| Super-Resolution | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| Super-Resolution | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| Super-Resolution | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| Super-Resolution | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| Super-Resolution | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| Super-Resolution | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 3D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 3D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 3D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 3D Human Pose Estimation | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 3D Human Pose Estimation | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 3D Human Pose Estimation | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 3D Human Pose Estimation | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 3D Human Pose Estimation | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| Video | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| Video | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| Video | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| Video | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| Video | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| Video | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| Video | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| Video | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| Pose Estimation | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| Pose Estimation | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| Pose Estimation | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| Pose Estimation | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| Pose Estimation | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| Pose Estimation | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| Pose Estimation | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| Pose Estimation | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 3D | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 3D | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 3D | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 3D | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 3D | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 3D | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 3D | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 3D | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 3D Face Animation | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 3D Face Animation | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 3D Face Animation | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 3D Face Animation | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 3D Face Animation | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 3D Face Animation | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 3D Face Animation | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 3D Face Animation | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| Image Super-Resolution | Set5 - 4x upscaling | PSNR | 30.49 | SRCNN |
| Image Super-Resolution | Set5 - 4x upscaling | SSIM | 0.8628 | SRCNN |
| Image Super-Resolution | Set14 - 4x upscaling | PSNR | 27.5 | SRCNN |
| Image Super-Resolution | Set14 - 4x upscaling | SSIM | 0.7513 | SRCNN |
| Image Super-Resolution | FFHQ 256 x 256 - 4x upscaling | FID | 147.21 | SRCNN |
| Image Super-Resolution | FFHQ 256 x 256 - 4x upscaling | MS-SSIM | 0.9 | SRCNN |
| Image Super-Resolution | FFHQ 256 x 256 - 4x upscaling | PSNR | 23.12 | SRCNN |
| Image Super-Resolution | FFHQ 256 x 256 - 4x upscaling | SSIM | 0.688 | SRCNN |
| Image Super-Resolution | IXI | PSNR 2x T2w | 37.32 | SRCNN |
| Image Super-Resolution | IXI | PSNR 4x T2w | 29.69 | SRCNN |
| Image Super-Resolution | IXI | SSIM 4x T2w | 0.9052 | SRCNN |
| Image Super-Resolution | IXI | SSIM for 2x T2w | 0.9796 | SRCNN |
| Image Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | FID | 31.84 | SRCNN |
| Image Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | MS-SSIM | 0.924 | SRCNN |
| Image Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | PSNR | 27.4 | SRCNN |
| Image Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | SSIM | 0.801 | SRCNN |
| Image Super-Resolution | Manga109 - 4x upscaling | PSNR | 27.58 | SRCNN |
| Image Super-Resolution | Manga109 - 4x upscaling | SSIM | 0.8555 | SRCNN |
| Image Super-Resolution | Urban100 - 4x upscaling | PSNR | 24.52 | SRCNN |
| Image Super-Resolution | Urban100 - 4x upscaling | SSIM | 0.7221 | SRCNN |
| Image Super-Resolution | BSD100 - 4x upscaling | PSNR | 26.9 | SRCNN |
| Image Super-Resolution | BSD100 - 4x upscaling | SSIM | 0.7101 | SRCNN |
| 2D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 2D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 2D Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 2D Human Pose Estimation | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 2D Human Pose Estimation | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 2D Human Pose Estimation | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 2D Human Pose Estimation | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 2D Human Pose Estimation | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 3D Absolute Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 3D Absolute Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 3D Absolute Human Pose Estimation | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 3D Absolute Human Pose Estimation | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 3D Absolute Human Pose Estimation | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 3D Absolute Human Pose Estimation | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 3D Absolute Human Pose Estimation | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 3D Absolute Human Pose Estimation | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| Video Super-Resolution | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| Video Super-Resolution | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| Video Super-Resolution | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| Video Super-Resolution | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| Video Super-Resolution | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| Video Super-Resolution | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| Video Super-Resolution | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| Video Super-Resolution | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 3D Object Super-Resolution | Set5 - 4x upscaling | PSNR | 30.49 | SRCNN |
| 3D Object Super-Resolution | Set5 - 4x upscaling | SSIM | 0.8628 | SRCNN |
| 3D Object Super-Resolution | Set14 - 4x upscaling | PSNR | 27.5 | SRCNN |
| 3D Object Super-Resolution | Set14 - 4x upscaling | SSIM | 0.7513 | SRCNN |
| 3D Object Super-Resolution | FFHQ 256 x 256 - 4x upscaling | FID | 147.21 | SRCNN |
| 3D Object Super-Resolution | FFHQ 256 x 256 - 4x upscaling | MS-SSIM | 0.9 | SRCNN |
| 3D Object Super-Resolution | FFHQ 256 x 256 - 4x upscaling | PSNR | 23.12 | SRCNN |
| 3D Object Super-Resolution | FFHQ 256 x 256 - 4x upscaling | SSIM | 0.688 | SRCNN |
| 3D Object Super-Resolution | IXI | PSNR 2x T2w | 37.32 | SRCNN |
| 3D Object Super-Resolution | IXI | PSNR 4x T2w | 29.69 | SRCNN |
| 3D Object Super-Resolution | IXI | SSIM 4x T2w | 0.9052 | SRCNN |
| 3D Object Super-Resolution | IXI | SSIM for 2x T2w | 0.9796 | SRCNN |
| 3D Object Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | FID | 31.84 | SRCNN |
| 3D Object Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | MS-SSIM | 0.924 | SRCNN |
| 3D Object Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | PSNR | 27.4 | SRCNN |
| 3D Object Super-Resolution | FFHQ 1024 x 1024 - 4x upscaling | SSIM | 0.801 | SRCNN |
| 3D Object Super-Resolution | Manga109 - 4x upscaling | PSNR | 27.58 | SRCNN |
| 3D Object Super-Resolution | Manga109 - 4x upscaling | SSIM | 0.8555 | SRCNN |
| 3D Object Super-Resolution | Urban100 - 4x upscaling | PSNR | 24.52 | SRCNN |
| 3D Object Super-Resolution | Urban100 - 4x upscaling | SSIM | 0.7221 | SRCNN |
| 3D Object Super-Resolution | BSD100 - 4x upscaling | PSNR | 26.9 | SRCNN |
| 3D Object Super-Resolution | BSD100 - 4x upscaling | SSIM | 0.7101 | SRCNN |
| 3D Object Super-Resolution | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 3D Object Super-Resolution | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 3D Object Super-Resolution | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 3D Object Super-Resolution | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 3D Object Super-Resolution | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 3D Object Super-Resolution | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 3D Object Super-Resolution | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 3D Object Super-Resolution | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 1 Image, 2*2 Stitchi | MSU Video Upscalers: Quality Enhancement | PSNR | 26.68 | SRCNN |
| 1 Image, 2*2 Stitchi | MSU Video Upscalers: Quality Enhancement | SSIM | 0.929 | SRCNN |
| 1 Image, 2*2 Stitchi | MSU Video Upscalers: Quality Enhancement | VMAF | 51.21 | SRCNN |
| 1 Image, 2*2 Stitchi | Ultra Video Group HD - 4x upscaling | Average PSNR | 37.52 | SRCNN |
| 1 Image, 2*2 Stitchi | Xiph HD - 4x upscaling | Average PSNR | 31.47 | SRCNN |
| 1 Image, 2*2 Stitchi | Vid4 - 4x upscaling | MOVIE | 6.9 | SRCNN |
| 1 Image, 2*2 Stitchi | Vid4 - 4x upscaling | PSNR | 24.68 | SRCNN |
| 1 Image, 2*2 Stitchi | Vid4 - 4x upscaling | SSIM | 0.7158 | SRCNN |
| 16k | Set5 - 4x upscaling | PSNR | 30.49 | SRCNN |
| 16k | Set5 - 4x upscaling | SSIM | 0.8628 | SRCNN |
| 16k | Set14 - 4x upscaling | PSNR | 27.5 | SRCNN |
| 16k | Set14 - 4x upscaling | SSIM | 0.7513 | SRCNN |
| 16k | FFHQ 256 x 256 - 4x upscaling | FID | 147.21 | SRCNN |
| 16k | FFHQ 256 x 256 - 4x upscaling | MS-SSIM | 0.9 | SRCNN |
| 16k | FFHQ 256 x 256 - 4x upscaling | PSNR | 23.12 | SRCNN |
| 16k | FFHQ 256 x 256 - 4x upscaling | SSIM | 0.688 | SRCNN |
| 16k | IXI | PSNR 2x T2w | 37.32 | SRCNN |
| 16k | IXI | PSNR 4x T2w | 29.69 | SRCNN |
| 16k | IXI | SSIM 4x T2w | 0.9052 | SRCNN |
| 16k | IXI | SSIM for 2x T2w | 0.9796 | SRCNN |
| 16k | FFHQ 1024 x 1024 - 4x upscaling | FID | 31.84 | SRCNN |
| 16k | FFHQ 1024 x 1024 - 4x upscaling | MS-SSIM | 0.924 | SRCNN |
| 16k | FFHQ 1024 x 1024 - 4x upscaling | PSNR | 27.4 | SRCNN |
| 16k | FFHQ 1024 x 1024 - 4x upscaling | SSIM | 0.801 | SRCNN |
| 16k | Manga109 - 4x upscaling | PSNR | 27.58 | SRCNN |
| 16k | Manga109 - 4x upscaling | SSIM | 0.8555 | SRCNN |
| 16k | Urban100 - 4x upscaling | PSNR | 24.52 | SRCNN |
| 16k | Urban100 - 4x upscaling | SSIM | 0.7221 | SRCNN |
| 16k | BSD100 - 4x upscaling | PSNR | 26.9 | SRCNN |
| 16k | BSD100 - 4x upscaling | SSIM | 0.7101 | SRCNN |