As living standards improve, demand for high-quality products in all fields, including video images, is increasing. High-quality video means a high resolution and good color. Compression technology is essential for the efficient transmission of high-resolution image data, and a method of restoring information lost during the compression process is required. In addition, medical images, CCTV, and thermal imaging, among other imaging types, do not have a high resolution because of the hardware limitations, and it is difficult to provide an image quality desired by users. An ultra-high-resolution restoration is therefore required to provide a good quality image. Single image super-resolution (SISR) is a technique for restoring a down-sampled image to an original super-resolution image.

Ultra-high super-resolution has made significant progress through recent deep learning technologies [1–4]. However, most studies have focused only on improving the performance of the complete image [5,6], and few studies have considered the image texture [7]. In particular, when an image is simply divided into high- and low-frequency regions and compressed, it is difficult to restore the image because the loss in the high-frequency region is large. In addition, the higher the frequency, the more important the information is, and it can be stated that restoring a high-frequency region well is more important than improving the overall performance.

We use a super-resolution generative adversarial network (SRGAN) [7] as the base model, and because SENet using squeeze and excitation (SE) blocks is superior to ResNet using residual blocks, we change the residual modules of SRGAN into a module network, SB blocks, or a combination of the two to improve the restoration performance of the texture area [4]. The remainder of this paper is organized as follows. Section 2 describes existing studies on high-resolution restoration methods. We propose the three models in Section 3. Section 4 describes the experimental results and an analysis of the proposed methods, and finally, Section 5 presents some concluding remarks and areas of future research.

One way to obtain a high-resolution image is to physically increase the number of pixels in the image sensor. However, this increases the size of the image sensor, which is not economical and makes it difficult to handle. There is another way to reduce the size of a single pixel to fit many pixels in a certain area. However, if the size of the pixel is restored as much as it is reduced, the quality of the final image deteriorates [8].

Therefore, research on the restoration of ultra-high-resolution images using software has been developed, and active research in this area is still ongoing [9]. Ultra-high-resolution image restoration can be broadly divided into multi-frame-based and single-image-based research. In the case of multi-frame-based research [10, 11], it is possible to restore a super-resolution image by complementing the information acquired for each frame; however, a low-resolution image may not be able to be restored, or the quality of the restored result may degrade owing to insufficient information. In the early stage of SISR image restoration studies, although research based on linear, bicubic, and Lanczon filters with a high execution speed were conducted [12], the resulting images were slightly blurred. To overcome these shortcomings, research on preserving the edges has been introduced [9].

With the recent introduction of deep learning technology, ultra-high-resolution images have been successfully restored. The relationship between high and low resolutions has been discovered in the same image. In particular, as shown in Figure 1, a convolutional neural network CNN [9], a very deep super-resolution (VDSR) [10], and a road structure refined CNN (RSRCNN) [13], and a deeply-recursive convolutional network (DRCN) [14], which are based on CNN [15–17], performed better than existing networks. The difference between a low-resolution image obtained using an existing interpolation method and a high-resolution image obtained from these models is shown in Figure 2 [9].

However, CNN-based algorithms using the mean squared error (MSE) based loss functions are still insufficient in representing the textures of ultra-high-resolution images. To address this issue, it is applied to restoring ultra-high-resolution images using SRGAN. An SRGAN is a super-resolution image reconstruction method based on a generative adversarial network [18].

2.1 Super Resolution GAN

Similar to other GAN models, a SRGAN consists of a generative model and a discriminative model. The SRGAN generates fake images similar to the real image from random noise in the generative model, and simultaneously learns whether the image generated in the discriminative model is real or fake through the process of discrimination. The discriminant model at the beginning of the learning easily determines the authenticity of the image generated by the generative model; however, as the learning progresses, the generative model generates an image that the discriminant model cannot distinguish.

As such, two neural networks with different purposes can be used to construct a system that generates an image similar to the real one. Existing super-simple image resolution techniques have difficulty in restoring texture, which provides detailed information about high-frequency components.

However, an SRGAN provides a solution to this issue. The structure of the SRGAN used is shown in Figure 3. To increase the accuracy, the network consists of a deep model, batch normalization, and skip connection and is trained using perceptual loss. Perceptual loss is the weighted sum of the content and adversarial losses. The VGG loss function is used as the content loss to improve the image texture expression. The core idea of the VGG loss is to find the difference between the image generated by the generator and the original image, not from the resulting images, but after passing through the feature map.

The goal of the proposed system is to learn the texture information. Because most texture regions are high-frequency signals based on edges, the high-frequency information is approximated through residual learning. In this section, a brief description of a residual network and SENet is provided. In addition, the structure of the proposed method for increasing the effective performance through residual learning is described.

3.1 Residual Network

ResNet [19] was developed to solve the problem in which the deeper the neural network depth is, the more difficult the learning becomes. Therefore, by learning the residuals, deep networks can be learned more easily than with GoogleNet [20, 21] and VGGNet [22]. In addition, it is easy to optimize the network, and a high accuracy is obtained through a deep model [23–25].

After learning and evaluating ImageNet using ResNet, the result won the ILSVRC 2015 challenge with a very small error of 3.57%, and a good performance was demonstrated in the detection and segmentation of the CIFAR-10 and COCO datasets. Residual learning can be considered an ensemble approach in which the identity mapping between layers and weights learned from shallow models is combined. This is also similar to adjusting the reflection ratio of the old and new memories in long short-term memory. Residual learning can be seen as approximating information between pixels, which improves the restoration performance [26, 27].

3.2 Squeeze and Excitation Block

In this study, the SE block, which is applied in SENet [28], was used to improve the performance. The SE block can be applied to any existing model, as shown in Figure 4. If the SE block shown in Figure 3 is added to VGGNet, GoogLeNet, and ResNet, among others, the performance will increase. Although the performance is greatly improved, the number of parameters does not increase significantly, and thus the increase in the number of computations is not large.

In general, to improve the performance, the number of computations increases by that amount. In the case of SENet, use of the SE block is efficient because the accuracy can be improved without significantly increasing the number of computations.

3.3 Proposed Module

The structure of the system module suggested in this study is shown in Figure 5. We applied the residual and SE blocks separately to the SRGAN and verified how the network affects the super-resolution by applying it concurrently. Figure 5(a) shows the conventional ResNet, and Figure 5(b) shows ResNet with the addition of an SE block. As mentioned earlier, this aims to maximize the texture information and minimize the smoothness. Figure 5(c) shows the addition of the term of the residual block to Figure 5(a) for use in learning when considering the characteristics at different high frequencies. Figure 5(d) shows the addition of the SE block to the ResNet shown in Figure 5(c).

4.1 Experiment Details

The experimental environment used in this study was as follows. The hardware consists of an Intel Xeon Gold 5120 CPU, 64 GB of RAM, and GPGPU is the RTX TITAN X of 24 GB. The deep learning framework is TensorFlow 1.14.0 in Ubuntu 18.04.4, and the dataset used for training is DIV2K 2017 [29]. The DIV2K data consist of high-resolution images with a pixel resolution of 2044×1404 and low-resolution images corresponding to each image. And the training set consists of 800 images, and the test set consists of 100 images.

To form the model, we apply 200 epochs and use the Adam optimizer with a learning rate of 0.0001 and a momentum of 0.9. This model has a batch size of 16. To be used for learning, the generator restores the reduced images (96 × 96 in size) to a pixel resolution of 384 × 384, and the discriminator determines the similarity by inputting the output of the generator and the high-resolution image, executing a binary classification to classify it into low- and high-resolution images, with learning progressing as a single network connecting the generator and discriminator. In the discriminator, the original high-resolution image and the resulting super-resolution image are compared and evaluated, and learning continues. During the learning, the sigmoid function included at the end of the discriminator is removed, and the loss function is calculated. The final loss value is calculated by adding the contradictory loss value calculated in the discriminator to the content loss value generated by the independent VGG at a given ratio. The builder’s formula and the SRGAN identification formula are shown in Equations (1) and (2) [7].

In the above equation, G is a generator receiving a low-resolution image input and therefore generates a super-resolution image. In addition, θ_G is the value of the G parameter, and the generated image is the value of the loss function determined by comparing it to the high-resolution image. Moreover, D is the generated image, which is a high-resolution image or a generated image, and θ_G denotes the augmentation of D.

To quantitatively measure the image quality, we used the peak signal-to-noise ratio (PSNR), which evaluates the image quality loss information; the structural similarity (SSIM); the visual information fidelity (VIF), which evaluates the accuracy of the visual information; and the relative polar edge coherence (RECO), which evaluates the similarity using polar coordinates, as in Equations (3)–(6) [30].

In Equation (3), MAX_I is the maximum pixel value of the image. In the case of an 8-bit gray image, it was 255. The higher the MSE is, the better the video quality.

The SSIM index is obtained by multiplying the brightness comparison, contrast comparison, and structure comparison values of the actual and reconstructed images. Because the human visual system is specialized in deriving the structural information of an image, as the key hypothesis, if the structural information is distorted, the perception will be significantly affected. In Equaton (4), μ_i is the average brightness, and σ_i is the variance of image i. In addition, C_i is a stabilization constant that supports a situation in which the discriminant has a value of close to zero. Moreover, (2μxμy+C1)(μx2+μy2+C1) is an equation comparing the brightness, and (2σxσy+C2)(σx2+σy2+C2) is the contrast comparison. In addition, (σxy+C3)(σxσy+C3) compares the structures of the two images.

VIF calculates the mutual information between C, which is the original image, and E, which is an image recognized by a human. That is the entropy they share. It also calculates the amount of mutual information between D, which is a distorted image, and F, which is recognized as a distorted image. The quality of the distorted image is then predicted by calculating the ratio between the two, as shown in Equation (5).

The ECO metric is an absolute measure, and thus it might be used in principle for no-Reference quality estimation, provided that a quality unit is defined. However, ECO suffers from the fact that it depends on the image content. To compensate for this, the RECO index, the ratio between the images I_(x1,x2) and a reference image Ī_(x1,x1) of the same scene, is defined through Equation (6).

4.2 Experiment Result

Table 1 shows the performance evaluation for each model, and as a result of the experiment, it can be confirmed that the residual and SE blocks are helpful in improving the performance. In particular, when using the residual and SE blocks, the performance can still be improved. As shown in Table 2, it was possible to see an increase in the operating speed of 1.1- to 2-fold compared to the existing SRGAN; however, the performance was improved, and the performance changed owing to the addition of blocks.

Thereafter, six images were tested with the proposed model, as shown in Table 2. In Table 2, Image #0 is the result of measuring the image quality of the penguin in Figure 6. In the case of Figure 5, the VIF obtained by the proposed method is shown to be the best, and it can be seen that the beak of the penguin in Figure 6 is well restored. In addition, #45 indicates the image in Figure 7, where it can be seen that the proposed the method is similar to the real image and has the smallest grid pattern.

Image #47 is shown in Figure 8, and in the case of the improved residual, a grating pattern is found, although not when applying the proposed method; in addition, the performance evaluation is also high. In the case of #62, as shown in Figure 9, the proposed method achieves the best results for pattern restoration of the hair, and the performance is also high.

In the case of #69, as shown in Figure 10, the performance was better than the proposed method when the residual improved because the pattern of hair was constant. For #82, as shown in Figure 11, the proposed method achieves the best quality evaluation, and the texture restoration is also the highest. Therefore, when using a residual block with an SE block, it was confirmed that the removal of the grid pattern results in a higher restoring force based on the similarity of the pixels.

Super-resolution image restoration is a major issue in the modern era of the storage and exchanging of image data. In this study, an SRGAN, which has contributed to this area, and three proposed models were tested using the DIV2K dataset to achieve an ultra-high-resolution restoration using a single image. A performance evaluation was conducted for PSNR, SSIM, VIF, and RECO. For the entire dataset, it depends on the model, but on average, 0.01% of VIF and SSIM and 1% of PSNR were better. Six images showed differences in the image quality according to the numerical results. However, RECO achieved worse results than the base model. With this, we know that the SE block improves the performance. The SRGAN+Residual+SE model is said to be the best because it takes a longer time than the other approaches. Therefore, the SRGAN+SE or SRGAN+Residual models can be used. However, there was a problem in that a smoothing phenomenon occurred again when the layer was deepened.

In future research, we will consider using SRGAN+SE or a combined SRGAN+Residual Markov random field to the smoothing problem for improving the restoration performance and restoring the resolution of the depth image.

Table 1. Image quality measurement from model of DIV2K 2017 dataset (unit: dB).

Table 2. Image quality measurement from DIV2K 2017 dataset.