닫기
Article Search
닫기

Original Article

Split Viewer

International Journal of Fuzzy Logic and Intelligent Systems 2020; 20(1): 8-16

Published online March 25, 2020

https://doi.org/10.5391/IJFIS.2020.20.1.8

© The Korean Institute of Intelligent Systems

Auto-Encoder Variants for Solving Handwritten Digits Classification Problem

Muhammad Aamir1, Nazri Mohd Nawi2, Hairulnizam Bin Mahdin1, Rashid Naseem3, and Muhammad Zulqarnain1

1Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Batu Pahat, Malaysia
2Soft Computing and Data Mining Center, Universiti Tun Hussein Onn Malaysia, Batu Pahat, Malaysia
3Department of IT and Computer Science, Pak-Austria Fachhochschule Institute of Applied Sciences and Technology, Haripur, Pakistan

Correspondence to :
Muhammad Aamir (amirr.khan1@gmail.com)

Received: November 30, 2019; Revised: February 20, 2020; Accepted: March 2, 2020

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Go to

Auto-encoders (AEs) have been proposed for solving many problems in the domain of machine learning and deep learning since the last few decades. Due to their satisfactory performance, their multiple variations have also recently appeared. First, we introduce the conventional AE model and its different variant for learning abstract features from data by using a contrastive divergence algorithm. Second, we present the major differences among the following three popular AE variants: sparse AE (SAE), denoising AE (DAE), and contractive AE (CAE). Third, the main contribution of this study is performing the comparative study of the aforementioned three AE variants on the basis of their mathematical modeling and experiments. All the variants of the standard AE are evaluated on the basis of the MNIST benchmark handwritten digit dataset for classification problem. The observed output reveals the benefit of using the AE model and its variants. From the experiments, it is concluded that CAE achieved better classification accuracy than those of SAE and DAE.

Keywords: Sparse auto-encoder (SAE), Denoising auto-encoder (DAE), Contractive auto-encoder (CAE), MNIST, Classification

Biographies

Go to

Muhammad Aamir has recently received his PhD in Information Technology from University Tunn Hussien Onn Malaysia. He did his Masters degree in Computer Science from City University of Science and Information Technology Pakistan. He had worked for two years in Xululabs LLC as data scientist. Currently he is working on research related to big data processing and data analysis. His fields of Interest are Data Science, Deep Learning, and Computer Programming.


Nazri Mohd Nazwi is Professor at the Faculty of Computer Science and Information Technology, University Tun Hussein Onn Malaysia. He obtained his PhD in Computer Science (Data mining) from Swansea University United Kingdom, his MSc in Computer Science from the University of Technology Malaysia. He has published more than 78 indexed journals and conference proceedings. His research interests are in the field of data analysis, database system, optimization methods and data mining techniques using Artificial Neural Network.


Hairulnizam Bin Mahdin is Associate Professor at the Faculty of Computer Science and Information Technology, University Tun Hussein Onn Malaysia. He obtained his PhD in Computer Science from Deakin University Australia, his MSc and BS in Computer Science from the University Putra Malaysia. His research interests are in the field of Data management, Database system, Internet of things and Big Data.


Rashid Naseem belongs to Landikotal, Khyber, KPK, Pakistan. He received the BCS degree in computer science from the University of Peshawar, Pakistan, in 2008 and the MPhil degree in computer science from the Quaid-i-Azam University, Pakistan, in 2011. He obtained PhD in Information Technology from the Universiti Tun Hussein Onn Malaysia in February 2017. He is currently Assistant Professor of Software Engineering at Pak-Austria Fachhochschule Institute of Applied Sciences and Technology, Mang Khanpur Road Haripur, Pakistan. His research interests include software modularization, architecture recovery, datamining and clustering techniques.


Muhammad Zulqarnain received his Bachelor and Master degree in Computer Science and Information Technology from The Islamia University of Bahawalpur (IUB), Pakistan. He received his M.Phil degree (Master of Philosophy) from National College of Business Administration and Economics, Lahore, Pakistan. He is currently pursuing Ph.D from University Tun Hussein Onn Malaysia. His research interest is Machine Learning and Deep learning for natural language processing and its application


Article

Original Article

International Journal of Fuzzy Logic and Intelligent Systems 2020; 20(1): 8-16

Published online March 25, 2020 https://doi.org/10.5391/IJFIS.2020.20.1.8

Copyright © The Korean Institute of Intelligent Systems.

Auto-Encoder Variants for Solving Handwritten Digits Classification Problem

Muhammad Aamir1, Nazri Mohd Nawi2, Hairulnizam Bin Mahdin1, Rashid Naseem3, and Muhammad Zulqarnain1

1Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Batu Pahat, Malaysia
2Soft Computing and Data Mining Center, Universiti Tun Hussein Onn Malaysia, Batu Pahat, Malaysia
3Department of IT and Computer Science, Pak-Austria Fachhochschule Institute of Applied Sciences and Technology, Haripur, Pakistan

Correspondence to:Muhammad Aamir (amirr.khan1@gmail.com)

Received: November 30, 2019; Revised: February 20, 2020; Accepted: March 2, 2020

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Auto-encoders (AEs) have been proposed for solving many problems in the domain of machine learning and deep learning since the last few decades. Due to their satisfactory performance, their multiple variations have also recently appeared. First, we introduce the conventional AE model and its different variant for learning abstract features from data by using a contrastive divergence algorithm. Second, we present the major differences among the following three popular AE variants: sparse AE (SAE), denoising AE (DAE), and contractive AE (CAE). Third, the main contribution of this study is performing the comparative study of the aforementioned three AE variants on the basis of their mathematical modeling and experiments. All the variants of the standard AE are evaluated on the basis of the MNIST benchmark handwritten digit dataset for classification problem. The observed output reveals the benefit of using the AE model and its variants. From the experiments, it is concluded that CAE achieved better classification accuracy than those of SAE and DAE.

Keywords: Sparse auto-encoder (SAE), Denoising auto-encoder (DAE), Contractive auto-encoder (CAE), MNIST, Classification

Fig 1.

Download
Figure 1.

Architecture of the standard AE.

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 2.

Download
Figure 2.

MNIST small subset (basic).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 3.

Download
Figure 3.

MNIST random-noise background digits (bg-rand).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 4.

Download
Figure 4.

MNIST rotation and image background digits (bg-img-rot).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 5.

Download
Figure 5.

ROC for the SAE based on a small subset (basic).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 6.

Download
Figure 6.

ROC for the SAE based on random noise background digits (bg-rand).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 7.

Download
Figure 7.

ROC for the SAE based on rotation and image background digits (bg-img-rot).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 8.

Download
Figure 8.

ROC for the DAE based on MNIST small subset (basic).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 9.

Download
Figure 9.

ROC for the DAE based on random noise background digits (bg-rand).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 10.

Download
Figure 10.

ROC for the DAE based on rotation and image background digits (bg-img-rot).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 11.

Download
Figure 11.

ROC for the CAE based on a small subset (basic).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 12.

Download
Figure 12.

ROC for the CAE based on random noise background digits (bg-rand).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Fig 13.

Download
Figure 13.

ROC for the CAE based on rotation and image background digits (bg-img-rot).

The International Journal of Fuzzy Logic and Intelligent Systems 2020; 20: 8-16https://doi.org/10.5391/IJFIS.2020.20.1.8

Table 1 . Performance evaluation of SAE, DAE, and CAE based on MNIST benchmark datasets.

Datasetsbasicbg-randbg-img-rot
ApproachExecution timeTraining errorTest errorExecution timeTraining errorTest errorExecution timeTraining errorTest error
SAE + Softmax24m 25s5.5913.5431m 25s8.9221.5440m 25s11.8726.58
DAE + Softmax31m 20s7.7315.7837m 20s10.3821.7845m20s17.7328.72
CAE + Softmax28m 55s3.3511.6532m 55s7.5820.6537m 55s11.0524.78

Algorithm 1. Auto-encoder.

Parameters initialization

- No. of hidden layers: h.

- Input feature set: [x1, x2, x3, …, xn].

- Encoding-activation function: EAF.

- Decoding-activation function: DAF.

- Inputs weights: Wi.

- Biasness values: bi.

Encoding

- Compute encoded inputs f(w) by multiplying xn and Wi.

- Compute biased inputs f(b) by adding bI to the encoded inputs.

- Compute f(x) using Eq. (1) by applying f(w) and f(b).

Decoding

- Compute decoded outputs f(w′) by multiplying xn and Wi.

- Compute biased outputs f(b′) by adding bI to the decoded outputs.

- Compute g(y) using Eq. (2) by applying f(w′) and f(b′).

Optimization

- Optimize the value of Eq. (5).


Article Tools

Metrics

Share this article on :

Related articles in IJFIS

Most KeyWord