Sound can be described as pressure caused by the vibration of an object. Sound is an important aspect in human relationships and communication. Recent research on sound recognition has received considerable attention because it is used in various fields such as bio-acoustic monitoring, wild animal intrusion detection, environmental sound, and audio surveillance [1]. The sound-recognition process includes three steps: signal preprocessing, feature extraction, and classification [2]. This study is focused on snoring sound classification because snoring has a significant impact on sleep health.

Sleep is unique to humans and animals. A phenomenon called snoring occurs when there is a sleep disorder. Snoring is classified as primary snoring, which involves no comorbidity, and obstructive sleep apnea (OSA), which poses health risks [3]. Snoring noise is estimated to reach approximately 90 to 100 dB, which can cause hearing loss in people next to the snoring person [4]. OSA is a sleep-related respiratory disorder that repeatedly causes a partial or complete obstruction of the upper airways during sleep [5]. An easy way to diagnose OSA is to use polysomnography; however, this method is expensive [6]. Signal processing and artificial intelligence (AI) are the most promising approaches among the existing methods for diagnosing snoring, as in the case of apnea [7]. In the AI field, classifiers such as a support vector machine (SVM), K-nearest neighbor (KNN), recurrent neural network (RNN), and convolutional neural network (CNN) are appropriate solutions. However, these classifiers rely on the feature extraction techniques used in signal processing to extract features from the snoring audio signal to conduct an accurate classification. In this case, snoring data and feature extraction are the most important factors because they are required to train the classifier. Several approaches to snoring dataset acquisition and frequently used feature extraction methods are described in [7]. According to the dataset made available by the INTERSPEECH 2017 ComParE Snoring Challenge [8], OSA is categorized into four classes based on the source of obstruction that leads to the snoring: velum (V), oropharyngeal (O), tongue (T), and epiglottis (E). Distinguishing background noise from sleep sounds (e.g., sleep talking), which are considered noise from real snoring sounds, is a subject that needs to be addressed. Addressing this challenge using a deep learning model will facilitate a more accurate classification/detection that distinguishes snoring sounds from non-snoring sounds. The authors in [9] stressed that this step is necessary but insufficient and further emphasized the need for the right choice of feature extraction techniques to acquire a desirable result.

In this paper, we presented a one-dimensional (1D) CNN for snoring and non-snoring sound classification based on several signal processing approaches for feature extraction. The main contribution of this study is to overcome the challenges mentioned above and provide a deep learning model that serves as an alternative method for distinguishing snoring sounds with an extremely high classification accuracy. The contributions of this study are summarized as follows:

• We generated spectrogram images from our dataset and fed them into the 2D-CNN network; however, the resulting accuracy was poor.

• We extracted unique features from our snoring/non-snoring sound dataset according to their importance in boosting the classification accuracy.

• The extracted features served as a new dataset (numerical data) and were fed into a 1D-CNN network. The resulting output was significant.

• Finally, we present the research results obtained by comparing and evaluating the proposed method with other existing approaches.

The rest of this paper is organized as follows: Section 2 describes previous related studies, including the feature extraction process and classification methods. Sections 3 and 4 present the proposed approach and experiment results and discussion, respectively. Finally, Section 5 provides some concluding remarks regarding this study.

To achieve this goal, a detailed review of related studies was conducted. First, we describe the snoring data acquisition methods. Based on previous research [7], snoring data sources can be categorized into four categories: online available snore sound corpora, snore data provided by medical organizations, snore data from subjects, and crowdsourced snore data. A typical example of an online snoring sound corpus is the Munich-Passau Snore Sound Corpus, which consists of 828 snore audio samples grouped into four categories: V, O, T, and E [8]. Some researchers have collected snore data from patients with apnea in hospitals or medical organizations [10]. Snoring audio can be captured from people who have consented to be subjected to sleep or other forms of sleep-induced substances to record snoring activity [11]. Finally, there is a process of collecting/aggregating snoring and related snore data from different online sources [7]. In this study, we plan to use the third method to collect snoring data directly from the subjects.

Conventional classification methods include SVM, KNN, CNN, RNN, and their hybrids. An SVM was originally developed to classify the data into two classes. According to the authors in [12], it is regarded as a technique used for the classification of linear and nonlinear datasets. A KNN searches for an n-dimensional pattern space using the training data and classifies them into k numbers based on the nearest neighbors [13]. An RNN are fashioned to learn from sequential information. One issue with an RNN is the vanishing or exploding gradient problem. This problem arises because of the long input-output sequence. Long short-term memory (LSTM) was designed to overcome this problem. LSTM works by allowing input x(t) at time t to influence the storage or overwriting of memories stored in the cell. CNN models are a type of deep neural network developed for image recognition, such that the model is fed with the image pixels as a 2D input. A CNN is based on the convolution of images and the extraction of salient features through filters that are learned by the network during the training phase. A 1D-CNN was cited in [14] as having less computational complexity over a 2D-CNN, having a shallow architecture with the potential to learn complex features, requiring fewer computational resources (using a CPU rather than a GPU), and being well suited for real-time and low-cost applications on hand-held devices.

Figure 1 shows the generic architecture for the snoring sound classification process [7]. In previous studies, snoring signals were converted into spectrogram images, and feature extraction techniques were then applied. Finally, the extracted features were passed into a classifier for classification.

Several approaches have been proposed for classifying snoring audio sounds. Many of these approaches are based on either the application of feature extraction techniques along with classifier(s), or the direct use of classifier(s) alone. A new rapidly developing field called music information retrieval (MIR), which encompasses musicology, digital signal processing, and machine learning (ML) [15], has recently gained the interest of researchers. In the literature, the approaches used in the field of MIR are efficient for extracting salient features from sound signals. These include Mel-frequency cepstral coefficient (MFCC), RollOff, short-time Fourier transform (STFT), spectral features, zero-crossing rate (ZCR), local binary pattern (LBP), histogram of oriented gradient (HOG), and pulse transition time (PTT). In this study, we combined several of these approaches to extract features from our data before training with a CNN. The results obtained prove the efficiency and effectiveness of MIR techniques.

In recent years, deep neural networks such as AlexNet and region-based CNN (R-CNNs) have been proposed to overcome the limitations of classic neural networks [16]. In this study, a new CNN was designed to classify sounds using the extracted feature sets. The CNN architecture used to classify the snoring and non-snoring feature datasets consists of four convolutional layers, a max-pooling layer, a dropout, and a softmax activation function.

3.1 Data Acquisition and Feature Extraction

We propose a 1D-CNN model for snoring sound classification based on multi-feature extraction. The essence of the multi-feature extraction process is the extraction of salient features that distinctly distinguish sound signals from closely related signals, which may not achievable using a single extraction technique. In this study, seven features were applied: STFT, root mean square (RMS), spectral centroid, bandwidth, RollOff, ZCR, and MFCC. The overall architecture of the proposed system is shown in Figure 2.

The authors in [7] identified the four most common categories of snoring data acquisition as a publicly available snore sound corpus, snore data provided by a medical organization, snore data created through subjects, and crowdsourced snore data. We adopted the third category by creating snoring data from six individuals [17], i.e., males between 25 and 30 years in age. Two Samsung Galaxy S8 phones were placed 1 m away from the head of the subjects, and recording was conducted during active sleeping hours between 11:30 pm and 5:30 am for two nights. All recorded snoring audio clips were divided into sections having a length of 40 second and sampled at 44.1 kHz; hence, a total of 1,080 snoring episodes were generated. However, owing to the presence of sections without audio, the size of the snoring episodes was reduced to 881. A total of 880 segmented non-snoring audios, including clock ticks, sleep-talk, yawning, pet snoring, door sound, and bed spring sound, were introduced, and snoring audios were labeled as 1, whereas the non-snoring audios were labeled as 0, forming a total data size of 1,761 [17].

A graphical image of the data acquisition is shown in Figure 3. Next, we transformed the audio into spectrogram images and built a 2D-CNN to train them. However, the results were not encouraging. Hence, we proceeded to extract features from the audio dataset, resulting in a new set of data. This feature set led to the use a 1D-CNN network because the dataset contained numeric values in one-dimension. The Excel chart at the bottom of Figure 3 shows the cross-section of the extracted features.

Seven feature extraction approaches were used to generate the feature dataset, as shown in Figure 3. This feature set was fed into a 1D-CNN for classification. The energy of an audio signal is referred to as the total magnitude or loudness of the signal x(n), and is represented as

The RMS is used to characterize the average of a continuously varying audio signal, and its value represents the total waveform. The RMS energy for each frame of the audio signal is defined as

where n denotes the number of waveforms in a frame. The frame length was set to 2048, and the hop length was set to 512. The essence of the RMS is enabling the detection of a sudden increase in audio signal energy, which helps to provide a clear distinction between silence or a low signal mode and voiced mode.

The MFCC of a signal is a set of audio signal features that provide a descriptive overview of a spectral envelop [7]. The number of MFCCs was set to 20 and the discrete cosine transform (DCT) type was set to 2. DCT is a function that logs energy from the filter bank and transforms it from the frequency domain to the time domain. In addition, the Mel filter bank estimates the amount of energy that exists at low frequencies.

The Mel frequency, Mel(f), can be defined as follows:

where f is the linear scale frequency, and m is the number of MFCCs. Figure 4 shows an MFCC plot with 20 features in the time series. The x-axis in Figure 4 is measured in seconds, whereas the y-axis is measured in decibels.

According to the authors of [18], the STFT of a signal for time t is computed as

where ω(t − τ) is a window function positioned at τ on the time axis and can be expressed in a Gaussian form.

The spectral centroid identifies the point where the center of mass of an audio sound exists by estimating the weighted mean of the frequencies within the audio signal. In the computation of the spectral centroid, each frame magnitude signal is normalized as follows:

where x(k) represents the discrete audio signal, and f(k) is the frequency at bin k. Hence, the spectral centroid is computed as [1]

The spectral bandwidth expresses the spectral spread of an audio signal. The spectral bandwidth at frame t is calculated as [1]

where x̃[k, t] is the spectral magnitude in frequency bin k, [k, t] is the frequency in bin k, and Sc[k] is the spectral centroid.

The spectral RollOff frequency is an audio feature that specifies the frequency that contains the majority of the total energy spectrum. According to the Librosa (a Python library for analyzing music and audio) documentation, distinct frames are defined as the center frequency for a spectrogram bin that contains at least 85% of the energy of the spectrum in a frame [19].

The ZCR is the rate of change of the signal sign from positive to negative, or vice versa. The ZCR splits a signal into K frames such that {f_i(n) : 1 ≤ i ≤ K}, and for each frame, zcr_i is computed as [1]

where r is the number of samples in each frame and

These seven features are representative features frequently used in signal processing.

3.2 Deep Learning Model and Implementation

We designed a new CNN model to classify snoring data based on multi-feature extraction. To achieve this goal, we utilized TensorFlow, an open-source library developed by Google that primarily focuses on deep learning. It has a comprehensive and flexible ecosystem of tools, libraries, and community resources that allows researchers and developers to easily build and deploy state-of-the-art ML-powered applications. TensorFlow offers multiple levels of abstraction, building, and training models using the high-level Keras API, which makes it easy to get started with TensorFlow and ML. Together with TensorFlow 2.0, Keras has been adopted than any other deep-learning solutions. We used the Keras and TensorFlow libraries for the 1D-CNN and 2D-CNN experiments, respectively. In this study, experiments were conducted using 2D-CNN and 1D-CNN training approaches with different hardware system configurations. We utilized Intel Core i7@3.60 GHz CPU and 32 GB RAM for the 1D-CNN, and another system with a hardware configuration of a single NVIDIA GTX 1080Ti 16 GB GPU, Intel Core i7@3.60 GHz CPU and 32 GB RAM, as in the case of the 2D-CNN. Figure 5 shows conceptual diagrams of the 1D-CNN and 2D-CNN models used in this study.

4.1 Experiments and Results

Initially, we experimented by converting our audio dataset into spectrogram images. The resulting images were then sent to a 2D-CNN classifier. A typical architecture of the proposed 2D-CNN classifier is shown in Figure 6. The input size of the classifier was 32×32×3, and the three convolution layers had ReLU activations. In front of each layer, a max-pooling layer reduces the spatial dimension. In addition, the classifier also had two fully connected layers, and batch normalization was used after each of these layers. We applied a softmax activation function to the second fully connected batch normalization layers. The Adam optimizer was also applied to optimize the loss based on the output of the cross-entropy loss function. Both the number of epochs and batch sizes were 100, and the learning rate was 0.0005. Consequently, an accuracy of 86.80% was achieved, as shown in Figure 7. However, their performance was unsatisfactory. Therefore, we attempted to improve the results using the feature extraction process.

We applied multi-feature extraction technology to significantly improve the classification accuracy. At the end of this process, the extracted feature data are forwarded to the 1D-CNN classifier, as shown in Figure 8. The generated feature data consisted of 26 columns and 1,761 rows. Feature data were divided into 80% for training and 20% for testing. In the proposed 1D-CNN shown in Figure 7, the input shape of 26×1 was converted into 13×2. Four convolutional layers were used, with two fully connected layers (density). The first convolutional layer (Conv1D 1) had 256 filters, a kernel size of 2, and a ReLU activation function. In the second convolution layer, 128 filters were used, and the activation function and kernel size were identical to those in the first layer. The second layer is a max-pooling layer with a window size of 2 to reduce the spatial dimensions of the input entering the third convolutional layer. The third and fourth convolution layers (Conv1D 3 and Conv1D 4) used the same filter size of 64, kernel size of 2, and ReLU activation function. After the fourth layer, we introduced a dropout mechanism by setting 50% of the active neurons in the network to zero, which helped prevent an overfitting during feature data training and increased the classification accuracy. Subsequently, the layer was flattened to obtain a fully connected layer (dense 1) with 32 neurons and ReLU activation. The second dense layer is the output layer with two neurons using a softmax activation function. The cross-entropy loss function is defined to calculate the cost using Eq. (10). As a result, we applied the Adam optimizer to minimize the loss. The network batch and epoch sizes were both set to 100.

where y_i and ŷ_i represents the target label and the estimated value, respectively.

As shown in Figure 7, the proposed network achieved an accuracy of more than 95% after 20 epochs, achieving 99.7% accuracy at the end of 100 epochs. The multi-feature approach used to extract the features can differentiate between snoring and non-snoring data and consequently pave the way for 1D-CNN classifiers to achieve such a high classification accuracy. Moreover, the architecture of the classifier, including the parameter settings, also contributes significantly to the results. Based on these results, a deep learning model was created to detect snoring/OSA with a high classification accuracy in the presence of noise disturbances. Table 1 lists the variation in accuracy when there were few combined features. However, the MFCC had an extremely large impact depending on the results achieved, whereas the spectral features did not significantly affect the classification accuracy. Nonetheless, the combination of these seven features provided the best overall accuracy.

4.2 Discussion

Table 2 presents the classification results of the existing research on snoring data [6,9–11,20–23]. From Table 2, it is evident that the multi-feature extraction techniques (ZCR+STFT+MFCC) along with the RNN classifier yielded the best result of 98.8% with one of the largest data sizes [7]. Compared with the highest result of 98.8% in Table 2, it can be seen that the accuracy of the model using the proposed MFCC+ZCR+STFT+Spectral+ RMS and 1D-CNN presented in Table 1 is 99.7%, which is a 0.9% improvement. When considering the error rate instead of the accuracy, the 1.2% error rate was reduced to 0.3%, showing that the error rate was reduced to only one-fourth. This confirms that the proposed method is highly effective in classifying data on snoring, which is a precursor to a sleep apnea transition. By contrast, the absence of a feature extraction technique may seriously affect the result of a classifier and thus reduce the classification accuracy, as cited in [22].

We presented a multi-feature extraction approach for the classification of snoring sounds using a CNN classifier. The features include the STFT, RMS, spectral centroid, bandwidth, RollOff, ZCR, and MFCC. The features extracted from the sound dataset were used to form a feature dataset for training. Therefore, a 1D-CNN and 2D-CNN were designed to classify the sound dataset using the extracted feature sets. The experiment results showed that the proposed 1D-CNN achieved accuracies of 99.7% for snoring sounds. This proves that the proposed method is useful for identifying snoring sound datasets. Our study provides a multi-feature extraction process that has the potential to extract essential features that can produce a high classification accuracy from audio sound data. The snoring CNN model can be adopted as an alternative and inexpensive method for diagnosing or detecting snoring/OSA. In addition, the snoring data acquisition approach can be adopted to capture audio datasets in the fields of signal processing, medical and musical information retrieval.

In the future, we plan to expand the field of application by applying the proposed method to other datasets, such as speech, environmental, and animal sounds. For example, cat sound classification research has been previously conducted in Korea [24]. However, it is not easy to apply a deep learning approach to the classification of cat sounds owing to data sparsity. In addition, the data collection method and the proposed feature extraction method in a non-face-to-face environment caused by coronavirus disease 2019 (COVID-19) are expected to be used in various ways in the work-from-home environment with sound data.

Table 1. Variation in accuracy of features using the proposed 1D-CNN on snoring dataset.

Table 2. Classification results in existing studies on snoring data.