People enjoy music in their leisure time. Now a days there is more and more music on personal computers, on mobile phone and on the Internet. The huge number of music can be specified by mood, which can help us to easily retrieve a proper set of music. There are two major aspects for automatically evaluating mood of music; one is mood classification [1, 2] and the other is mood regression. In the mood classification the category named by an adjective term is automatically given to a song. In the mood regression the mood is evaluated by values on the proper scales, each of which represents the degree of a specific emotion state.

Whatever types of the detection we take, it is necessary to specify an appropriate emotion space to evaluate the mood. Even though there is no standardized space, however, Thayer’s two-dimensional emotion space model as in Figure 1 is well recognized as such space, in which the fundamental aspects of music mood are represented in a two-dimensional space of valence and arousal [3]. In the model, valence axis describes the continuous scale from pleasantness (happy, pleased, hopeful, etc.) to unpleasantness (unhappy, annoyed, hopeful, etc.). The arousal axis represents the degree of calming or exciting. Each axis spans the bipolar scale values from −1 to 1. The emotion is represented by a point or region [4] based on the values of coordinates, each of which corresponds to the positive or negative degrees of the feelings. Sometimes the mood categories can be designated in the two-dimensional emotion space as in Figure 1 [5].

The authors [2, 6] concludes that number of musical features like tempo, pitch, rhythm and harmony makes listener perceive and correctly identify a specific intended emotional expressions of mood like sad, fear, humorous, happy, and exiting. The study of lyric-based music mood recognition [7] has been performed to observe the relationship between content of lyrics and mood using OPM songs. KeyGraph keyword generation algorithm and word level features like TF-IDF have been examined by using various parameters and threshold to extract the keywords from lyrices. Automatic music mood recognition using support vector regression [8, 9] is studied by mapping the various music features into Thayer’s two-dimentional emotion space model which can later predict the values for valence and arousal. Similarly a 5-layer convolutional neural network (CNN) trained on Million Song dataset [10] for music tagging is used to extract the features to detect the music mood with respect to valence and arousal [11]. Similar audio network [?] pre-trained on Million Song dataset is used to classify the GTZAN, 1517-Artists, Unique and Magnatagatune datasets.

The deep neural network needs large amount of labeled dataset, but music information retrieval community (MIR) still have scarcity of such data especially for the music mood. Transfer learning also called inductive transfer is one of the way that can resolve the problem of limited number of labelled data set. Transfer learning reuses knowledge of the source domain to solve a new task in the target domain. For example, knowledge gained while recognize cars could apply when trying to recognize trucks [13]. Transfer learning has been applied successfully in many domains such as visual object recognition [14], web document classification [15, 16], and human action recognition [17]. There were several studies [18–21] in the field of machine learning using transfer learning approach. A natural RGB images to malignancy classification is used in [22] for contrast-enhanced MR images. The POS tagging on Penn Treebank data is used to transfer the knowledge of source task to target task POS tagging on microblogs with limited available annotation for sequence tagging data [23]. Similarly, the patterns of human epithelial type 2 (HEp-2) cells are studied by applying transfer learning from a pre-trained deep CNN to extract the generic features [24].

In this work, we extract the features of original and augmented EmoMusic dataset by reusing the layer-wise weights from the pre-trained network [11]. The pre-trained network is trained on Million Song dataset for music tagging. After extracting features, multilayer perceptron (MLP) with three or four hidden layers trained with 50% dropout is used to detect the scores in valence and arousal. In the experiment the number of data is increased by augmenting the original Emo-Music dataset using time stretching. The ground truth label for augmented data in valence and arousal dimension is set to be identical as original data. The experimental result of 10-fold cross-validation exhibits that MLP with the original EmoMusic dataset is similar to state-of-the-art accuracy in SOA(1) [11] and SOA(2) [25]. The highest R²-score of 79.64% for valence and 85.61% for arousal is achieved by the augmented EmoMusic dataset which is better than the state of art in EmoMusic dataset.

The organization of this paper is as follows. In Section 2, the methods of audio, feature extraction, mood detection using the MLP for regression and data augmentation method are described. Section 3 presents about the experimental results. Finally, we draw the conclusion in Section 4.

2.1 Feature Extraction

The audio file from the EmoMusic dataset is preprocessed using Librosa library [26] to generate the Mel-spectrogram. The single channel audio files amounts to the length of 29 seconds, with sampling rate of 12, 000 Hz. The size of FFT is 512 with the hop length of 256, which produces 1, 360 frames for a song. The number of Mel-bins is 96 so that the size of the generated Mel-spectrogram is 96×1360, which is fed into the pre-trained network for feature extraction.

2.2 Mood Detection by Multilayer Perceptron and Its Training

After obtaining the features from the pre-trained network, the mood regressor is constructed by a MLP. By the empirical experiment we set three or four hidden layers, and one output layer as shown in Figure 3. The numbers of units in the hidden layers are also empirically set according to the data size in the experiment. The last layer has two nodes, each of which evaluates the mood in the Thayer’s two-dimensional scales of valence and arousal. Because the value of each bipolar dimension spans [−1, 1], the tangent hyperbolic is adapted as activation function [27]. We also construct same size MLP for both original and augmented data in our experiment. However, we didn’t mention the result because the amount of original data is not sufficient for huge size MLP so that resulting accuracy is decrease and network seems to be overfitted.

To avoid over fitting in the training, 50% of dropout [28] is applied to the units in each hidden layer. The network is trained with 10-fold cross-validation with stratified splits. Adam optimizer [29] is used for updating weight and bias to minimize the mean squared error (MSE) loss function. During training batch size of 100 and epochs of 300 are used.

As mentioned earlier there are 32 features from each layer of pre-trained network by taking the global average pooling. In the experiment, we compare the performance for the hierarchical features from each different layer of pre-trained CNN. In addition to the 5 sets of 32-dimensional features, a set of all the concatenated features from 5 layers is considered to compare the performance.

For the comparison, we use each of these six sets of features for training and obtain the percentage of R² scores, separately. The R² score in Eq. (2) is the normalized version of the mean squared error in Eq. (1). In the equations, y_i, ŷ_i, and y represent the i-th true value, i-th predicted value, and the average of truth values over n data, respectively.

2.3 Data Augmentation

In the data-driven approach, the large number of data is essential to avoid overfitting. Therefore, the augmentation is required to artificially increase the number of data, especially when the number of data is not sufficient. Because the expensive psychological experiment is necessary to obtain the set of labelled data for mood detection, the size of labeled set is usually not so big for the data-driven approach.

The EmoMusic dataset has only 744 labelled data for mood regression. Therefore, we try to increase the number of Emo-Music dataset by time stretching with the modest scaling rates. Note that the too much shortened or stretched audio data can change pitch and tempo, which again makes the mood change differently from the original data. We use a randomly selected scaling factor in the small interval around 1.0 as [0.95, 1.05]. Note that the larger(less) than 1 scaling factor produces time stretched(shortened) audio data. After augmentation, we can have 3 × 744 new EmoMuisc data including two times of time stretched and the original data.

We use Librosa library for increasing the number of data by the time stretching. For time-stretching the EmoMusic dataset, at first, short time Fourier transform (STFT) of audio data is computed. Then we draw two randomly chosen stretching or shrinking rates from the uniform distribution in the interval.

In order to perform the time stretching, we take STFT of audio data. Then the STFT matrix of time-frequency representation is stretched or shorten based on the phase vocoder implemented in [30]. After the time stretching, the resulting time-frequency representation can be transformed back into the audio data in time dimension with the inverse STFT (ISTFT).

Each music in EmoMusic dataset used in our experiment has its own emotionally evaluated scores in two scales of valence and arousal in Thayer’s emotion space. There are eight genres of music included in EmoMusic data set including Blues, Electronic, Rock, Classical, Folk, Jazz, Country and Pop, which is annotated via crowdsourcing named Amazon’s Mechanical Turk.

The two types of dataset are used in our experiment as in Table 2. One is original EmoMusic dataset [31] and another is our augmented EmoMusic data. The original dataset consists of 744 audio data and another data set additionally includes the 2×744 time stretched data. The emotional score for augmented data is the same as original data, which is a pair of continuous values from 1 to 9 in arousal and valence scales. In order to adapt to tangent hyperbolic activation function, each of the score is normalized from −1 to 1.

3.1 Experimental Results

The R² scores of ten-fold cross-validation is visualized using box and whisker plot [32], which is convenient for quickly summarizing the results. A box and whisker plot for a symmetrically distributed data has the mean close to the median line [33].

In each experiments in Table 2, layer_1, layer_2, layer_ 3, layer_4 and layer_5 represent the separate 32-dimensional features from layer one to layer five, and layer_all denotes the concatenated 160-dimensional features of all layers as in Figure 2. The mean R² score over 10-folds is labelled by a green triangle located inside the box in both valence and arousal plots.

3.2 Discussion

From the results of our experiments we can conclude that higher layer features as well as assorted features of all layers are better than lower layer features. The box and whisker plot in Figures 4 and 5 defines the distribution of R² score from 10-fold data. In most of the cases, for augmented data, the mean approximately coincides with the median line, which means the distribution of the 10-fold results is symmetric. However, the valence in experiment 1, the interquartile range (IQR) and length of two whisker is so large, which implies the results obtained from 10-folds have large variations. In contrast, there is less variation in the results of experiment 2.

The results form experiment 2 are smaller IQR and shorter length of two whiskers than those from experiment 1, which implies the augmented data by time stretching is valid to obtain stable mood detection in the MLP-based regressor. This results coincide well with the conclusion that deep learning and other modern nonlinear machine learning techniques can obtain better result with data augmentation [34, 35].

Considering the stable results from experiment 2, we can carefully conclude that the features from the higher layer provides better results in mood estimation of music. Usually, the CNN features in higher layer have wider receptive fields. This implies the features extracted from longer time durations are more effective to detect the mood of music.

3.3 Result of Comparison

We compare the mean over 10-fold results from layer all in Table 4 with SOA(1) [11] and SOA(2) [25]. The SOA(1) adopted transfer learning technique to solve the mood detection of Emo-Music dataset and achieved the highest R² score of 65.6% and 46.2% in valence and arousal scales, respectively. On the other hand, SOA(2) reports R² score of 70.4% and 50.0% using music features with a recurrent neural network as a classifier. SOA(2) exploits 4777 audio features including quantiles, mean, standard deviation, zero crossing rate, MFCC, spectral energy, etc.

In Table 5, MLP with the augmented data performs the best scores of 79.64% and 85.61% in valence and arousal scales, respectively. Similarly, MLP with original data in experiment 1 also performs better result in valence, which is around 6% higher than SOA(1), and 2% higher than SOA(2). But the score in arousal is 3% less than SOA(2) and 2% higher than SOA(1).

This paper proposes an automatic mood detection of music with a composition of transfer learning and multilayer perceptron. The 5-layered CNN pre-trained on Million Song dataset is used to extract the features from EmoMusic dataset. We obtain a set of features from the different five layers, which is fed into MLP-based regression. Through the regression network we estimate the mood of music on Thayer’s two-dimensional emotion space, which consists of the axes corresponding to arousal and valence. Because the EmoMusic dataset does not provide enough number of data for training, we augment the data by time stretching to make it tripled. We perform the experiment with the augmented data as well as the original EmoMusic dataset. Box and whisker plot along with the mean of ten-fold cross-validation has been used for evaluating the proposed mood detection. In terms of the percentage of R² score for measure of accuracy, the proposed MLP shows 79.64% and 85.61% in valence and arousal scales, respectively, which is the state of art for EmoMusic dataset.

Based upon our results we conclude that data augmentation technique plays a significant role for stable and efficient detection of music mood. In addition, the features taken from the convolution filters with larger receptive field is more effective for the mood detection in MLP-based regressor.

We believe that the performance can be further improved by the ensemble strategy of time segmented emotional evaluations, and the deliberate feature selection method.

No potential conflict of interest relevant to this article was reported.

The research leading to these result, authors would like to thank Korean Ministry of Education for funding. The National Research Foundation of Korea (NRF) support us under Basic Science Research Program (NRF-2015R1D1A1A01058062).

Table 1. Structure of CNN for feature extraction.

Table 2. EmoMusic and augmented data sets for experiment.

Table 3. MLP structure for experiment 1 of original data.

Table 4. MLP structure for experiment 2 of augmented data.

Table 5. Comparison with SOA(1) and SOA(2).