Speech Emotion Recognition using a Novel Noise Augmentation Approach and Deep Learning

Abstract

Speech emotion recognition (SER) based on deep learning methods are practically more suitable than existing conventional machine learning methods for categorizing emotions from speech.  However, two limitations of existing speech emotion recognition (SER) frameworks are overfitting on scarce training inputs and poor robustness to environmental noise. This work proposes a novel augmentation scheme based on speech splitting and real environmental noise mix-up augmentation and, transfer learning by finetuning lower layers of pretrained networks to handle these aforementioned limitations. The speech splitting and real noise mix-up augmentation are adopted as a scheme to create additional speech samples required for training. Then, pre-trained networks are adapted for speech emotion recognition and finetuned with the generated training datasets to develop a model robust to noisy environment. Thereby, improving the classification performance in the wild. The novelty of this work is in utilizing speech splitting and real noise modulation as an augmentation scheme to increase the number of speech training samples for speech emotion recognition. Also, this augmentation scheme combined with transfer learning makes the SER model more robust to speech degradation due to noise. The Interactive Emotional Dyadic Motion Capture (IEMOCAP) database was utilized to generate benchmark datasets. Findings showed that, under three environmental conditions of SNR 15dB, 20dB, and 40dB, the proposed SER framework achieved an improvement of 9%, 6% and 5%, respectively, as compared to the state-of-the-art method. The proposed augmentation scheme for SER using deep learning can be useful in characterizing the emotions of speakers conversing virtually on online platforms.