Computational cardio-physiological model of emotion using phonocardiography

Abstract

Several studies on physiological-based human emotion have suggested that emotion causes variations in various physiological parameters. As one of the physiological parameters, heart sound signals (also referred to as phonocardiography) may infer emotions and can possibly be used for emotion recognition. For this purpose, the use of Phonocardiography (PCG) signal is substantially cheaper, and the process of acquiring the signal for heart sound analysis is comfortable as compared to other physiological measures. Capturing heart-sound signals using PCG does not require touching the surface of the human body directly. Hence it offers a convenient and practical usage in various applications of emotion recognition. Additionally, unlike the use of electrocardiography (ECG) that reflects only heartbeats through the electrically conductive system of the heart, the PCG can also reflect the muscle contraction sound of the heart. Nevertheless, the use of PCG in the emotion recognition domain is still scarce in the research literature. As such, this thesis explored usability and methods for modelling emotion recognition using PCG signals. The thesis is developed with four major phases. (i) Since PCG data for emotion recognition are not widely available, the first phase performs the creation of the corpus for both PCG and EEG, hence, the performance for both modalities can be compared. (ii) The second phase investigates the most suitable method for building a computational model for PCG-based emotion recognition. Three cepstral-based features, namely, MFCC, LFCC, and GFCC, are considered in the experiment. The DNN, XGBoost, and Decision tree are selected as the classifiers. The initial experiments of this research indicate that the best model for recognizing emotion is achieved at 87% accuracy rate by using combination of MFCC feature extraction and DNN classifier, (iii) The third phase compares PCG-based emotion recognition using heart sound signal (PCG) with EEG modality. The experimental results implied that with techniques used in phase two, the PCG signal could achieve comparatively robust performance in recognizing emotion as compared to the EEG modality. (iv) In the fourth phase, a new computational approach is proposed and implemented by incorporating signal decomposition techniques such as Empirical Mode Decomposition (EMD). As the main issue with this approach is feature dimensionality, the PCA feature reduction technique is adopted in the proposed method. The proposed method demonstrated a robust and optimal performance of a PCG-based emotion recognition model, achieving overall accuracy rate at 98%. Overall, this research has highlighted the potential use of PCG signals for emotion recognition as an alternative to other commonly discussed modalities such as EEG. Additionally, the thesis also empirically proved that with proper methods in pre-processing the signal and the right feature extraction process and the suitable classifier, the PCG signal could achieve optimal performance in recognizing emotion. As future works, the proposed approach can be used to build a wide range of practical application of emotion recognition such as Ambient Assisted Living (AAL), whereby the patient’s mental state is required to be continuously monitored.