Machine learning-based facial expression recognition using stretchable strain sensors for rehabilitation system

Abstract

Facial expression recognition (FER) enables computers or machine to identify human emotions. The FER system is used in self-driving cars, healthcare and smart environments. Most of the facial expression systems are based on computer vision and image processing technologies. Computer vision technologies are quite expensive since they need massive memory and computation resources. It also depends on the environment changes. However, sensors technologies overcome all the limitations because it does not need a massive amount of memory, expensive computation resources, and not depend on environment changes. This study aims to develop FER systems based on stretchable strain sensors data using machine learning in driving a rehabilitation system. Two different stretchable strain sensors (commercial and developed) are used to recognize four facial expressions (neutral, happy, sad and disgust). This study mainly focuses on the developed stretchable strain sensor, but this sensor is still developed in the laboratory and is not stable. So, the commercial stretchable strain sensor is used for the analysis of the developed sensor performance. The stretchable strain sensors data is time-series data with noise and high dimensionality. The datasets are normalized and aggregated to remove noise and high dimensionality. It is processed as an input to the machine learning model, and then it is compiled and fitted by five machine learning models, which are K- Nearest Neighbour (KNN), Decision Tree (DT), Support Vector Machine (SVM), Logistic Regression (LR) and Random Forest (RF) models. The training and testing results show that the RF model achieves the highest accuracy than other machine learning models. The RF FER model is then implemented in the experimental hardware test of the facial expression-driven rehabilitation system. When facial expression neutral, happy, sad, and disgust emotion, the elbow rehabilitation system (ERS) motor speed is 60%,80%,0% and 30% of its full speed, respectively. The simulation results show that RF has achieved 96% and 90% accuracy, respectively, in recognizing the correct facial expression using the three commercial and four developed sensors, respectively. The offline hardware experimental test results show that the facial expression has driven rehabilitation system has successfully achieved 93% and 83% accuracy. The three commercial and four developed stretchable strain sensor data to drive the rehabilitation system’s speed according to the facial expression displayed. In the real-time experimental test on five subjects, the system has achieved an accuracy of 75% in regulating the rehabilitation system's speed based on the actual users' facial expressions. The proposed study's limitation is that the stretchable strain sensors are uncomfortable for data collection and tests. However, the experimental results have proven that the proposed methods can drive the rehabilitation machine to move according to the recognized facial expression. The proposed system can enhance the rehabilitation system's comfort and safety according to the patients need. It will help the patients to recover better and faster and eventually improves their quality of life.