Robust deep learning based open world mask face recognition [EMBARGOED]

Abstract

Mask Face Recognition (MFR) utilizing mask and non-mask images as input has been considered an intriguing issue in the research field over a couple of years in the Covid-19-like situation. Recognizing masked faces can be difficult due to the variety of factors involved, such as the mask's orientations, opacities, patterns, and kinds. Hence the large occlusion from mask face recognition is non-trivial. The MFR model may need to be retrained or finetuned to apply an MFR algorithm for new identities. As a result, the necessity for open world Masked Face Recognition is evident. To our knowledge, no attempts of MFR on the open set protocol have been made. This research aims to develop a robust deep learning-based open-world mask face recognition system and to perform an open-world evaluation of MFR and Non-MFR models on real and synthetic datasets. For this purpose, a framework for evaluating SOTA FR algorithms is developed for faces covered with masks. A real-mask and synthetic mask face dataset are curated and human-verified for training and testing MFR systems. A complete experimental protocol for these datasets is developed to measure the performance of the MFR system in the open world. Apart from evaluating SOTA algorithms, a proposed MFR system is developed and benchmarked. Both simulated or synthetic masked face MLFW datasets and real masked face MFD-k datasets have been used to evaluate the effectiveness of these algorithms in open-world settings. The overall performance of the SOTA 2D face recognition algorithm is worse than the random classifier, with the best performance by Dlib of 43.46% on the real mask face dataset. Whereas for synthetic data FaceNet was best, with an accuracy of 53.23%. The ConvNeXt-tiny outperforms the MFR challenge by over 20% on the best CNN counterpart ResNet. The adaptive threshold scheme from Faizabadi et al. improves the performance up to 99.83% accuracy on all pairs using real mask face dataset MFD-K_test and MLFW performance to 98.5%. In summary, the proposed framework of the preprocessing pipeline, dataset, evaluation protocols, and empirical studies proved effective in developing the robust MFR model in open-world settings.