Robust automatic license plate recognition (ALPR) system at low visibility using deep neural networks

Abstract

Automatic License Plate Recognition (ALPR) has become a common study area because of its many practical applications, such as automatic toll collection and traffic law enforcement. However, most existing methods of Malaysian ALPR are not robust enough to be used in everyday situations. The lack of high-quality benchmarked datasets that accurately represent real-world complexities in Malaysian license plates (LP) and the absence of a comprehensive dataset to demonstrate system robustness is a significant limitation. In addition, the reliance on shallow techniques in the studies on Malaysian ALPR causes inefficiency of the systems, particularly in handling complicated scenarios involving different image backgrounds and variations in LP size or shape and also the non-standard LPs. This dissertation presents a robust Malaysian ALPR system based on the single-shot detector You Only Look Once (YOLO) in two stages; license plate detection (LPD) and license plate recognition (LPR). The system is designed by evaluating and optimizing different models with different dataset optimization to achieve the best speed versus accuracy trade-off in ALPR system. The models are trained using a large-scale dataset containing images from several places around Malaysia, with the addition of data augmentation techniques to make them robust under various circumstances (e.g., with variations in lighting, camera position and settings, and license plate types). A dataset augmentation has also been accomplished by systematically generating a large, controlled synthetic dataset. The purpose is to achieve a balanced dataset and ensure the robustness of the dataset in terms of variations that exist in Malaysian license plates in the form of non-standardized license plates and special license plates. Thus, this work introduces a dataset for Malaysian ALPR with more than 176,000 images from real-world scenarios and synthetically produced images covering various aspects. This dataset will be public to the research community. The name of this dataset is Malaysian Number Plate and in short (MYNO). This dataset can be used for further training and evaluation of ALPR models. A separate challenging dataset is created for testing the models. Many experiments are carried out in detail with different models, data size, number of epochs, and real and synthetic datasets. When adding the synthetic dataset, the system performed better with 97.6% mAP compared to 85.5% mAP for the only real-world dataset at the same number of epochs. The proposed system achieved a recognition rate of 98.1% mAP on a real-world dataset collected from different toll plazas around Malaysia containing comprehensive environment distinctions with over 50 thousand labeled images. The system was tested on a challenging test dataset with low visibility and an unconstrained environment, resulting in 95.96% end-to-end accuracy. The results demonstrate the significance of incorporating synthetic datasets into the training process for improved performance in ALPR systems. The inclusion of a synthetic dataset led to a substantial increase in mean average precision (mAP), with a notable improvement of 12.1% when combined with the synthetic dataset. The system showcases its effectiveness in handling diverse environmental conditions by achieving 98.1% mAP on a real-world dataset collected from various toll plazas in Malaysia. In addition, achieving an impressive end-to-end accuracy of 95.96% despite low visibility further validates the system’s performance on challenging dataset.