Improvement of deep reinforcement models using extreme learning machine for autonomous agents in unstructured environment

Abstract

Creating an autonomous agent, that gets real observations such as sensory data and images from the surrounding environment and learns optimal sequential actions, has been considered as one of the main goals of Artificial General Intelligence (AGI). Deep (Hierarchical) Reinforcement Learning (HRL/DRL) can address this objective. Traditional deep reinforcement learning methods suffer from long learning and training time resulted from the need to fine-tune the weights iteratively in the network. This research investigates the previous problem by utilizing a random weights generation approach that is based on Extreme Learning Machine. This method benefits from the randomness of input weights and least square solution in output weights calculation to reduce the training time by an order of magnitude. Hierarchical ELM (H-ELM) and Local Receptive Field ELM (LRF-ELM) are recent versions of multilayer ELM to respectively learn and extract features by hierarchical learning scheme. They have outperformed other existing deep models in terms of learning time (speed). H-ELM’s architecture was found to be similar to gradient-based (GB) auto-encoder without weights fine-tuning. However, H-ELM gives higher learning speed compared to the GB autoencoder. Moreover, LRF-ELM was found as similar to Convolutional Neural Network (CNN) without weights fine-tuning. It has outperformed the traditional CNN in the term of learning time. Therefore, in this research, the proposed method, which combines RL with H-ELM or LRF-ELM, is an efficient solution to approximate the action-value function and learn an optimal policy directly from visual data (images) in a short time. In addition, this research proposed a novel method called Convolutional H-ELM (CH-ELM) which is a combination of pre-trained CNN and H-ELM. This method has outperformed either CNN or H-ELM in terms of accuracy and RMSE. The experimental results have been analyzed and evaluated in different applications such as target reaching arm, 2D maze navigation, slide puzzle game , objects sorting, and rock-paper-scissor game. The data samples have been trained and tested to investigate the robustness of the proposed systems. It was found that the proposed models can reduce the learning time by an order of magnitude in various tasks without degrading the performance. The big improvement in learning speed in the proposed method can neglect the slight drop in accuracy in few tasks compared to traditional methods. Therefore, the proposed method can balance the trade-off between learning speed and good performance. In addition, it is able to run on traditional CPUs that are available in the most of the low cost embedding systems.