Comparative Analysis of Deep Q-learning Algorithms for Throwing Objects Using a Robot Manipulator

Mohammad Al Homsi^1*Maja Trumic²Adriano Fagiolini¹Giansalvo Cirrincione³

• ¹University of Palermo, Palermo, Italy
• ²School of Electrical Engineering, University of Belgrade, Belgrade, Serbia
• ³University of Picardie Jules Verne, Amiens, Picardy, France

Recent advances in artificial intelligence have attracted significant attention due to AI's ability to solve complex problems and the rapid development of learning algorithms and computational power. Among the many AI techniques, transformers stand out for their flexible architectures and high computational capacity. Unlike traditional neural networks, transformers use mechanisms such as self-attention with positional encoding, which enable them to effectively capture long-range dependencies in sequential and spatial data. This paper presents a comparison of various deep Q-learning algorithms and proposes two original techniques that use self-attention into deep Q-learning. The first technique is Structured Self-Attention with deep Q-Learning, and the second uses Multi-Head Attention with deep Q-Learning. These methods are compared with different types of deep Q-Learning and other temporal techniques in uncertain tasks such as throwing objects to unknown targets. The performance of these algorithms is evaluated in a simplified environment, where the task involves throwing a ball using a robotic arm manipulator. This setup provides a controlled scenario to analyze the algorithms' efficiency and effectiveness in solving dynamic control problems. Additional constraints are introduced to evaluate performance under more complex conditions, such as a joint lock or the presence of obstacles like a wall nearby a robot or a target. The output of the algorithm includes the correct joint configurations and trajectories for throwing to unknown target positions. The use of multi-head attention has enhanced the robot's ability to prioritize and interact with critical environmental features. The paper also includes a comparison of temporal difference algorithms to address constraints on the robot's joints. These algorithms are capable of finding solutions within the limitations of existing hardware, enabling robots to interact intelligently and autonomously with their environment.