Advanced DNS Tunneling Detection: A Hybrid Reinforcement Learning and Metaheuristic Approach

MAHMOUD SAMMOUR¹Mohd Fairuz Iskandar Othman^1*ASLINDA HASSAN¹Omar Bhais²Mohammed Saad Talib³

• ¹Department of Computer Systems and Communication, Universiti Teknikal Malaysia Melaka, Durian Tunggal, Malaysia
• ²Department Intelligent Computing And Analytics, Universiti Teknikal Malaysia Melaka, Durian Tunggal, Malaysia
• ³University of Babylon, Hillah, Iraq

ABSTRACT DNS tunneling remains a critical network threat, exploiting the inherent trust in the DNS protocol for unauthorized communication, data exfiltration, and firewall evasion. Addressing this challenge, this paper introduces a novel, hybrid feature selection framework that integrates the Random Forest classifier with an Enhanced Reinforcement Learning-Guided Grey Wolf Optimizer (EnhancedRLGWO). The EnhancedRLGWO employs a Dueling Deep Q-Network and strategic Opposition-Based Learning to intelligently navigate the feature space and identify an optimal, minimal subset. Evaluated against the benchmark CIRA-CIC-DoHBrw-2020 dataset, the proposed approach achieved a state-of-the-art accuracy of 99.82% and a weighted F1-score of 99.79% using a highly compact subset of only 12 features. This performance significantly outperforms existing machine learning-based DNS tunneling detection systems, such as a hybrid feature selection model achieving 98.3% accuracy and a full 28-feature Random Forest baseline (98.50% accuracy). The experimental results showed the robustness of this method in identifying various types of DNS tunneling attacks, including Iodine, DNS2TCP, and DNScat2, while maintaining performance and accuracy.