Information Theory Meets Deep Neural Networks: Theory and Applications, Volume II

This Research Topic is the second volume in the series: Information Theory Meets Deep Neural Networks: Theory and Applications. The previous volume can be viewed here: Volume I

Deep Neural Networks (DNNs) have become one of the most popular research directions in the field of machine learning and achieved impressive results in multiple tasks. However, understanding the workings and mechanisms of DNNs remains challenging. Additionally, how to apply existing information theory methods to the training and optimization of neural networks is also a topic of great interest. Information theory is a mathematical method for representing and analyzing information and can be used to study the basic characteristics of data such as structure and distribution. In the study of DNNs, information theory has also been widely applied to explain and optimize the performance of neural networks. For example, the Information Bottleneck theory has been used to explain the abstract representations of neural networks, and entropy and mutual information have been used to evaluate the complexity and generalization performance of models.

The problem that we aim to address is the need for a deeper understanding of how information theory and deep neural networks can be combined to enhance our knowledge of the organization and function of the brain. Despite advances in neuroscience and machine learning, these fields still lack a comprehensive and integrated understanding of the brain's information processing mechanisms. Recent advances in computational neuroscience have revealed the remarkable similarity between animal-brain systems and artificial neural networks that are essential for advanced machine learning techniques. Furthermore, applying information theory to the neural code has enabled us to characterize and quantify the complexity of neural network dynamics, aiding in the explanation of neural responses to sensory inputs and the formation of memories. This Research Topic aims to stimulate new research directions, encouraging researchers to pursue an interdisciplinary approach in the exploration of information processing in the brain, and will provide insights into the functions and mechanisms of the brain, advancing our ability to build intelligent systems based on brain-inspired principles.

This Research Topic aims to combine information theory with the field of deep neural networks to explore how information theory is applied to the training and optimization of neural networks and how neural networks can be used to solve problems in information theory. Specifically, the topics include but are not limited to:

- The application of the Information Bottleneck theory in deep neural networks

- Model selection and tuning problems in information theory and deep learning

- Entropy and mutual information measures in neural networks

- Information theory-based methods for deep neural network compression and acceleration

- Sampling and probability problems in deep learning

- Applications of deep neural networks in information theory, such as communication and coding

We sincerely invite researchers and engineers in related fields to submit original and high-quality papers. Through theoretical analysis and experimental verification, we hope to jointly promote the progress of information theory and deep neural networks.