AUTHOR=Su Jiahao , Li Jingling , Liu Xiaoyu , Ranadive Teresa , Coley Christopher , Tuan Tai-Ching , Huang Furong 

TITLE=Compact Neural Architecture Designs by Tensor Representations

JOURNAL=Frontiers in Artificial Intelligence

VOLUME=Volume 5 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2022.728761

DOI=10.3389/frai.2022.728761

ISSN=2624-8212

ABSTRACT=We propose a framework of {\TNNlong}s ({\TNN}s), which extends existing linear layers on low-order tensors to multilinear operations on higher-order tensors. Learning {\TNN}s is challenging in general,
which corresponds to tensor decomposition in a nonlinear setting. We address this problem by deriving backpropagation rules for {\TNN}s, with a novel suite of generalized tensor algebra. Our {\TNN}s have three advantages over existing networks: First, {\TNN}s naturally apply to higher-order data without flattening, which preserves their multi-dimensional structures. Second, compressing a pre-trained network into a {\TNN} results in a model with similar expressive power but fewer parameters. Finally, {\TNN}s interpret advanced compact designs of network architectures, such as bottleneck modules and interleaved group convolutions. Experiments on VGG, ResNet, and Wide-ResNet demonstrate that {\TNN}s outperform the state-of-the-art low-rank methods on a wide range of backbone networks and datasets.