AUTHOR=Ling Charles X. , Wang Ganyu , Wang Boyu 

TITLE=Sparse and Expandable Network for Google's Pathways

JOURNAL=Frontiers in Big Data

VOLUME=Volume 7 - 2024

YEAR=2024

URL=https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1348030

DOI=10.3389/fdata.2024.1348030

ISSN=2624-909X

ABSTRACT=Recently, Google introduced Pathways as its next-generation AI architecture. Google's Pathways needs to address three challenges simultaneously. First, it must be able to learn one general model for several continuous tasks rather than many specific models. During the learning process, old tasks should not be forgotten, and tasks can leverage each other. Second, the Pathways can learn from multi-modal data such as image and audio data. Third, the Pathways must be sparse, both in learning and in deploying. Current lifelong multi-task learning cannot answer these three challenges well. In this paper, we propose SEN, a Sparse and Expandable Network, which is quite simple yet effective in resolving these challenges. SEN introduces a novel "task dispatcher", a classifier that makes the whole network sparse. When learning a new task, SEN freezes the current network to avoid forgetting and expands the network to learn the new task. In addition, the dispatcher detects relevant tasks to leverage for knowledge transfer. Experiments demonstrate that SEN can effectively answer the three challenges, and exhibit many important features of lifelong multi-task learning.