AUTHOR=Liu XiaoLe , Yu Si-yang , Flierman Nico A. , Loyola Sebastián , Kamermans Maarten , Hoogland Tycho M. , De Zeeuw Chris I. 

TITLE=OptiFlex: Multi-Frame Animal Pose Estimation Combining Deep Learning With Optical Flow

JOURNAL=Frontiers in Cellular Neuroscience

VOLUME=Volume 15 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/cellular-neuroscience/articles/10.3389/fncel.2021.621252

DOI=10.3389/fncel.2021.621252

ISSN=1662-5102

ABSTRACT=Animal pose estimation tools based on deep learning have greatly improved animal behaviour quantification. These tools perform pose estimation on individual video frames, but do not account for variability of animal body shape in their prediction and evaluation. Here, we introduce a novel multi-frame animal pose estimation framework, referred to as OptiFlex, that integrates a flexible base model to account for variability in animal body shape with an OpticalFlow model to incorporate temporal context from nearby video frames. Pose estimation can be further improved using multi-view information to leverage all four dimensions (3D space and time). We evaluate OptiFlex using datasets of four different lab animal species (mouse, fruit fly, zebrafish, and monkey) and introduced an intuitive evaluation metric - animal percentage of correct key points (aPCK). Our evaluations show that OptiFlex provides prediction accuracy that outperforms current deep learning based tools, and that it can be readily applied to analyse a wide range of behaviours across different animals.