CiteScore 3.36
More on impact ›

Original Research ARTICLE Provisionally accepted The full-text will be published soon. Notify me

Front. Robot. AI | doi: 10.3389/frobt.2019.00060

Intermediated Reality: A Framework for Communication through Tele-Puppetry

  • 1Edinburgh Napier University, United Kingdom

We introduce Intermediated Reality (IR), a framework for intermediated communication enabling collaboration through remote possession of entities (e.g. toys) that come to life in mobile Mediated Reality (MR). As part of a two-way conversation demonstration, each person communicates through a toy figurine that is remotely located in front of the other participant. Each person's face is tracked through the front camera of their mobile devices and the tracking pose information is transmitted to the remote participant's device along with the synchronized captured voice audio, allowing a turn-based interactive avatar chat session, which we have called ToyMeet. By altering the camera video feed with a reconstructed appearance of the object in a deformed pose, we perform the illusion of movement in real-world objects to realize collaborative tele-present augmented reality (AR).

In this turn based interaction, each participant first sees their own captured puppetry message locally with their device's front facing camera. Next, they receive a view of their counterpart's captured response locally (in AR) with seamless visual deformation of their local 3D toy seen through their device's rear facing camera.

We detail optimization of the animation transmission and switching between devices with minimized latency for coherent smooth chat interaction. An evaluation of rendering performance and system latency is included. As an additional demonstration of our framework, we generate facial animation frames for 3D printed stop motion in collaborative mixed reality. This allows a reduction in printing costs since the in-between frames of key poses can be generated digitally with shared remote review.

Keywords: augmented reality, Mixed reality, collaboration, Diminished realiy, Real-time graphics, Human Computer Interaction (HCI), Tele-Puppetry, Collaborative Mixed Reality (CMR) System

Received: 20 Feb 2019; Accepted: 03 Jul 2019.

Edited by:

Thierry Duval, IMT Atlantique Bretagne-Pays de la Loire, France

Reviewed by:

Hideyuki Nakanishi, Osaka University, Japan
Mark Billinghurst, University of South Australia, Australia  

Copyright: © 2019 Casas Cambra and Mitchell. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mr. Llogari Casas Cambra, Edinburgh Napier University, Edinburgh, United Kingdom, llogari92@gmail.com