A Reflection on the State of Multisensory Human–Food Interaction Research

We present a perspective article on the state of multisensory human–food interaction (MHFI) research and lay out some reflections for research and development in this area of inquiry, based on a revision of the different spaces that we have co-created with researchers in this space. We begin by conceptualizing and defining MHFI, before moving onto presenting some of its major themes, as well as possible ways in which such themes can guide future research in the area. This article provides key definitions and foundations for the area of MHFI, as well as a first point of contact for those interested in it.


INTRODUCTION
The emerging field of Human-Food Interaction (HFI) is thought to be an area of Human-Computer Interaction (HCI) research that addresses our interactions with food (Comber et al., 2014). HFI specifically focuses on the role of technology in supporting and enriching food practices throughout the food chain, that is, from growing, through experiencing, to disposing (Khot and Mueller, 2019).
A Google Scholar search of "Human-Food Interaction" up to 2010 results in 18 research articles; however, the same search between 2010 and 2020 results in 508 (up to September 22, 2021). This, of course, is not an exhaustive index of the size of the field (perhaps some works might not talk directly about HFI, yet still be part of the field), considering that before researchers started to use the term HFI, research was already being conducted within HCI on human-food interaction (Grimes and Harper, 2008). Yet, this is initial evidence that HFI has grown significantly over the last decade. HFI has given rise to three communities of researchers, namely, Food CHI focusing on food and interaction design; another group of researchers focusing on artificial intelligence (AI) approaches to HFI; and Multisensory Human-Food Interaction (MHFI), which highlights the multisensory aspects of HFI (see Bertran et al., 2019, for an exhaustive survey of the field of HFI).
In the present article, we present our perspective on MHFI based on the work of multiple workshops and a special issue that we have co-created. While this area of research is still in its infancy, we have seen an increasing interest in fields as diverse as HCI, psychology, sensory science, and marketing, as well as an emerging effort to study the intersection between the senses, food, and technology (e.g., Crofton et al., 2019;Petit et al., 2019). Last year, the specialized ACM International Conference on Multimodal Interaction (ICMI) workshop on MHFI was conducted for the fourth time , yielding a total of 32 research articles across the four workshops. In addition, one special issue has been presented (with another one currently in course), together with the journals Frontiers in Psychology, Computer Science, and Nutrition, resulting in a total of nine articles. Importantly, contributions to MHFI have not been limited to this workshops and special issues. For all the aforesaid reasons, we thought it was time to present an article to reflect on the state of the MHFI research area from our perspective, and to contribute to the discussion on current advances and future directions, by considering where it comes from and where it is heading to. To this end, this perspective article focuses on the following aims: (1) define MHFI and present its current state, (2) reflect on the opportunities and challenges to develop this interdisciplinary area of inquiry, (3) reflect on the way forward to maximize the transfer of MHFI research into practice, and (4) present some general conclusions and takeaways. In addressing these aims, we hope to provide a starting point for reflection and discussion on the future developments of the MHFI area for those interested in this emerging topic.

Defining MHFI
As suggested by Velasco and colleagues (2018), the growing interest in HFI to capitalize on multisensory processes to create, modify, and enhance our food-related experiences may be explained, at least in part, by two observations. First, eating and drinking are among the most multisensory events in our everyday lives. Indeed, we interface with food through most, if not all our senses (Prescott, 2015;Spence, 2017). Second, technology is ubiquitous and there are growing efforts toward developing multisensory technologies, that is, technologies that are designed to stimulate the human senses beyond audition and vision allow researchers and practitioners to precisely control sensory quality, quantity, and delivery (e.g., haptic stimulation in mid-air, digitally-controlled smell delivery, electric taste devices; Cornelio et al., 2021;Covaci et al., 2018;Obrist et al., 2017;Velasco and Obrist, 2020). This context paved the way for MHFI as an area of inquiry. As such, it was conceived to focus mainly on the understanding of the multisensory process associated with our interaction with food (mostly eating) and on capitalizing on them when designing novel technologies and food interaction systems (Nijholt et al., 2016).
It is clear, however, that the initial scope of MHFI need to be broadened as the area progressed. First, our interactions with food are not limited to eating. Indeed, research from different fields have characterized the food interaction journey as consisting of different stages such as growing/purchasing, cooking, eating, and disposing Schifferstein, 2016). Secondly, the world is currently facing multiple challenges concerning food including, but not limited to, unsustainable food practices, food and climate change, food waste, obesity, (mal) nutrition, and hunger (FAO, 2018). With this in mind, MHFI can contribute beyond eating (e.g., nudging, expectations development, and disposing), to tackle other interaction stages (pre-eating and post-eating), as well as important food-related challenges that humanity faces (e.g., associated with health and sustainability). In fact, MHFI can connect research on the senses, food, and multisensory technologies to design any kind of food interaction and experience.
So how should we define MHFI? We follow the definition of Choi et al. (2014) of HFI as the interrelationship between self and food, though we include others (a social element) as part of our definition, considering that HFI can also involve food interactions between selves, mediated by technology (e.g., commensality, see Spence et al., 2019). Therefore, we define MHFI as a research area that studies the role of the senses in the interrelationship between self, others, and food, and that capitalizes on such understanding to modify existing and/or create new self-others-food interrelationships through technology. Note that the fact that food experiences are multisensory in nature makes HFI multisensory per se; however, the term MHFI involves the word "multisensory" as it places the senses at the center of, and emphasizes their role in HFI research and practice. While MHIF is a multidisciplinary area of study as defined above, it is worth mentioning that there is extensive research on multisensory perception and its relationship to food in fields such as psychology and sensory science (e.g., Prescott, 2015;Spence, 2017) and thus MHFI can build on such research to modify existing and/or create new self-others-food interrelationships through technology.
Here, it is important to mention that, in HCI, interaction is defined as any communication between a user and a computer, be it direct or indirect (Hornbaek and Oulasvirta, 2017). As such, MHFI also involves communication as part of it. Importantly, however, MHFI can also be about designing interactive interfaces between humans and digital technology and about designing interactive experiences . It is perhaps useful to think about MHFI as multisensory experiences in HFI. In other words, "In the context of HFI, multisensory experiences refer to impressions formed by specific food-related events, whose sensory elements (e.g., intrinsic and extrinsic to the food, see, for example, Wang et al., 2019) have been carefully crafted by someone for a given receiver (e.g., diners). For instance, to create the impression of a taste, say "sweet", colors, textures, and specific Frontiers in Computer Science | www.frontiersin.org December 2021 | Volume 3 | Article 694691 smells can be considered in a specific event." (Velasco and Obrist, 2021, p. 3). Given that experiences are only one part of MHFI, this definition may be broadened as follows: MHFI refers to self-food interrelationships formed by specific food-related events, whose sensory elements have been carefully designed by somebody for a given receiver or group of receivers.

Major Themes in MHFI
To formulate our perspective on the current state of MHFI research, we revisited the various contributions to our four ICMI workshops on MHFI, as well as one research topic in Frontiers (n 41). We identified key themes, associated with the research and development process of the area, that emerged from the articles (see Supplementary Appendix SA1, for the titles of the articles and themes identified). Although the number of articles is relatively small, it is representative if one considers that MHFI focuses only on a subset of research of HFI (see Introduction). Importantly, because this article is intended to reflect upon our perspective in MHFI, the sample of articles serves that purpose. After reviewing these articles, we identified, through a series of iterations, five major themes. Below, we present a summary of such themes and the proportions of articles in each theme (see also Supplementary Appendix SA1).
1) Data collection and analyses: Articles in which a system for data collection and/or analyses are presented. Number of articles 5 (12.3%). 2) Psychological mechanisms: Articles presenting studies designed to better understand psychological mechanisms associated with MHFI (e.g., crossmodal effects on flavor perception, such as the influence of, say, auditory stimuli on perceived taste intensity). Number of articles 13 (31.7%). 3) Design studies: Articles studying design approaches and frameworks. Number of articles 5 (12.2%). 4) Augmentation and interfaces: Articles focused on food augmentation processes and interfaces. Number of articles 7 (17.1%). 5) Applications-Commensality, education, entertainment, and/ or health: Articles studying interactions specifically targeting one of these areas. Number of articles 11 (26.8%).
The majority of articles (about a third) have focused on psychological mechanisms, followed by specific applications in commensality, education, entertainment, and/or health, followed by augmentation and interfaces, design studies, and data collection and analyses. Here, it is worth mentioning that research and development in MHFI can capitalize on existing research from other fields (e.g., research on the neuroscience of flavor perception to develop new systems; Prescott, 2015).

Foundations of MHFI: Connecting Fields, Research, and Practice
Following our analyses of the articles and the key themes that we identified, we see that there is possibility for guiding research in MHFI in such a way that it starts from an understanding of MHFI psychological processes, which result in applications (Figure 1).
In Figure 1, we present the themes associated with progress in MHFI. The first and second themes consist of elucidating psychological mechanisms and data collection and analytical methods, both of which support the human understanding foundations of MHFI. The third theme consists of design studies to develop frameworks and the fourth theme on the development of specific interfaces and augmentation technologies, both of which support the user interaction foundations. These technological interfaces can then be used to target specific applications in various areas such as commensality, education, entertainment, and/or health, which constitutes the fifth theme. Note that it is possible that specific interfaces are already designed with applications in mind, though.
It is perhaps worth illustrating with a now classic example of MHFI, namely, the Chewing Jockey, which is a system that monitors mastication and synchronizes sound-delivery to it (Koizumi et al., 2011). This technology capitalized (1. Psychological mechanisms) on previous studies developed to understand and document the role of auditory cues on modulating texture and taste perception (Zampini and Spence, 2004). Said studies have suggested, for instance, that the crispiness of potato chips can be enhanced by chewing sounds or white noise with a high-pass filter (see also Spence, 2015, for a review). Based on this idea and aiming to redesign the eating experience (3. Design studies and frameworks), Koizumi et al. (2011) developed the chewing jockey technology (4. Interfaces and augmentation). Once this interface was designed, the authors moved on to specific applications (5. Applications). Here, the authors suggested at least two applications. The first consisted of using the system to enhance texture perception for the elderly (health), and the second, to design novel fun interactions (entertainment), such as mapping the sounds of screaming sounds to gummy bear chewing. While this study did not fully capitalize on available design frameworks for sound augmentation (3), it follows general experience design guidelines. In addition, while data collection (2) was not a part of it, given that the system uses a chewing tracking system, it is possible to collect data, as well (see also Lin et al., 2020, FoodFab andNarumi et al., 2011 Metacookie, for other examples). All in all, the possible themes associated with research and development in MHFI allow researchers and practitioners to think of how to connect everything from basic research on multisensory influences on self-others-food interrelationships all the way to possible applications. It is important to mention here that one of the key characteristics of this area of research and practice is its interdisciplinary nature, involving fields such as, though not limited to, psychology and neuroscience, sensory science, HCI, and marketing. This interdisciplinary work can guarantee the strong conceptual and practical foundations of every step of the research and development process (Figure 1).

RESEARCH DIRECTIONS AND CONCLUSIONS
MHFI is a nascent area of research and, as such, there are multiple unanswered questions and directions for research that need to be addressed. For example: Considering those questions, we reflect below upon some key areas for future research and development in the area of MHFI, especially based on the discussions in our workshops and special issues. We also reflect on how to approach MHFI research and put it into practice.

Direct Interaction With Food: Designing Experiences That Enhance the Eating Experience
Designing technology around the ingestion process, we can use the senses to highlight flavor as well as influence appetite. For instance, for the elderly (Doets and Kremer, 2016), work in MHFI may contribute to making up for losses in smell and taste perception as well as promote desire to eat. MHFI could also solve challenges around eating in extreme situations (e.g., space exploration) where technology and psychological understanding are needed to create new ways of eating (Obrist et al., 2019). Referring to our framework (Figure 1), this area relies on making use of existing knowledge in psychological mechanisms to develop design frameworks, interfaces, and applications to support people's eating experiences.

Social Aspect: Designing Interaction With Others Around Food
Food is a means for socialization and sharing (Niewiadomski et al., 2019). The social aspects around food can include food growing, producing, purchasing, preparation, eating, sharing, and disposing . Remote commensality is a special area of interest (e.g., Ceccaldi et al., 2020), which can be enhanced by considering the multisensory processes associated with social dining .

Change Attitude Towards Food: Designing Food Interaction to Nudge People
Beyond direct ingestion, how can we use the senses to change people's mindset about specific foods (de Vries et al., 2020;Zhao et al., 2016)? This could be influencing their food-based decisionmaking-including attitude towards certain foods, purchase intentions, and disposal habits (Cadario and Chandon, 2020;Hollands et al., 2017). For example, the integration of augmented reality and other visually enabling technologies in the process of food purchases can influence the way in which people develop purchase intentions (Petit et al., 2021;Toet et al., 2017). MHFI has already seen several theoretical developments, but what is missing is the integrated use of novel technologies and data collection mechanisms that can measure large quantities of real-world data.

Digital Augmentation: Technology Enabled Food Interaction Experiences
An increasing number of digital technologies are being developed to stimulate our chemical senses and thus create new MHFIs and/ or to study multisensory processes (Cornelio et al., 2021). Indeed, there is increasing interest in the way in which technologies can be used to create and/or augment eating and drinking experiences digitally Vi et al., 2017). Note, however, that augmentation is not limited to the chemical senses. Indeed, a wide range of research directions and applications (e.g., context enhancement, food structure and texture, and sensory augmentation) have been forwarded for augmented reality in food interactions (see Narumi, 2016;Crofton et al., 2019). With respect to the pipeline framework, this is an exciting area with many new interface developments. However, what is less clear are how such interactions can spread outside the lab and be deployed in the real world.

Ethical Considerations: Responsible Innovation Around Food Interaction
Recently, Velasco and Obrist (2021) indicated that, as there is scope for development in MHFI, there are also key responsibilities, and thus, we need to consider the ethical implications of this area of research. This is particularly important when moving from lab-based explorations into realworld deployments. What are key ethical reflections in MHFI? Consider the abilities of digital technologies like food 3D printers and virtual and augmented (VR/AR) reality. Those technologies enable us to create/design realities that are not matching the physical world. For example, we can now change the appearance of food to make it look more appetizing, or we can change the infill structure of 3D printed food to affect people's feeling of satiety (as in FoodFab by Lin et al., 2020). We can create food perception illusions that can deceive people, but in this case benefit the person (eating less, which in light of a global challenge of obesity can be considered a desirable intervention). However, who decides about the beneficiaries and when it is ok to create such experiences? While this will require ongoing discussion on the topic, it is key to treat receivers of MHFIs fairly by considering their differences and similarities, avoiding biases, and ensuring accessibility of technology (see also Choi et al., 2014). This is an overarching concern that touches upon all elements of the pipeline framework.

Open a Dialogue on How to Approach MHFI in Research and Practice
While the themes identified in Figure 1 are  The list of questions raised is evidence of the idea that we are only at the beginning of understanding and exploring the areas and themes around MHFI, and while we wish to provide answers, it is more our intention to open up a dialogue with the community, and as part of that, in effect, continue our past efforts (workshops, special issues), with this perspective article.
MHFI applications are often developed in the laboratory. Brands have recently focused on AR mobile applications to highlight the sensory aspects of their products (Jacobsen et al., 2021). How can the area move research and development from just workshop demos to wider adoptability? Should we engage companies as collaborators, or encourage entrepreneurship among MHFI researchers? Should people work with relevant stakeholders (e.g., hospitals and schools) with an interest in putting research into practice? These questions deserve discussion. Notably, our position is that, in order to make MHFI research relevant in both basic and applied research, in theory development and practical implications, having relevant stakeholders involved will be critical, from consumers, through researchers, to firms and/or other applied contexts. What is more, as the area develops, it will be important to develop both qualitative reviews and meta-analyses that help shape the foundations of MHFI beyond our perspective.

CONCLUSION
We presented here our perspective on the state of MHFI research. We started by placing it in the broader context of HCI and more particularly HFI, and then defining it. Building on the four ICMI workshops on MHFI, as well as one research topic in Frontiers, we identified five key themes of research in this area, namely, (1) data collection and analyses, (2) psychological mechanisms, (3) design studies, (4) augmentation and interfaces, and (5) applications-commensality, education, entertainment, and/or health. These themes can constitute a compass for the interdisciplinary development of this area, from basic research to practice.
In addition to these themes, we described some key areas of research we believe will be crucial in the development of MHFI, which include: (1) designing experiences that enhance the eating experience, (2) interaction with others around food, (3) changing mindsets and attitudes, (4) interfaces and technologies, and (5) ethics.
We believe that research in MFHI should be approached in a way that connects basic and applied research, and which results, in the end, in applications, potentially co-developed with stakeholders in the applied world.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.