A human-in-the-loop haptic interaction with subjective evaluation

1 Introduction

With the development of cyber-physical systems and the advancement of Industry 4.0, one of the challenges in Human-Computer Interaction (HCI) is fully immersive multisensory remote physical interaction technologies. Compared with the traditional HCI of audio and visual, researchers are paying more attention to touch sensory, which is called haptic. The haptic modality enables humans to change the physical world remotely. Haptic perception promotes HCI from the audio-visual experience to the audio-visual-haptic interaction. The applications of haptic perception in HCI can enrich the interaction details and effectively improve the immersion and realism of interaction. Therefore, haptic interaction systems are widely used in industrial automation, autonomous driving, smart grid, gaming, health, education and other industry applications that require millisecond response.

A typical haptic interaction system includes a leader end to send commands, a follower end to execute commands, and a bi-directional communication channel [Liu and Zhao (2019)]. At the leader end, a human operator interacts with a haptic interface and collects the kinesthetic signals (position or velocity data). The signals are processed (sampling, quantization, and encoding) and then transmitted to the follower end. At the follower end, a remote robot receives the haptic data to complete a task and sends haptic feedback (position, force data) and audio/visual signals to the leader end. The human operator adjusts operations based on the audio, visual and haptic feedback. The communication channel provides the audio and visual signals transmission and bi-directional exchange of the haptic information. Different from the traditional audiovisual interaction, the haptic interaction system involves human operators and bi-directional exchange over the network. It closes a control loop between the human operator and the remote robot. It is obvious that in the human-in-the-loop haptic interaction system, the quality of experience (QoE) of the human operator plays an important role (Liu and Zhao (2019)).

QoE has been considered a key metric from the user’s perspective to evaluate the end-to-end performance. It reflects the degree of overall satisfaction or annoyance gained from the application [Zhao et al. (2017)]. It is helpful for user-centric system design and optimization. To date, there have been studies on QoE in haptic interaction. Jay [Jay et al. (2007)] demonstrated that haptic feedback was very sensitive to low levels of delay. Hoshino et al. [(Hoshino et al. (2011)] investigated the influence of inter-stream synchronization error between olfactory and haptic media. Tatematsu et al. [Tatematsu et al. (2010)] investigated the influences of the difference in network latency among haptic media, sound, and video on QoE for a haptic media, sound and video transfer system. Chaudhari et al. [Chaudhari et al. (2011)] explored the impact of data compression on QoE. Hamam et al. [Hamam and Saddik (2013)] [Hamam et al. (2014)] created a taxonomy for the possible parameters that can be used to evaluate a haptic audio virtual environment. The taxonomy’s higher-level organization was divided into Quality of Service (QoS), and User eXperience (UX). They investigated the impact of media synchronization, fatigue, haptic rendering, degree of immersion, and user intuitiveness and then proposed the mathematical models for QoE. Tasaka [Tasaka (2016)] employed a haptic-audiovisual communication system and built a Bayesian structural equation model with audiovisual quality, haptic quality, and user experience quality. The paper demonstrated that haptic quality has much stronger effects on user experience quality than audiovisual quality [Tasaka (2019)].

Notwithstanding these efforts, compared with QoE-related technologies in traditional audiovisual interaction, QoE in haptic interaction is still in its infancy [Liu and Zhao (2019)]. Firstly, the research on the influencing factors of haptic interaction QoE is not comprehensive. Haptic interaction scenarios in the current research are different, and there is a lack of analysis of key QoE influencing factors in the typical haptic interaction scenario. Secondly, there is no large-scale public database for the research of haptic interaction QoE. In this paper, the following works are accomplished to address the above two problems.

1. Based on a haptic interaction scene, we establish a haptic-visual interaction framework. Meanwhile, we analyze the influencing factors of QoE and select the key factors to provide a theoretical basis for the follow-up research.

2. Based on the key influencing factors, we design subjective evaluation experiments and establish a subjective evaluation database of haptic interaction. The database can provide a research basis for further exploring the quantitative relationship between various influencing factors and interactive QoE.

2 Subjective evaluation method

2.1 System architecture

In order to further analyze the influencing factors of haptic interaction QoE, based on a balance ball VR game scene, we establish a haptic-visual interaction system that combines eye tracking. The system consists of a leader end, a follower end and a bi-directional communication channel as shown in Figure 1.

FIGURE 1

FIGURE 1. Haptic-visual interaction system.

The leader end includes a haptic device (Geomagic Touch) with a sampling rate of 1000 Hz, a screen-based eye tracker (TobiiPro Spectrum 150) and a data analysis software (Tobii ProLab) which provides a visual user interface and efficient gaze data processing at a sampling rate of 150 Hz. The human operator watches the eye tracker screen and operates the haptic device to complete the virtual interactive tasks. Haptic data, including position coordinates, direction, grip angle, line speed, angular speed and grip angular speed, are captured by the device and sent to the salver end over the communication channel. Meanwhile, the operator’s eye movement information, including the gaze points, gaze directions, and fixation types, are recorded.

A communication network based on the UDP/IP protocol is employed to transmit the haptic information and visual signals in real time. In general, UDP is a connectionless protocol that is specifically chosen for time-sensitive applications. UDP protocol over Ethernet has been shown to be a viable method of communication between haptic devices and remote computers. Therefore, UDP is used for data transmission in the proposed system.

The follower end includes a virtual interactive platform and a camera. The virtual interactive interface is rendered using the received haptic information and the CHAI3D library. The virtual interactive interface includes a starting area, a workspace and a target area, as shown in Figure 2. When an interactive task is initialized, the red cube of the target object is placed in the starting area. A virtual gripper represents the pen-touch touch device. The operator controls the gripping tool to grip the target object and move it to the target area by manipulating the pen-touch touch device. The camera captures the virtual interactive interface. Then the video data are sent to the leader end over the communication channel.

FIGURE 2

FIGURE 2. Virtual interactive interface.

2.2 Parameter selection

The taxonomy in (Hamam and Saddik (2013)) groups the QoE parameters into QoS and UX. In this paper, based on the haptic-visual interaction system, we further divide the influencing factors into three categories: task-related, system-related, and user-related. As shown in Table1, six influencing factors that are relevant to haptic interaction are selected as follows.

1) Video transmission delay (system-related influencing factors). Haptic interaction systems usually provide the human operator with visual feedback from the follower end. Transmission delay can cause the media asynchronization. Zeng et al. investigated the influence of inter-stream synchronization errors among haptic media, sound, and video on QoE assessment (Zeng et al. (2013)). In this paper, we investigate the impact of video transmission delay on QoE. We consider the haptic media stream as the standard media stream and the average additional delay of the video stream is set to 0 ms, 170 and 425 ms, respectively.

2) Haptic data loss (system-related influencing factors). A haptic interaction system usually needs to transmit and receive a packet every millisecond. Otherwise, the stability of the system cannot be guaranteed. In other words, haptic interaction systems are very susceptible to data loss. To study the impact of the haptic data loss on QoE, we transmit haptic data at a packet rate of 1000 packets/s and drop the haptic data at a packet rate of 0 packets/s, 57 packets/s, or 114 packets/s.

3) Haptic quality (system-related influencing factors). In general, a kinesthetic compression scheme based on perceptual deadband could be utilized to reduce package rates in a haptic interaction system. Deadband parameters typically range from 5% to 15%. An appropriate deadband parameter is beneficial for haptic interactive systems, while excessive deadband parameter may cause data distortion and system instability. In this paper, we set deadband parameters of 0, 0.0742 and 0.1484 to investigate its influence.

4) Task complexity (task-related influencing factors). The increase in task complexity leads to an increase in task failure rate and poor task performance. In this paper, we use the complexity of the target movement trajectory to represent the task complexity. We design three interactive tasks, including one random-trajectory and two fixed-trajectory tasks. In the random-trajectory task, an operator moves the virtual object from starting area to target area according to any trajectory, as shown in Figure 3. In the fixed-trajectory tasks, the object motion strictly follows two specific trajectories, as shown in Figure 4 and Figure 5. According to the results of the questionnaire, the task complexity levels are shown in Table2. The higher the value, the higher the task complexity.

5) Task performance (task-related influencing factors). In general, an interactive system has stable performance and low task complexity, which can achieve higher task performance, better task results and shorter operating time, and vice versa. Therefore, the quality of task presents the quality of task performance and is usually quantified by measuring the task completion time. We compute the task completion time based on the duration of the eye-tracking record and compare it with a time threshold to determine whether the task is completed on time or not. The task is considered to fail if the time exceeds the threshold.

6) User perception state (user-related influencing factors). The operator’s attention is an important state for an operator to perform stable operations. In (Xue et al. (2019)), the authors observed the correlation between visual attention and kinesthetic information. Due to the rendering of virtual environment, gaze and kinesthetic positions are in different coordinate spaces. Thus, we use gluUnProject function to map the gaze positions coordinates to the plane of object coordinates. Both the gaze and the kinesthetic positions are 3D signals after coordinate transformations. We calculate the Pearson correlation coefficient between the coordinates of the gaze point and the coordinates of the target object during the haptic interaction tasks. The correlation represents the operator’s perception state.

TABLE 1

TABLE 1. The key QoE influence factors.

FIGURE 3

FIGURE 3. Designed interactive task:random-trajectory.

FIGURE 4

FIGURE 4. Designed interactive task:fixed-trajectory.

FIGURE 5

FIGURE 5. Designed interactive task:fixed-trajectory.

TABLE 2

TABLE 2. Task complexity and movement trajectory.

2.3 Experiment setup

Based on the combination of four quantitative parameters shown in Table 3, we construct 81 haptic interaction tasks. We recruit 25 subjects, including 10 males and 15 females aged 20 to 25, to conduct subjective experiments. All subjects have normal or corrected-to-normal visual acuity without hand disabilities. To ensure the reliability and adequacy of data, they are informed of the experimental goal and pre-trained to familiarize themselves with the haptic device. In the experiment, they watch the eye tracker screen and operate the haptic device to complete the virtual interactive tasks. Then, based on their satisfaction of the operating experience, they rate the overall QoE (5: excellent, 4: good, 3: fair, 2: poor and 1: bad). Each operator gives a score to each test to obtain the mean opinion score (MOS), which represents the average QoE. When the subject completes the virtual interactive task, we record the task completion time, the subject’s gaze point coordinates, the target object movement coordinates, and the overall QoE score.

TABLE 3

TABLE 3. Value of four quantitative parameters.

3 Subjective evaluation results

3.1 Experimental data screening

In order to use the collected subjective data as a solid “ground truth”, it is crucial to validate the reliability of the data [Zhang and Liu (2019)]. Therefore, we remove outliers and test whether the sample size is sufficient.

Firstly, following the subjective data collection, subject rejection strategy is applied to identify potential outliers in the rating process. We remove the abnormal data by comparing the correlation between the subjective rating and the MOS value. We used Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank order Correlation Coefficient (SRCC) as the evaluation criteria. If PLCC or SRCC is less than 0.7, the subjective data is removed. As shown in Figure 6, five subjective data are removed and 20 subjective data can be used to analyze the results.

FIGURE 6

FIGURE 6. Correlation between each subject’s rating and MOS.

Secondly, we use data saturation as a guiding principle to check whether the sample size is sufficient. As shown in Figure 7, the value of PLCC of the subjective rating and the MOS value is increasing when the number of subjects increases. The saturation occurs with 18 participants. It demonstrates that the number of 20 observers is fairly sufficient to yield a subjective database.

FIGURE 7

FIGURE 7. Data saturation validation.

Thirdly, we calculate the deviation of subjective ratings to further validate data reliability. As shown in Figure 8, the subjective scores cluster closer to the MOS value. The subjective scores are relatively consistent.

FIGURE 8

FIGURE 8. The obtained MOS value of each task (The horizontal axes corresponds to the task number, and the vertical axes corresponds to the MOS value. The red star indicates the obtained MOS value. The blue bar indicates the deviation of the subjective scores).

3.2 Discussion

Based on the subjective experiments, we establish a haptic interaction subjective evaluation database. The subjective database contains a total of 1620 records. In order to further analyze the influencing factors of haptic interaction QoE, we calculate the PLCC, SRCC and Kendall Rank-order Correlation Coefficient (KRCC) between each influencing factor and MOS values, as shown in Figure 9.

FIGURE 9

FIGURE 9. Correlation between each influencing factor and QoE.

Obviously, the haptic packet loss has the greatest impact on QoE. A virtual interaction interface is rendered based on CHAI3D. The haptic loss results in the inability to compute forces properly and render the virtual space, which seriously affects the stability of the haptic-visual interaction system. Therefore, the haptic data loss is highly negatively correlated with the MOS value, and the correlation coefficient is higher than other factors. Task performance also has a highly negative correlation with the MOS value. The value of SRCC is 0.9049. When the video transmission delay, the packet loss rate and the perceived deadband parameter are large, the interactive quality is poor. The operator is unable to perform the interaction task consistently and they spend more time completing the task. As a result, the operator may lose patience and the quality of user experience decreases. Video transmission delay can affect the interactive operation. However, through multiple interaction tasks, the operator is familiar with the task and thus avoids the negative impact of latency. Therefore, there is a negative correlation between the delay and overall user satisfaction, but the correlation is low. Similarly, once the operator is familiar with the target movement trajectory, the task is no longer complex, so the task complexity has less impact on QoE. The perceived deadband parameter is used as a threshold to determine whether to update the sample data. When the deadband parameter exceeds the user-perceivable threshold, the operator can perceive signal distortion. However, due to the limitations of the haptic device, this distortion is perceptible but does not seriously affect the execution of the interaction task. Therefore, the correlation between the perceived deadband parameter and the MOS value is weak.

In summary, the user experience of haptic interaction is influenced by the network conditions (e.g. delay and packet loss), the complexity of assigned tasks (e.g. task complexity and the quality of task performance), and the user perception state. Thoughtful consideration of all impact factors will provide better guidance to various applications. The most important thing in haptic interaction is to ensure the reliability of haptic data transmission. We can use an error correction coding algorithm to improve the fault tolerance for the haptic interaction system. When an operator performs repetitive tasks, the dead zone parameter can be increased to compress the haptic data, and the operator has tolerance for video transmission delay. Regardless of the task complexity, a good task result leads to good QoE. There is a relationship among QoE, quality of task and quality of control.

4 Conclusion

In the existing QoE-related research on haptic interaction system, researchers select different QoE influencing factors for different application scenarios to build QoE evaluation models. The models have high complexity and low generality and portability, so it is necessary to design an effective haptic interaction evaluation method for typical application scenarios. To solve this problem, we build a haptic-visual interaction system, select key QoE influencing factors, design subjective evaluation experiments, and establish a subjective evaluation database for haptic interaction. The database can provide a research basis for further exploring the quantitative relationship between various influencing factors and interactive QoE.

There is no general architecture for a haptic-visual interaction system, so we construct a haptic-visual interaction framework based on the typical application scenarios of haptic operation. Some of our parameters may not be applicable to other scenarios, but our approach is available. On the other hand, the designed interactive task is built in a virtual environment. As for the next step of our research, we will build a haptic-visual interaction platform in the real world, design more operation tasks, and investigate the impact of the key factors to improve the generalizability of the database.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

YF performed the research and analysed data. YQ performed the research and collected data. TZ contributed the idea, designed the research and collected the funding to perform the research. All authors were involved in writing the manuscript.

Funding

This work was supported in part by the National Science Foundation of China (Grant No. 62171134) and in part by the Natural Science Foundation of Fujian Province, China (Grant No. 2022J02015).

Conflict of interest

YQ was employed by Sangfor Technologies. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Chaudhari, R., Steinbach, E., and Hirche, S. (2011). “Towards an objective quality evaluation framework for haptic data reduction,” in IEEE World Haptics Conference, 21-24 June 2011, 539–544.

CrossRef Full Text | Google Scholar

Hamam, A., Saddik, A. E., and Alja’Am, J. A. (2014). A quality of experience model for haptic virtual environments. ACM Trans. Multimed. Comput. Commun. Appl. 10, 1–23. doi:10.1145/2540991

CrossRef Full Text | Google Scholar

Hamam, A., and Saddik, A. E. (2013). Toward a mathematical model for quality of experience evaluation of haptic applications. IEEE Trans. Instrum. Meas. 62, 3315–3322. doi:10.1109/tim.2013.2272859

CrossRef Full Text | Google Scholar

Hoshino, S., Ishibashi, Y., Fukushima, N., and Sugawara, S. (2011). “Qoe assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error,” in IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), Naples, FL, May 10–12, 2011, 1–6.

CrossRef Full Text | Google Scholar

Jay, C., Glencross, M., and Hubbold, R. (2007). Modeling the effects of delayed haptic and visual feedback in a collaborative virtual environment. ACM Trans. Comput. Hum. Interact. 14, 8. doi:10.1145/1275511.1275514

CrossRef Full Text | Google Scholar

Liu, Q., Zhao, T., Pei, Y., Gong, M., Cui, X., Pan, J., et al. (2019). Aqp-1 gene knockout attenuates hypoxic pulmonary hypertension of mice. Arterioscler. Thromb. Vasc. Biol. 17, 48–62. doi:10.1161/ATVBAHA.118.311714

PubMed Abstract | CrossRef Full Text | Google Scholar

Tasaka, S. (2019). “Bayesian categorical modeling of multidimensional qoe in haptic-audiovisual communications,” in IEEE International Conference on Communications (ICC), Shanghai, China, 20-24 May 2019.

CrossRef Full Text | Google Scholar

Tasaka, S. (2016). “Bayesian structural equation modeling of multidimensional qoe in haptic-audiovisual interactive communications,” in IEEE International Conference on Communications (ICC), Kuala Lumpur, Malaysia, 22-27 May 2016, 1–6.

CrossRef Full Text | Google Scholar

Tatematsu, A., Ishibashi, Y., Fukushima, N., and Sugawara, S. (2010). “Qoe assessment in haptic media, sound and video transmission: Influences of network latency,” in IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), Vancouver, BC, June 8–10, 2010.

CrossRef Full Text | Google Scholar

Xue, H., Zhao, T., Chen, W., Liu, Q., Zheng, S., and Chen, C. W. (2019). “Visual attention and haptic control: A cross-study,” in IEEE Fifth International Conference on Multimedia Big Data, Singapore, 11-13 Sept. 2019.

CrossRef Full Text | Google Scholar

Zeng, Q., Ishibashi, Y., Fukushima, N., Sugawara, S., and Psannis, K. E. (2013). “Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble,” in 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE), Tokyo, Japan, 1-4 Oct. 2013, 466–470.

CrossRef Full Text | Google Scholar

Zhang, W., and Liu, H. (2019). “Visual attention and haptic control: A cross-study.” in IEEE Fifth International Conference on Multimedia Big Data 11-13 Sept. 2019, Singapore

Google Scholar

Zhao, T. S., Liu, Q., and Chen, C. W. (2017). QoE in video transmission: A user experience-driven strategy. IEEE Commun. Surv. Tutorials 19, 285–302. doi:10.1109/comst.2016.2619982

CrossRef Full Text | Google Scholar