Attitudes and Expectations of Clinical Research Participants Toward Digital Health and Mobile Dietary Assessment Tools: Cross-Sectional Survey Study

Abstract

Background:

The adoption of health technologies is key to empower research participants and collect quality data. However, the acceptance of health technologies is usually evaluated in patients or healthcare practitioners, but not in clinical research participants.

Methods:

A 27-item online questionnaire was provided to the 11,695 members of a nutrition clinical research participant database from the Nantes area (France), to assess (1) participants' social and demography parameters, (2) equipment and usage of health apps and devices, (3) expectations in research setting and (4) opinion about the future of clinical research. Each item was described using frequency and percentage overall and by age classes. A global proportion comparison was performed using chi-square or Fisher-exact tests.

Results:

A total of 1529 respondents (81.0% women, 19.0% men) completed the survey. Main uses of health apps included physical activity tracking (54.7%, age-related group difference, p < 0.001) and food quality assessment (45.7%, unrelated to age groups). Overall, 20.4% of respondents declared owning a connected wristband or watch. Most participants (93.8%) expected the use of connected devices in research. However, protection of personal data (37.5%), reliability (35.5%) and skilled use of devices (28.5%) were perceived as the main barriers. Most participants (93.3%) would agree to track their food intake using a mobile app, and 80.5% would complete it for at least a week while taking part in a clinical study. Only 13.2% would devote more than 10 min per meal to such record. A majority (60.4%) of respondents would accept to share their social media posts in an anonymous way and most (82.2%) of them would accept to interact with a chatbot for research purposes.

Conclusions:

Our cross-sectional study suggests that clinical study participants are enthusiastic about all forms of digital health technologies and participant-centered studies but remain concerned about the use of personal data. Repeated assessments are suggested to evaluate the research participant's interest in technologies following the increase in use and demand for innovative health services during the pandemic of COVID-19.

Attitudes and expectations of clinical research participants towards digital health and mobile dietary assessment tools: Main results of a cross-sectional survey study.

Introduction

Patients and Health Care Workers Attitudes Toward Digital Health

There is an increasing use and demand in health technologies going hand in hand with hardware penetration in the global population (1) and the overall automation of daily life, expected to improve society, economy and quality of life (2). While public awareness in digital health is rising (3), attitudes toward health technologies may vary, as some users can be interested in managing appointments and the self-tracking of fitness, diet (4), or vital signs (3, 5) as reflected by recent surveys (3, 4). Recently, patients reported the frequent use of health technologies, including websites (while decreasing), mobile apps, electronic medical records, with a noticeable trend for wearable and smart devices as well as social media (3). Intention to use such technologies has been assessed in samples of patients (6–8), students (9), health workers (10) and in multiple groups at the same time (11–13) to identify demographic or socio-economic determinants of the successful implementation of health technologies. However, these determinants are not usually evaluated in groups of clinical study participants specifically, and it is not known whether the recent trends associated with the use of health technologies are consistent with expectations from research participants in clinical settings.

The Specific Case of Clinical Research Participants

Volunteering for a clinical study is a personal choice (14), which can be driven by the potential of benefiting personally as well as the possibility to help others (15): Despite the risks associated with the exposure to novel methods and solutions, research participants remain motivated by a desire to contribute to science (16). We can therefore expect that study participants are keen to complete study-specific surveys, monitor their vital signs, or share their health information with investigators. Patient empowerment has been defined by Affinito et al. as “the control of patients over their health and condition, as well as their ability to be more involved in their healthcare” (17). The digital transformation of healthcare allows patients to manage their conditions by facilitating diagnosis, prevention, and treatment (17–19). Therefore, the implementation of mobile technologies in clinical research represents an opportunity to facilitate and streamline the collection of quality data (20), inform clinical care and decision-making (21) while emancipating research participants (22). While participant engagement is getting more attention from sponsors (23), multiple tools may be implemented to collect more data and empower research participants. However, it was reported that the average number of protocol endpoints and procedures recently doubled in <10 years to support secondary and exploratory parameters (24). In a context of clinical research transformation supported by the use of technology, the overuse of tools or the use of inappropriate ones may bear the risks of unnecessary burden and lack of compliance, leading to missing or incorrect data and early dropout (21). Assessing participants' equipment and intention to use tech-enabled health solutions is key to ensuring the scientific validity of research studies. Still, only a few studies recently evaluated how health technologies are perceived among clinical study participants. One study recently evaluated the indicators of retention of a large sample of research participants in remote, digital setting (25). Some initiatives, like the Trial Feedback Questionnaire (26) were developed and may be used to gather participant feedbacks on the tools that are used in clinical studies. However, this questionnaire can only be used in a single indication or study at a time and after completion, preventing a proactive implementation in research settings (27). Meanwhile, quality and relevance of research can be improved by patient and public involvement in research, which is not systematically considered in research protocols (28).

Survey Objectives

As clinical research participants are conclusively affected by study designs and by the implementation of technologies (28), we considered them as our survey study population. Our objective was to evaluate their attitudes and expectations toward digital patient-generated health data and food tracking mobile apps and understand if their choices are associated with age groups.

Methods

Survey Design

Surveys are a cost-effective and non-interventional option to collect qualitative and semi-quantitative information to enable further research (29). An original survey was designed to understand expectations and concerns of clinical research participants toward digital health. A brief introduction was provided to describe the aims of the survey. The first section was designed to collect social and demography parameters. As research participants often have the possibility of bringing their own pocket-sized solutions instead of being provided with study-specific hardware, it was decided to investigate their equipment and their expectations in clinical research separately. Current equipment was assessed in a second section, while expectations in research settings were investigated in a third one. Furthermore, the appeal for rather advanced technologies which are not commonly used, such as sharing of social media posts and interactions with chatbots, is not frequently evaluated. As the sharing of such information can be perceived as controversial, those items were explored separately at the end of the questionnaire and was the purpose of a fourth section. Due to the limited published work investigating research participants interest and expectations toward in digital health, main items were designed to understand appeal for popular technologies (3–5), and understand concerns raised by their implementation in research. The use of free text was avoided as much as possible to focus on qualitative modalities. In this study, we referred to “connected devices” when assessing participant's interest in wearable and non-wearable smart health monitoring solutions.

Items of the survey were provided to members of the personnel affiliated to the sponsor and investigator to gather feedback during its design phase. The list of items was eventually narrowed down to a 27-item multiple-choice questionnaire (Additional File 1). To avoid underestimation of both expectations and especially concerns associated with the use of digital tools, it was decided not to limit the number of answers when selecting the modalities. Questions were formulated to limit ambiguity as much as possible.

Recruitment of Respondents

No health or personal data was collected as the survey items were designed to avoid the collection of sensitive or personally identifiable information. Participants were considered eligible to participate in this survey study if they were 18 years old or older, and currently screened or enrolled in a clinical study. Former clinical research participants were also considered eligible. No quota-based sampling was performed. A total of 11,695 members (8,386 women, 3,309 men) of a clinical research participant database (owned by Biofortis) from the Nantes region in France were contacted by email in May 2019 to read an information sheet and complete this anonymous questionnaire. The entire panel was contacted. The participating site, specialized in the conduct of nutrition clinical studies, was selected based on the research team's interest in the survey study and their ability to recruit a large sample of respondents from various age ranges and conditions, including healthy volunteers.

Participants completed an online version of the questionnaire hosted on Microsoft Forms (30) which allowed them to remain anonymous. No individually identifiable information or health data was collected, and no risk associated with data privacy was identified. Following a legal opinion from an independent expert consultant, this survey was not considered a clinical study requiring approval from an independent ethics committee. The survey was available online from the 23rd of May 2019 to the 1st of July 2019. No reminders were sent during this period.

Statistical Analyses

Each item of the survey was described using frequency and percentage overall and by age classes. For each item, a global proportion comparison was performed between the age classes using the chi-square test, or the Fisher-exact test if any expected cell count was inferior to 5 as an alternative to the chi-square test. In case of global effect, pairwise proportion comparisons between age classes were completed using chi-square test or Fisher-exact test accordingly. The results of comparisons were adjusted to account for multiple comparisons according to the Bonferroni-Holm method. All analyses were performed using SAS v9.4®. A margin error of 5% was used for statistical tests.

Results

Study Participants

All 11,695 members of the clinical research participants database (8,386 women, 3,309 men) were contacted to complete our survey. Among the 1,529 respondents who completed the survey, 81.0% were female (Figure 1). A share of 14.77% of women accepted to complete the survey (1239/8386), while 8.76% of men (290/3309) completed it. Most of respondents were aged 25–54 years old (64.9%) and lived in medium-size (10,000–50,000 inhabitants) and large cities (>50,000 inhabitants) (28.6 and 40.0%, respectively). While 35.4% of them had received primary education, 15.0% reached third level education. However, 11.1% declared having not received any formal education (Table 1).

Figure 1

Smartphone Use, Expectations and Concerns

Most (95.1%) reported owning a smartphone, with the younger age-classes (18–44 years; 100% in the 18–24, 98.8% in the 25–34 and 98.8% in the 35–44) showing a marked difference with the oldest (45 and over; 93.8% in the 45–54, 91.7% in the 55–64 and 84.6% in the 65 and over) (p < 0.001).

Regarding the use of health-related mobile apps, 78.1% of participants declared using at least one of them and 54.6% declared using two or more of them. Statistically significant differences were observed between age groups (p < 0.001): 71.6% of the participants aged 18–24 and 64.5% aged 25–34 declared using at least two health-related mobile apps whereas they were 45.3% aged 55–64 and 32.2% aged 65 or over to report the same use. However, there was no between-age-groups difference regarding neither the frequency of use of such apps (p = 0.093) with a majority of participants (76.2%) using them at least once a week, nor the duration of use (p = 0.376) with a majority of participants declaring a duration of 1–5 min per use (56.6%).

The main expectation toward such apps was the ability to monitor their physical activity (54.7%) which was more prevalent (p < 0.001) in the participants up to 44 years (ranging from 57.6 to 63.7%) as compared to the oldest ones from 55 and over (ranging from 43.2 to 53.6%). The ability to assess food quality arrived second (45.7%) without between-age-group difference. About one third of the study participants pointed out weight loss monitoring, with a preference for the individuals up to 44 years (p < 0.001). Other marked expectations included maintaining/improving their health and sleep monitoring. The latter was age dependent (p < 0.001) as it appeared particularly important for the youngest participants (51% of the 18–24 years old) and not that much for the ones aged 65 and over (17.2%). Interestingly, few (7.8%) declared having no expectation, but the feature was more present in the oldest classes (p < 0.001). Details about other expectations are provided in Table 2.

The most frequently expressed concern toward such apps was the frequency of advertisements (46.6%) irrespective of the age class of the respondents. Data protection (37.0%) and lack of reliability (23.7%) were also frequently expressed, both showing a difference in prevalence between age groups (p < 0.001 and p = 0.002, respectively). The youngest (18–24 years) participants were less concerned about personal data protection and more about reliability issues than their older counterparts. Details about other perceived issues are provided in Table 3.

Connected Watch/Wristband Use, Expectations and Concerns

Among participants to the survey, 20.4% declared using a connected watch or wristband, with a significant between-group difference (p = 0.050). Most of them declared using them for assessing physical activity (76.9%), monitoring sleep (39.4%), during sport (37.2%), for cardiovascular monitoring (32.4%) and receiving smartphone notifications (29.5%). Distribution of these uses were not different among the age classes of the participants, except for the latter (p = 0.008) for which respondents aged 65 years and over had no interest (4.3%).

The most frequent concerns regarding this type of device were concerns toward their price (46.2%) which was significantly more reported in the youngest classes than in the oldest (p < 0.001), and the lack of perceived usefulness (30.8%) which was more often reported in the oldest classes (p < 0.001). Results per age group and between-group differences are detailed in Table 4.

Expectations and Concerns Regarding These Connected Devices in Clinical Research

In the context of clinical research, 93.8% of participants were favorable to the use of connected (smart) objects with between-group differences (p < 0.001) with the youngest classes being more often inclined to use connected devices, irrespective of the type of device (p < 0.001 for weighting scales, smartwatches/wristbands, patches, plates and glasses). In the context of clinical research, the protection of personal data (37.5%) remained the main concern with the youngest class (18–24 years old) being significantly less often concerned than the oldest (p < 0.001). Other concerns included the reliability (35.5%) and the ability to use the device (28.5%), the former being more reported by the youngest study population (p < 0.001) and the latter by the oldest one (55 years and over, p > 0.001). These classes also more often dreaded the loss of human interaction and expressed more frequently concerns about the ease of use than the younger classes (p < 0.001 and p = 0.009 respectively). These results are presented in Figure 2.

Figure 2

During their participation in a clinical research, participants would mostly prefer having both on-site and remote conduct of a clinical study (68.1%), the oldest class preferring more often exclusive physical setting than the youngest (p < 0.001). The same significant tendency (p < 0.001) was noted regarding the best way to provide information regarding a clinical research protocol. Overall, respondents preferred a fun mobile app (50.2%) rather than paper explanation (25.0%) or video support and discussion with a professional (24.7%). The oldest class (65 years old and over) preferred paper explanations (43.8%) to mobile apps (36.7%). Regarding the way to report health-related data, mobile app questionnaire was preferred (64.2%) to computer questionnaire (29.0%) and paper questionnaire (6.8%) except in the oldest group in which computer questionnaire was preferred (58.6%) over the former (between age group difference: p < 0.001). Six out of ten participants would agree to share in a secure and anonymous system their social media content with research staff with significant difference between age groups (p < 0.001). Indeed, less than half of the respondents aged 65 years and over were inclined to share this content. Finally, 82.2% of participants would agree to interact with a chatbot and send their data to a member of the research team with notably the oldest class being less likely to agree than the others (p < 0.001). Results and responses per age group and between-group differences are detailed in Table 5.

Focus on Food Tracking Apps

Most of participants (93.3%) declared being prepared to keep a systematic record of their food consumption using a mobile app, with the oldest agreeing less often than the youngest (p < 0.001). Among them, 80.5% would agree to complete such record for at least a week and 52.6% for 4 weeks or over with significant between-group difference (p = 0.010). Regarding the expected completion time, 38.1% of the respondents would agree to devote up to 3 min per meal, and 44.1% of the respondents would agree to devote up to 10 min per meal to such record, with no significant between-group difference (p = 0.070). Results and responses per age group and between-group differences are detailed in Table 6.

Discussion

Principal Findings

By considering clinical study participants as a specific population, we confirmed their appeal for wearables and health “in pocket” technologies in research setting. This inclination to use digital tools is consistent with recent results obtained in general population (31). We also confirmed the rising trends associated with the use of smartphone apps, smart devices, and social media in medical research, which were previously reported in surveys conducted in healthcare users (3). Additionally, we were able to better understand the research participants' interest in tools of uncertain appeal, including dietary assessment tools (by considering the expected completion time and total duration), sharing of social media information, as well as chatbot conversations. Even though participants' equipment and enthusiasm seemed more important in younger age groups, our results suggest that the interest of older age classes should not be underestimated: most of these respondents expect the use of more connected tools in research practice, whatever their age. Besides, our results show that the use of paper diaries was only preferred on rare occasions, suggesting that participants recruited with the use of online tools should be provided with mobile apps over computer or paper solutions. Indeed, we identified that mobile apps are expected to provide informed consent and track health data during study conduct. Among these options, the use of a mobile app as a food diary represents a good compromise to track food intake at the touch of a finger, if its completion time does not exceed 3 min per meal. The use of mobile apps as food diaries has already been identified as a way to reduce the time to report dietary data and improve its quality (32, 33). We also noted that participants' enthusiasm toward the use of more connected tools does not go hand in hand with the conduct of remote studies. If given a choice, participants confirmed that they would rather opt for a hybrid study design, relying on both on-site and digital conduct. This can be explained, in part, by the loss of human contact which was perceived by respondents as one of their concerns in case of the use of health technologies. Our results suggest that the conduct of fully remote studies should not be considered systematically. We also identified that the protection of personal data was the most frequently reported concern, suggesting that more efforts are needed to describe risks associated with the management of individual health data and the use of digital health solutions (34).

While participants highlighted the management of their personal data as their main concern, most of the respondents would agree to share their social media content with researchers. Health information is already shared by many web users on web forums and digital platforms (35, 36). The processing of online natural language has been identified as an opportunity to enable sentiment analysis (37) or assess quality of life (38) which can be used as a complement to conventional methods of heath information tracking (39) by considering environmental, psychological and lifestyle factors (40, 41). Interestingly, even more participants confirmed that they would agree to interact with a chatbot during the conduct of clinical studies. Chatbots, while extensively used to facilitate hospital admissions and anticipate health checkup to support decision-making, are not currently used in many research studies for data acquisition purposes. Part of the research fields include mental health studies (42), including trials (43), and are also considered in behavior change studies (44), pediatric studies (45) and to evaluate how they can improve the management of chronic diseases (46, 47). Considering the importance of patient engagement in research (23, 28), we believe that their enthusiasm toward social media features and chatbots should also be leveraged to optimize retention rate throughout study participation.

Strengths and Limitations

By conducting this survey study, we were able to obtain answers from many research participants interested in nutrition clinical studies, with an important variability in terms of age and level of education and the use of a detailed online questionnaire. To our knowledge, this is the first assessment of both equipment and expectations toward digital health and nutrition apps that was done by considering participants to nutrition clinical studies as a specific survey population. We believe that the findings of this cross-sectional survey study can serve as a starting point for repeated assessments in samples of clinical research participants to increase reproducibility in additional context and geographies. To that extent, our questionnaire is provided as Supplementary Material and may be used under Creative Commons license for non-commercial purposes (Additional File 1). While this survey mushroomed into an important dataset, comparison with prior work is however limited, as we could not identify prior research findings related to expectations toward digital heals and mobile dietary assessment tools obtained in samples of clinical research participants.

A main limitation of our study is the absence of quota-based sampling which would have facilitated further analyses in sub-groups of respondents. While feasible on a larger scale, the focus on clinical study participants as our study sample prevented the use of further screening efforts and of a larger sample of respondents. Ideally, variables including age, gender and chronic conditions could have been used to enable a quota-based sampling reflecting the French population characteristics. Another limitation of our study is the lack of information regarding the nature of chronic conditions. This information would be needed to further explore the relationship between specific indications, nutrition-related concerns, and expectations toward digital health in research settings. However, the type of chronic condition was considered as a sensitive health data for this online survey study, considering applicable data privacy regulations. Finally, the data acquisition method can also be associated with a willingness to participate in our survey study. As the respondents are participants in nutrition clinical studies who were invited to complete an online questionnaire, they can be more likely to provide study data, including nutrition information, with the use of digital tools. We therefore identified a risk of response bias when interpreting results related to the preferred way of providing health and nutrition data while taking part in a clinical study.

The successful implementation of participant-centered digital tools also relies on their endorsement by research sponsors and investigators. While our results suggest an important appeal of clinical research participant toward health technologies, the viewpoint of investigational sites personnel should also be assessed, as usage by health practitioners and may not necessarily correlate with positive assessment (11). Further work is therefore needed to continue to evaluate the drivers of successful implementation of digital health technologies based on acceptability of digital solutions by health practitioners (47). These drivers may include the availability of site staff to manage new tools and train participants and may be evaluated by conducting qualitative survey studies. Such studies may consider the opinion of both research staff and clinical study participants to identify the predictors of the implementation of digital health technologies (and associated bottlenecks) in clinical research settings.

The screening period should also be considered when interpreting our results, as enrolment was performed ahead of the COVID-19 worldwide outbreak. Despite the availability and democratization of novel health technologies in clinical practice ahead of this pandemic, some patients experienced telemedicine for the first time during this pandemic as confirmed by recent surveys (48). This surge was not only explained by concerns related to coronavirus disease, but also due to the increasing use and demand in telemedicine for behavioral health and chronic conditions to cope with the lack of conventional face to face appointments during the pandemic (49). Following the currently rising use and demand in telehealth services, we expect clinical study participants' appeal for digital health will continue to increase. Therefore, there is an opportunity to repeat our assessment regularly to better support the implementation of health technologies when designing clinical studies based on more recent results.

Conclusion

Our study suggests that clinical study participants are keen to use various forms of digital health tools in clinical research setting, including digital health apps, food tracking apps, social media information sharing and interactions with chatbots. However, we also identified that clinical study participants remain concerned about the management of their personal data, the reliability of digital health solutions and the loss of interactions with investigators. Specific attention to ensure a high level of data security and privacy in future clinical studies should contribute to an enhanced satisfaction and trust from the participants and consequently to a higher participation rate and the collection of quality data. We believe that these results will provide supportive details for investigators and sponsors to foster the conduct of more participant-centered studies. Further work is however needed to better understand the association between interest in technologies and specific chronic conditions. Repeated assessments are suggested, as clinical research participants' propensity for technology is expected to continue to evolve following the recent surge in use and demand for innovative health services during the pandemic of COVID-19.

Funding

This work was funded by Danone Nutricia Research.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  SteinhublSRMuseEDTopolEJ. The emerging field of mobile health. Sci Transl Med. (2015) 7:283rv3. 10.1126/scitranslmed.aaa3487

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  European commission. Attitudes Towards the Impact of Digitisation Automation on Daily Life. (2017). Available online at: https://ec.europa.eu/digital-single-market/en/news/attitudes-towards-impact-digitisation-and-automation-daily-life (accessed January 25, 2021).

  • Google Scholar

• 3.

  Accenture. 2018 Consumer Survey on Digital Health. Available online at: https://www.accenture.com/t20180306t103559z__w__/us-en/_acnmedia/pdf-71/accenture-health-2018-consumer-survey-digital-health.pdf (accessed January 25, 2021).

  • Google Scholar

• 4.

  HossainILimZZNgJJLKohWJWongPS. Public attitudes towards mobile health in Singapore: a cross-sectional study. Mhealth. (2018) 4:41. 10.21037/mhealth.2018.09.02

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  AjanaB. Digital health and the biopolitics of the quantified self. Digit Health. (2017) 3:2055207616689509. 10.1177/2055207616689509

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  GrasslNNeesJSchrammKSpratteJSohnCSchottTCet al. A web-based survey assessing the attitudes of health care professionals in Germany toward the use of telemedicine in pregnancy monitoring: cross-sectional study. JMIR Mhealth Uhealth. (2018) 6:e10063. 10.2196/10063

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  JenssenBPMitraNShahAWanFGrandeD. Using digital technology to engage and communicate with patients: a survey of patient attitudes. J Gen Intern Med. (2016) 31:85–92. 10.1007/s11606-015-3517-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  ManginDParascandaloJKhudoyarovaOAgarwalGBismahVOrrS. Multimorbidity, eHealth and implications for equity: a cross-sectional survey of patient perspectives on eHealth. BMJ Open. (2019) 9:e023731. 10.1136/bmjopen-2018-023731

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  MontagniICariouTFeuilletTLangloisETzourioC. Exploring digital health use and opinions of university students: field survey study. JMIR Mhealth Uhealth. (2018) 6:e65. 10.2196/mhealth.9131

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  YadaNHeadM. Attitudes toward health care virtual communities of practice: survey among health care workers. J Med Internet Res. (2019) 21:e15176. 10.2196/15176

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  FlynnDGregoryPMakkiHGabbayM. Expectations and experiences of eHealth in primary care: A qualitative practice-based investigation. Int J Med Inform. (2009) 78:588–604. 10.1016/j.ijmedinf.2009.03.008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  WernhartAGahbauerSHaluzaD. eHealth and telemedicine: Practices and beliefs among healthcare professionals and medical students. PLoS ONE (2019) 14:e0213067. 10.1371/journal.pone.0213067

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  MayerGGronewoldNAlvarezSBrunsBHilbelTSchultzJH. Acceptance and expectations of medical experts, students, and patients toward electronic mental health apps: cross-sectional quantitative and qualitative survey study. JMIR Ment Health. (2019) 6:e14018 10.2196/14018

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  World medical association. World medical association declaration of helsinki: ethical principles for medical research involving human subjects. JAMA. (2013) 310:2191–4. 10.1001/jama.2013.281053

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  SacristánJAAguarónAAvendaño-SoláCGarridoPCarriónJGutiérrezAet al. Patient involvement in clinical research: why, when, and how. Patient Prefer Adherence. (2016) 10:631–40. 10.2147/PPA.S104259

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  WendlerDKrohmalBEmanuelEJGradyCESPRITGroup. Why patients continue to participate in clinical research. Arch Intern Med. (2008) 168:1294–9. 10.1001/archinte.168.12.1294

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  AffinitoLFontanellaAMontanoNBrucatoA. How physicians can empower patients with digital tools. J Public Health. (2020). Available online at: https://link.springer.com/content/pdf/10.1007/s10389-020-01370-4.pdf

  • Google Scholar

• 18.

  European Union Law. Communication From the Commission to the European Parliament, the European Council, the Council, the European Economic and Social Committee and the Committee of the Regions on enabling the Digital Transformation of Health and Care in the Digital Single Market; Empowering Citizens and Building a Healthier Society. Available online at: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A233%3AFIN (accessed January 30, 2022)

  • Google Scholar

• 19.

  US Food Drug Administration. FDA Announces New Steps to Empower Consumers and Advance Digital Healthcare. Available online at: https://www.fda.gov/news-events/fda-voices/fda-announces-new-steps-empower-consumers-and-advance-digital-healthcare (accessed January 30, 2022)

  • Google Scholar

• 20.

  SteinhublSRWolff-HughesDLNilsenWIturriagaECaliffRM. Digital clinical trials: creating a vision for the future. NPJ Digit Med. (2019) 2:126. 10.1038/s41746-019-0203-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  CalvertMKyteDDuffyHGheorgheAMercieca-BebberRIvesJet al. Patient-reported outcome (PRO) assessment in clinical trials: a systematic review of guidance for trial protocol writers. PLoS ONE. (2014) 9:e110216. 10.1371/journal.pone.0110216

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  CoravosAGoldsackJCKarlinDRNebekerCPerakslisEZimmermanNet al. Digital medicine: a primer on measurement. Digit Biomark. (2019) 3:31–71. 10.1159/000500413

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  LevitanBGetzKEisensteinELGoldbergMHarkerMHesterleeSet al. Assessing the financial value of patient engagement: a quantitative approach from CTTI's patient groups and clinical trials project. Ther Innov Regul Sci. (2018) 52:220–9. 10.1177/2168479017716715

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  Clinical Trials Transformation Initiative. Tufts CSDD Reports Number of Endpoints Per Protocol Doubled Between 2004 and 2012. Available online at: https://www.ctti-clinicaltrials.org/news/tufts-csdd-reports-number-endpoints-protocol-doubled-between-2004-and-2012 (accessed May 05, 2021)

  • Google Scholar

• 25.

  PratapANetoECSnyderPStepnowskyCElhadadNGrantDet al. Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. npj Digit Med. (2020) 3:21. 10.1038/s41746-020-0224-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  BrohanEBonnerNTurnbullAKhanSDewitOThomasGet al. Development of a patient-led end of study questionnaire to evaluate the experience of clinical trial participation. Value Health. (2014) 17:A649. 10.1016/j.jval.2014.08.2358

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  MansonSCBonnerNBennettBDewitOThomasG. Development, scoring and use of a patient questionnaire to measure the practical patient experience within oncology clinical trials. J Clin Oncol. (2015) 33:e20567. 10.1200/jco.2015.33.15_suppl.e20567

  • CrossRef
  • Google Scholar

• 28.

  AguayoGAGoetzingerCScibiliaRFischerASeuringTTranVet al. Methods to generate innovative research ideas and improve patient and public involvement in modern epidemiological research: Review, patient viewpoint, and guidelines for implementation of a digital cohort study. J Med Internet Res. (2021) 23:e25743. 10.2196/25743

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  YaddanapudiSYaddanapudiLN. How to design a questionnaire. Indian J Anaesth. (2019) 63:335–7. 10.4103/ija.IJA_334_19

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30.

  Microsoft Forms. Available online at: https://forms.office.com/ (accessed January 25, 2021).

  • Google Scholar

• 31.

  Accenture. Digital Health Technology Vision. (2020). Available online at: https://www.accenture.com/us-en/insights/health/accenture-digital-health-technology-vision-2020 (accessed January 25, 2021).

  • Google Scholar

• 32.

  AmoutzopoulosBSteerTRobertsCCadeJEBousheyCJCollinsCEet al. Traditional methods new technologies - dilemmas for dietary assessment in large-scale nutrition surveys and studies: a report following an international panel discussion at the 9th international conference on diet and activity methods (ICDAM9), Brisbane, 3 September 2015. J Nutr Sci. (2018) 7:e11. 10.1017/jns.2018.4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  KhazenWJeanneJFDemaretzLSchäferFFagherazziG. Rethinking the use of mobile apps for dietary assessment in medical research. J Med Internet Res. (2020) 22:e15619. 10.2196/15619

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  United States Federal Trade Commission. Does Your Health App Protect Your Sensitive Info? Available online at: https://www.consumer.ftc.gov/blog/2021/01/does-your-health-app-protect-your-sensitive-info (accessed January 25 2021).

  • Google Scholar

• 35.

  FoxSDugganM. Health Online. Pew Research Center. (2013). Available online at: http://www.pewinternet.org/2013/01/15/health-online-2013/ (accessed May 05 2021).

  • Google Scholar

• 36.

  NIH. National Cancer Institute. Health Information National Trends Survey. In the Last 12 Months, Have You Used the Internet for the Following Reason: Shared Health Information on Social Media Sites? Available online at: https://hints.cancer.gov/view-questions-topics/question-details.aspx?PK_Cycle=13&qid=1496 (accessed May 05, 2021).

  • Google Scholar

• 37.

  GabarronEDorronzoroERivera-RomeroOWynnR. Diabetes on Twitter: a sentiment analysis. J Diabetes Sci Technol. (2019) 13:439–44. 10.1177/1932296818811679

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  FagherazziGRavaudP. Digital diabetes: perspectives for diabetes prevention, management and research. Diabetes Metab. (2019) 45:322–9. 10.1016/j.diabet.2018.08.012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  SamarasLGarcía-BarriocanalESiciliaM-A. Syndromic Surveillance Models Using Web Data: A Systematic Review in Innovation in Health Informatics, 1st Edition, A Smart Healthcare Primer 39–77.New York, NY: Elsevier Science Publishing Co Inc., Imprint by Academic Press Inc. (2019).

  • Google Scholar

• 40.

  MaoJJChungABentonAHillSUngarLLeonardCEet al. Online discussion of drug side effects and discontinuation among breast cancer survivors. Pharmacoepidemiol Drug Saf. (2013) 22:256–62. 10.1002/pds.3365

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41.

  SchäferFFaviezCVoillotPFoulquiéPNajmMJeanneJFet al. Mapping and modeling of discussions related to gastrointestinal discomfort in french-speaking online forums: results of a 15-year retrospective infodemiology study. J Med Internet Res. (2020) 22:e17247. 10.2196/17247

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42.

  Abd-AlrazaqAAAlajlaniMAlalwanAABewickBMGardnerPHousehM. An overview of the features of chatbots in mental health: a scoping review. Int J Med Inform. (2019) 132:103978. 10.1016/j.ijmedinf.2019.103978

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43.

  FitzpatrickKKDarcyAVierhileM. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled Trial JMIR Ment Health. (2017) 4:e19. 10.2196/mental.7785

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  PereiraJDíazÓ. Using health chatbots for behavior change: a mapping study. J Med Syst. (2019) 43:135. 10.1007/s10916-019-1237-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  ComendadorBEVFranciscoBMBMedenillaJSMaeS. Pharmabot: a pediatric generic medicine consultant chatbot. J Automat Control Eng. (2015) 3:137–40. 10.12720/joace.3.2.137-140

  • CrossRef
  • Google Scholar

• 46.

  KowatschTNißenMShihCHRüeggerDVollandDFillerAet al. Text-based healthcare chatbots supporting patient and health professional teams: preliminary results of a randomized controlled trial on childhood obesity. Persuasive Embodied Agents for Behavior Change (PEACH2017) Workshop, co-located with the 17th International Conference on Intelligent Virtual Agents (IVA 2017). Stockholm (2017).

  • Google Scholar

• 47.

  FagherazziGGoetzingerCRashidMAAguayoGAHuiartL. Digital health strategies to fight COVID-19 worldwide: challenges, recommendations, and a call for papers. J Med Internet Res. (2020) 22:e19284. 10.2196/19284

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 48.

  French Digital Health Agency (Agence du numérique en Santé) – 2021 Survey Evaluating the Implementation of Telemedecine. Available online at: https://esante.gouv.fr/sites/default/files/media_entity/documents/odoxa-pour-lans-et-le-mag-de-la-sante—barometre-telemedecine-vague-3—janvier-2021.pdf (accessed January 25, 2021).

  • Google Scholar

• 49.

  Uscher-PinesLThompsonJTaylorPDeanKYuanTTongIet al. Where virtual care was already a reality: experiences of a nationwide telehealth service provider during the COVID-19 pandemic. J Med Internet Res. (2020) 22:e22727. 10.2196/22727

  • Pubmed Abstract
  • CrossRef
  • Google Scholar