Attrition and representativeness in development and validation of online symptom checkers—a case study on the Rheumatic? Questionnaire

Abstract

Background:

Online symptom checkers are often developed and validated on data subject to self-selection and selective attrition, potentially introducing biases in prediction models.

Objectives:

To assess recruitment, selection, and attrition patterns in a large Dutch online symptom checker for musculoskeletal complaints and to evaluate potential biases by comparing participant characteristics across recruitment sources and with external target populations.

Methods:

Using data from the online Dutch Rheumatic? Questionnaire on musculoskeletal complaints, we compared baseline characteristics and key self-reported symptoms between responders to the follow-up survey and nonresponders. The survey responders were furthermore compared according to source of recruitment to the questionnaire, i.e., via primary care clinics, secondary care clinics, or via different online sources. Sex, age and BMI distributions from the total study group were compared to external data of potential target populations of primary and secondary care patients within the Netherlands.

Results:

The total study group of answers to the questionnaire comprised 31,457 responders, of which 50% (n = 15,591) responded to the follow-up survey. Study participants were predominantly female (76%), middle-aged (one-third 50–60 years), never-smokers (66%), and overweight. While participants recruited through healthcare settings resembled target populations, follow-up survey responders were older, had more rheumatic diagnoses (49% vs. 32%), and reported more symptoms than non-responders. Participant characteristics varied by recruitment source, with social media attracting younger females while healthcare routes reached more diverse populations with varying symptom presentations.

Conclusion:

Patterns of recruitment and attrition produced differences in participant characteristics. Healthcare-based recruitment yielded participants resembling intended target populations, and follow-up survey responders differed on some points from nonresponders. Awareness of these selection processes is essential when using real-world symptom checker data for model development.

Introduction

Online symptom checkers (OSCs) and digital triage tools promise the possibility of optimizing the diagnostic process, as well as aiding health professionals in decision making, therefore aiding the promotion of health equity (Abramoff et al., 2023). For an OSC to be of value, it is crucial that it performs well on the intended target population. Many OSCs rely solely on textbook knowledge or are validated only using clinical vignettes or physician review of patient records (Wallace et al., 2022; El-Osta et al., 2022; Fraser et al., 2018), limiting their validity. Current recommendations promote that OSCs should be developed and validated using real-world data from broadly representative populations (Wallace et al., 2022; Sperrin et al., 2022).

However, using real-world data introduces methodological challenges. The PROBAST (Prediction model Risk Of Bias ASsessment Tool) guidelines describes several domains of risk of bias when using real-world data for model development (Wolff et al., 2019). Particularly, they highlight selective participation and incomplete follow-up. If certain groups such as healthier users or individuals with milder symptoms, are less likely to participate or complete follow-up, systematic bias may arise, leading to reduced generalizability of prediction models to the intended users (Wolff et al., 2019; Lasko et al., 2024; Bethlehem, 2010; Gustavson et al., 2012; de Leeuw and Lugtig, 2015; Keiding and Louis, 2016). Selective dropout may further distort associations between predictors and outcomes, particularly when outcomes depend on follow-up surveys. Therefore, evaluating recruitment sources, participant characteristics, and follow-up completeness is essential to understand potential selection bias when real-world data are used for OSC development.

Although potential problems with selective participation, such as the risk of collider bias, have been discussed (Munafo et al., 2018), few studies so far have described the selection processes in real-world online recruitment for OSCs.

The European SPIDeRR project (Stratification of Patients using advanced Integrative modeling of Data Routinely acquired for diagnosing rheumatic complaints) offers a relevant test case. The project aims to develop an OSC specifically for people with musculoskeletal complaints (MSCs). Musculoskeletal symptoms are highly frequent, comprising 22% of the consultations of primary care physicians in the Netherlands (der van Linden et al., 2004; Finckh, 2009; Raciborski et al., 2017; Jain et al., 2023; Knitza et al., 2021). Despite the benefits of early diagnosis, diagnostic delays remain common (Raciborski et al., 2017), prompting many individuals to seek online explanations for their symptoms (Jain et al., 2023; Knitza et al., 2021; Knevel et al., 2022).

SPIDeRR’s OSC, Rheumatic? (Knevel et al., 2022) is a questionnaire collecting information on symptoms, past diagnoses and basic characteristics (Lundberg et al., 2023). The OSC aims to provide accurate disease probabilities for different settings: (1) people online with MSCs, (2) general practitioners (GPs) using the OSC for triage and referral, and (3) for rheumatologists to optimize the diagnostic process. Participants are followed for 1 year to collect information on subsequent referral, diagnosis and medication.

In this study, we assessed recruitment, selection, and attrition patterns among users of the Rheumatic? Questionnaire. We compared participant characteristics across recruitment sources and with external data from Dutch primary and secondary care populations with MSCs. In addition, we examined selective follow-up by comparing baseline-only participants with those who completed the first follow-up survey.

Methods

Study design and setting

Rheumatic? (Elsa Science AB, 2024) is an online questionnaire about current symptoms, past diagnoses and demographic information (all questions provided in Supplementary Table 1). Participants were recruited through multiple channels: online advertisements, the website and newsletters of the Dutch Arthritis Society (ReumaNederland), primary care (GP) clinics, and secondary care rheumatology outpatient clinics. Participants consented to be contacted for follow-up. Four follow-up surveys were sent via e-mail. The first immediately after the baseline questionnaire (included questions about source of recruitment, user-experience and further diagnoses), and the last three after 3, 6 and 12 months, including questions regarding subsequent rheumatologic diagnoses, medication usage, GP visits and referrals.

Data processing

We included Dutch adults who completed Rheumatic? Between 24 July 2021 and 13 July 2023 and provided informed consent. Responses were excluded if demographic information was missing or if email addresses were missing, duplicated, or invalid, ensuring that follow-up questionnaires could be delivered and linked. For the first follow-up survey, only fully completed surveys submitted within 7 days of baseline were included.

Variables

For the present study we considered age, sex, height, weight, smoking and alcohol habits, rheumatic diagnosis within near family and key symptoms (pain, morning stiffness, all day stiffness, exhaustion, reduced endurance or none). Age was classified according to <40, 40–50, 50–60 and 60+ years. Body mass index (BMI) was derived from midpoint values of weight and height categories (5 kg and 5 cm intervals) and grouped as <18.5, 18–24.9, 25–30 and 30+ (Weir and Jan, 2024). Family diagnosis was defined as reporting a rheumatic-related diagnosis within close family (yes/no/do not know). Previous diagnosis was defined as reporting any previously given rheumatic-related diagnosis at baseline or in the follow-up survey (yes/no/do not know). Recruitment sources (via a rheumatology clinic, GP, the ReumaNederland website/newsletters, other online sources (with a free text option) or other sources) and further rheumatic diagnoses were solely indicated in the follow-up survey.

Website analytics

Daily inclusion counts were compared with daily numbers of Dutch new visitors to the website (Elsa Science AB, 2024) hosting the questionnaire between July 2021 and December 2023. All website analytics were summarized using Google Analytics (Google Analytics, 2024).

Expected target populations

To assess representativeness, we collected data from the same time-period (24 July 2021 to 13 July 2023) from (i) all GP episodes with MSCs-related ICPC-codes (L01–L20, L29, L83–L94, L98, L99, T92, thus excluding MSC consultations with fractures, infections and innate diseases) (Hofmans-Okkes and Lamberts, 1996; Ardesch et al., 2023), (ii) regional rheumatology clinics in the South West Netherlands (RZWN), and (iii) the academic rheumatology clinic of Leiden University Medical Centre (LUMC). Age, sex and BMI (when available) were compared between the study population and these potential target populations.

Statistical analysis

Contingency tables summarized baseline characteristics of all participants and compared responders vs. nonresponders to the first follow-up survey. Differences in percentages were presented with 95% confidence intervals (CIs) and visualized in a forest plot.

Recruitment source differences were assessed using contingency tables stratified by reported source of recruitment coded as GP, rheumatology clinic, other sources in general, and online sources. The latter was subdivided based on free text answers into the ReumaNederland website, Facebook, Google, Instagram, or other online sources. We used only participants’ answers to the first follow-up survey since this survey contained information on sources of recruitment.

Representativeness relative to target populations was assessed by comparing distributions of age, sex, and BMI between responders to the follow-up survey and external clinical populations, grouped as above. In the external data, BMI was only calculated when available.

Recruitment patterns over time were examined using frequency plots of inclusion date and Google hit-data with 7-day rolling means and furthermore stratified by recruitment source for responders to the follow-up survey.

All data handling and analysis were performed using R version 4.1.3 (R Core Team, 2023), using ggplot2 for plots (Wickham, 2016), and gt and gtsummary for tables (Sjoberg et al., 2021; Iannone et al., 2024).

Results

Characteristics of all Rheumatic? Participants

After data processing 31,159 participants were included (Supplementary Figure 1). Participants were predominantly female (76%), most commonly aged 50 to 60 years (31%), never-smokers (66%) and frequently overweight. About half (51%) reported no previous rheumatic diagnosis, and 48% reported a close family member having a rheumatic diagnosis. Pain (83%) and morning stiffness (62%) were the most frequent symptoms. 29% reported swelling in one or more body parts and 1% reported having none of the key symptoms. Characteristics of participants are fully specified in Supplementary Table 2.

Differences between follow-up survey participants and non-participants

Of all participants, half (N = 15,591) responded to the first follow-up survey. Characteristics of follow-up responders and nonresponders are presented in Supplementary Table 2, and differences in percentage are shown in Figure 1 and Supplementary Table 2. Follow-up participation did not yield differences in sex, alcohol consumption, family history of rheumatic disease, or having a complete absence of key symptoms. However, responders were older, more frequently reported a prior rheumatic diagnosis (49% vs. 32%), were less frequently current smokers and more often had a BMI above 25. Responders more frequently reported key symptoms, except for exhaustion which was reported more frequently among nonresponders than responders.

Figure 1

Characteristics by source of recruitment

Most of the participants with information on recruitment [12,574 (81%)] were recruited online, with the largest proportion originating from ReumaNederland communication channels, Google, and social media platforms (Table 1). Smaller proportions were recruited through secondary care rheumatology clinics, general practitioners (GPs), or other offline sources.

• Social media recruitment predominantly reached younger women, with women accounting for 95 to 98% of participants recruited via Facebook and Instagram. These participants also showed slightly higher BMI values, with the highest prevalence of obesity observed among Facebook recruits.

• Other online recruitment sources, including Google and disease-specific websites, reached a more heterogeneous population. Participants recruited via Google less often had a prior rheumatic diagnosis.

• Healthcare-based recruitment through GPs and rheumatology clinics resulted in more balanced sex distributions and older age profiles. Participants recruited via rheumatology clinics most frequently reported swelling, while those recruited through GPs reported fewer symptoms overall (Figure 2).

Figure 2

Family history of rheumatic disease varied modestly across recruitment sources, and smoking patterns showed little variation.

Recruitment tendencies

Daily recruitment rates varied over time and are shown in Figure 3. Distinct recruitment peaks coincided with newsletter campaigns conducted by ReumaNederland, annual World Arthritis Day and targeted social media campaigns. Source-stratified analyses highlighted the effect of paid social media campaigns on increases in recruitment, whereas organic posts on the ReumaNederland social media site had limited impact (Supplementary Figure 2). Daily recruitment closely followed traffic to the questionnaire website (Supplementary Figure 3).

Figure 3

Comparison with target populations in primary and secondary care

Compared with patients from Dutch secondary care rheumatology clinics, the overall Rheumatic? Cohort included a higher proportion of women. However, participants recruited via secondary care closely resembled secondary care patients with respect to sex. Patients from the academic rheumatology clinic were younger and were more present in the lower BMI groups. However, participants from the regional rheumatology clinic resembled Rheumatic? Rheumatology recruits more closely (Table 2).

In comparison with primary care data from GPs, women were overrepresented in the overall Rheumatic? Cohort. This imbalance was not observed among participants recruited through GPs, whose sex distribution closely matched that of the primary care population. The overall Rheumatic? Cohort held relatively more middle-aged participants (50–60 years), and those recruited via GPs tended to be older than individuals in the external GP data. BMI distributions were comparable between all groups (Table 2).

Discussion

In this cohort study of more than 31,000 individuals with musculoskeletal symptoms using the Rheumatic? Online symptom checker (OSC), we identified distinct recruitment, selection, and attrition patterns with direct implications for representativeness and potential bias in real-world OSC data. Recruitment through primary and secondary care settings yielded participant characteristics closely resembling their intended target populations, whereas online recruitment—particularly via social media—resulted in younger and predominantly female participants. In addition, follow-up survey responders differed systematically from nonresponders, indicating systematic attrition.

Our findings are consistent with prior studies of non–disease-specific OSCs, which reported a predominance of female users (62–85%) and younger mean ages (Kopka et al., 2023; Winn et al., 2019; Arellano Carmona et al., 2022; Morse et al., 2020). Pain was also the most frequently reported symptom in those studies, mirroring our observations. Studies on OSCs specifically targeting musculoskeletal complaints remains limited. A small German study evaluating musculoskeletal-focused OSCs and an Australian primary care study reported sex and age distributions broadly comparable to our findings, although sample sizes were substantially smaller (Knitza et al., 2021; Haas et al., 2023). Together, these data suggest that online symptom checkers for musculoskeletal complaints consistently attract selective user groups, particularly when participants are recruited outside healthcare settings.

Recruitment source and representativeness

Participants recruited through healthcare settings closely matched external primary and secondary care populations with respect to sex, age, and BMI. This alignment suggests that OSC deployment within clinical pathways may yield datasets representative of these intended users. In contrast, participants recruited online were younger and more frequently female, with notable variation between online recruitment channels. Social media based recruitment resulted in an almost exclusively female population, whereas Google-based recruitment reached a more heterogeneous group with fewer prior diagnoses. Importantly, our findings demonstrate that “online users” do not constitute a homogeneous population. Marked differences in demographic and clinical characteristics were observed across online recruitment sources. As other authors previously have suggested, these patterns likely reflect differences in platform use, content targeting, and health-seeking behavior, underscoring that recruitment strategy itself is a key determinant of dataset composition (Bol et al., 2020).

Selective attrition and follow-up

Follow-up survey responders differed from nonresponders in age, symptom burden, smoking status, and prevalence of prior rheumatic diagnoses. A follow-up response rate of 50% is comparable to other longitudinal studies, including surveys in rheumatology and other domains (Garcia et al., 2014; Wu et al., 2022). The higher rate of follow-up participation among older participants and nonsmokers aligns with previous population-based studies (Okpara et al., 2023; Canivet et al., 2021; Bamer et al., 2020). However, the greater symptom burden and higher prevalence of prior diagnoses among responders suggest that disease severity and health awareness influenced continued participation.

Implications for prediction model development

Selective attrition complicates the interpretation and generalizability of analyses relying on follow-up data. If associations between predictors and outcomes differ between responders and nonresponders, prediction models derived from follow-up data may be biased. Multiple imputation and sensitivity analyses under alternative missingness mechanisms may be considered to assess robustness (Sterne et al., 2009; van Buuren, 2018). Where feasible, linkage to external data sources may help validate outcomes and improve handling of missing data. In the Rheumatic? Study, planned linkage with electronic health records from primary and secondary care may address these limitations.

Before developing prediction models using OSC data, the intended use and target population should be clearly defined. For Rheumatic?, potential applications range from a general online self-assessment tool to decision support in primary or secondary care. Each use case implies a different target population and, consequently, different requirements for representativeness.

When discrepancies between the study population and target population exist, statistical approaches such as reweighting or poststratification may improve transportability (Wolff et al., 2019; Keiding and Louis, 2016; Petersen et al., 2024; Zhang et al., 2014). Additionally, when recruitment sources are strongly associated with participant characteristics, incorporating recruitment source as a predictor or exploring interactions with other predictors may be warranted. Separate validation of prediction models across recruitment sources may further enhance robustness. More research is needed to compare different weighting and calibration methods in the case of making prediction models with multiple target groups from a heterogeneous population.

Handling missing follow-up data

Missing follow-up data introduces additional methodological challenges. When missingness is associated with participant characteristics, standard assumptions such as missing at random must be carefully evaluated. Multiple imputation and sensitivity analyses under alternative missingness mechanisms may be considered to assess robustness (Sterne et al., 2009; van Buuren, 2018; Biering et al., 2015). Where feasible, linkage to external data sources may help validate outcomes and improve handling of missing data. In the Rheumatic? Study, planned linkage with electronic health records from primary and secondary care may address these limitations.

Strengths and limitations

This study provides one of the first large-scale, empirical descriptions of recruitment and participation patterns in an OSC targeting musculoskeletal complaints. The large sample size enabled detailed stratified analyses across recruitment sources and follow-up status. Although formal external reference data for the purely online target population were unavailable, the observed recruitment and attrition patterns reflect well-described mechanisms of self-selection and differential participation in digital health research. These mechanisms are unlikely to be specific to musculoskeletal complaints and may be relevant to a broad range of online symptom checkers and digital triage tools. This study is limited by incompleteness of external reference data: BMI was incomplete, and we lacked measures of socioeconomic status and digital literacy which could have provided extra insight in the comparison. Furthermore, recruitments peaks tied to specific campaigns may have altered the sample composition, limiting inferences about routine use of the symptom checker.

Conclusion

Our study underlines the challenges that arise when recruiting real-world participants online. Recruitment and attrition processes play a central role in shaping the composition of real-world online symptom checker datasets. In this study, we observed that patterns of recruitment and attrition created differences in participant characteristics. However, we also observed that recruitment via general practitioners and rheumatology clinics yielded participant samples resembling their intended clinical target populations. These patterns reflect general mechanisms of self-selection and differential participation in digital health research and underscore the importance of explicitly evaluating recruitment and attrition when developing and validating prediction models using real-world OSC data.

References

• 1

  AbramoffM. D.TarverM. E.Loyo-BerriosN.TrujilloS.CharD.ObermeyerZ.et al. (2023). Considerations for addressing bias in artificial intelligence for health equity. npj Digit. Med.6:170. doi: 10.1038/s41746-023-00913-9,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  ArdeschF. H.MeulendijkM. C.KistJ. M.VosR. C.VosH. M. M.de Kiefte- JongJ. C.et al. (2023). The introduction of a data-driven population health management approach in the Netherlands since 2019: the extramural LUMC academic network data infrastructure. Health Policy132:104769. doi: 10.1016/j.healthpol.2023.104769

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  Arellano CarmonaK.ChittamuruD.KravitzR. L.RamondtS.RamirezA. S. (2022). Health information seeking from an intelligent web-based symptom checker: cross-sectional questionnaire study. J. Med. Internet Res.24:e36322. doi: 10.2196/36322,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  BamerA. M.McMullenK.GibranN.HolavanahalliR.SchneiderJ. C.CarrougherG. J.et al. (2020). Factors associated with attrition of adult participants in a longitudinal database: a national institute on disability, independent living, and rehabilitation research burn model system study. J. Burn Care Res.41, 270–279. doi: 10.1093/jbcr/irz186,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  BethlehemJ. (2010). Selection bias in web surveys. Int. Stat. Rev.78, 161–188. doi: 10.1111/j.1751-5823.2010.00112.x

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  BieringK.HjollundN. H.FrydenbergM. (2015). Using multiple imputation to deal with missing data and attrition in longitudinal studies with repeated measures of patient-reported outcomes. Clin. Epidemiol.7, 91–106. doi: 10.2147/CLEP.S72247,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  BolN.StrycharzJ.HelbergerN.van de VeldeB.de VreeseC. H. (2020). Vulnerability in a tracked society: combining tracking and survey data to understand who gets targeted with what content. New Media Soc.22, 1996–2017. doi: 10.1177/1461444820924631

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  CanivetC.NilssonA.BjörkJ.MoghaddassiM.ÖstergrenP. O. (2021). Assessment of selection bias due to dropouts in the follow-up of the Scania public health cohort. Scand. J. Public Health49, 457–464. doi: 10.1177/1403494820919544,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  de LeeuwE. D.LugtigP. (2015). “Dropouts in longitudinal surveys,” in Wiley StatsRef: Statistics Reference Online, (Hoboken, NJ: John Wiley & Sons, Ltd.), 1–6.

  • Google Scholar
  • View reference in article

• 10

  der van LindenM.WestertG.de BakkerD.SchellevisF. (2004). Tweede Nationale Studie naar ziekten en verrichtingen in de huisartspraktijk. Klachten en aandoeningen in de bevolking en in de huisartspraktijk. Utrecht: NIVEL.

  • Google Scholar
  • View reference in article

• 11

  El-OstaA.WebberI.AlaaA.BagkerisE.MianS.SharabianiM. T. A.et al. (2022). What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study. BMJ Open12:e053566. doi: 10.1136/bmjopen-2021-053566,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  Elsa Science AB. (2024). Available onlinea at: https://rheumatic.elsa.science/en/. (Accessed April 4, 2025)

  • Google Scholar
  • View reference in article

• 13

  FinckhA. (2009). Early inflammatory arthritis versus rheumatoid arthritis. Curr. Opin. Rheumatol.21, 118–123. doi: 10.1097/BOR.0b013e3283235ac4

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  FraserH.CoieraE.WongD. (2018). Safety of patient-facing digital symptom checkers. Lancet392, 2263–2264. doi: 10.1016/S0140-6736(18)32819-8,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  GarciaI.PortugalC.ChuL. H.KawatkarA. A. (2014). Response rates of three modes of survey administration and survey preferences of rheumatoid arthritis patients. Arthritis Care Res.66, 364–370. doi: 10.1002/acr.22125,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  Google Analytics. (2024). Available onlinea at: http://google.com/analytics. (Accessed March 13, 2024).

  • Google Scholar
  • View reference in article

• 17

  GustavsonK.von SoestT.KarevoldE.RoysambE. (2012). Attrition and generalizability in longitudinal studies: findings from a 15-year population-based study and a Monte Carlo simulation study. BMC Public Health12:918. doi: 10.1186/1471-2458-12-918,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  HaasR.GorelikA.BusijaL.O'ConnorD.PearceC.MazzaD.et al. (2023). Prevalence and characteristics of musculoskeletal complaints in primary care: an analysis from the population level and analysis reporting (POLAR) database. BMC Prim. Care24:40. doi: 10.1186/s12875-023-01976-z,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  Hofmans-OkkesI.LambertsH. (1996). The international classification of primary care (ICPC): new applications in research and computer-based patient records in family practice. Fam. Pract.13, 294–302. doi: 10.1093/fampra/13.3.294,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  IannoneR. C. J.SchloerkeB.HughesE.LauerA.SeoJ. (2024). gt: Easily create presentation-ready display tables R package version 1.3.0.9000. Available online at: https://github.com/rstudio/gt

  • Google Scholar
  • View reference in article

• 21

  JainA.JosephS.JamesJ.JamesT. S.KumarK.RazaK.et al. (2023). Delay in diagnosis of rheumatoid arthritis: reasons and trends over a decade. Rheumatol. Int.43, 503–508. doi: 10.1007/s00296-022-05187-0,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  KeidingN.LouisT. A. (2016). Perils and potentials of self-selected entry to epidemiological studies and surveys. J. R. Stat. Soc. A179, 319–376. doi: 10.1111/rssa.12136

  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  KnevelR.KnitzaJ.HensvoldA.CirciumaruA.BruceT.EvansS.et al. (2022). Rheumatic?—a digital diagnostic decision support tool for individuals suspecting rheumatic diseases: a Multicenter pilot validation study. Front. Med.9:774945. doi: 10.3389/fmed.2022.774945,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  KnitzaJ.MohnJ.BergmannC.KampylafkaE.HagenM.BohrD.et al. (2021). Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res. Ther.23:112. doi: 10.1186/s13075-021-02498-8,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  KopkaM.ScatturinL.NapieralaH.FurstenauD.FeufelM. A.BalzerF.et al. (2023). Characteristics of users and nonusers of symptom checkers in Germany: cross-sectional survey study. J. Med. Internet Res.25:e46231. doi: 10.2196/46231,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  LaskoT. A.StroblE. V.SteadW. W. (2024). Why do probabilistic clinical models fail to transport between sites?npj Digit Med.7:53. doi: 10.1038/s41746-024-01037-4,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 27

  LundbergK.QinL.AulinC.van SpilW. E.MauritsM. P.KnevelR. (2023). Population-based user-perceived experience of Rheumatic?: a novel digital symptom-checker in rheumatology. RMD Open9:e002974. doi: 10.1136/rmdopen-2022-002974,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  MorseK. E.OstbergN. P.JonesV. G.ChanA. S. (2020). Use characteristics and triage acuity of a digital symptom checker in a large integrated health system: population-based descriptive study. J. Med. Internet Res.22:e20549. doi: 10.2196/20549,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  MunafoM. R.TillingK.TaylorA. E.EvansD. M.DaveyS. G. (2018). Collider scope: when selection bias can substantially influence observed associations. Int. J. Epidemiol.47, 226–235. doi: 10.1093/ije/dyx206,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  OkparaC.AdachiJ.PapaioannouA.IoannidisG.ThabaneL. (2023). Exploring participant attrition in a longitudinal follow-up of older adults: the Global Longitudinal Study of Osteoporosis in Women (GLOW) Hamilton cohort. BMJ Open13:e066594. doi: 10.1136/bmjopen-2022-066594,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  PetersenG. L.JorgensenT. S. H.MathisenJ.OslerM.MortensenE. L.MolboD.et al. (2024). Inverse probability weighting for self-selection bias correction in the investigation of social inequality in mortality. Int. J. Epidemiol.53:dyae097. doi: 10.1093/ije/dyae097,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  R Core Team (2023). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.

  • Google Scholar
  • View reference in article

• 33

  RaciborskiF.KlakA.KwiatkowskaB.BatkoB.Sochocka-BykowskaM.Zon-GiebelA.et al. (2017). Diagnostic delays in rheumatic diseases with associated arthritis. Reumatologia55:29056771, 169–176. doi: 10.5114/reum.2017.69777,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  SjobergD. D.CurryM.LaveryJ. A.LarmarangeJ. (2021). Reproducible summary tables with the gtsummary package. R J.31, 570–580. doi: 10.32614/RJ-2021-053

  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  SperrinM. R.RichardD.CollinsG. S.MartinG. P. (2022). Targeted validation: validating clinical prediction models in their intended population and setting. Diagn. Progn. Res.6:24. doi: 10.1186/s41512-022-00136-8

  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  SterneJ. A.WhiteI. R.CarlinJ. B.SprattM.RoystonP.KenwardM. G.et al. (2009). Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ338:b2393. doi: 10.1136/bmj.b2393,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  van BuurenS. (2018). Flexible Imputation of Missing Data. 2nd Edn Boca Raton., MA: CRC Press.

  • Google Scholar
  • View reference in article

• 38

  WallaceW.ChanC.ChidambaramS.HannaL.IqbalF. M.AcharyaA.et al. (2022). The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. npj Digit Med.5:118. doi: 10.1038/s41746-022-00667-w,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  WeirC. B.JanA. (2024). “BMI classification percentile and cut off points,” in StatPearls, (Treasure Island (FL): StatPearls Publishing).

  • Google Scholar
  • View reference in article

• 40

  WickhamH. (2016). ggplot2: Elegant Graphics for Data Analysis. New York: Springer.

  • Google Scholar
  • View reference in article

• 41

  WinnA. N.SomaiM.FergestromN.CrottyB. H. (2019). Association of use of online symptom checkers with patients' plans for seeking care. JAMA Netw. Open2:e1918561. doi: 10.1001/jamanetworkopen.2019.18561,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  WolffR. F.MoonsK. G. M.RileyR. D.WhitingP. F.WestwoodM.CollinsG. S.et al. (2019). PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann. Intern. Med.170, 51–58. doi: 10.7326/m18-1376,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 43

  WuM. J.ZhaoK. L. Y.Fils-AimeF. (2022). Response rates of online surveys in published research: a meta-analysis. Comput. Hum. Behav. Rep.7:100206. doi: 10.1016/j.chbr.2022.100206

  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  ZhangX.HoltJ. B.LuH.WheatonA. G.FordE. S.GreenlundK. J.et al. (2014). Multilevel regression and poststratification for small-area estimation of population health outcomes: a case study of chronic obstructive pulmonary disease prevalence using the behavioral risk factor surveillance system. Am. J. Epidemiol.179, 1025–1033. doi: 10.1093/aje/kwu018,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article