Recommendations for prediction models in clinical practice guidelines for cardiovascular diseases are over-optimistic: a global survey utilizing a systematic literature search

Abstract

Background:

This study aimed to synthesize the recommendations for prediction models in cardiovascular clinical practice guidelines (CPGs) and assess the methodological quality of the relevant primary modeling studies.

Methods:

We performed a systematic literature search of all available cardiovascular CPGs published between 2018 and 2023 that presented specific recommendations (whether in support or non-support) for at least one multivariable clinical prediction model. For the guideline-recommended models, the assessment of the methodological quality of their primary modeling studies was conducted using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

Results:

In total, 46 qualified cardiovascular CPGs were included, with 69 prediction models and 80 specific recommendations. Of the 80 specific recommendations, 74 supported 57 models (53 were fully recommended and 4 were conditionally recommended) in cardiovascular practice with moderate to strong strength. Most of the guideline-recommended models were focused on predicting prognosis outcomes (53/57, 93%) in primary and tertiary prevention, focusing primarily on long-term risk stratification and prognosis management. A total of 10 conditions and 7 types of target population were involved in the 57 models, while heart failure (14/57, 25%) and a general population with or without cardiovascular risk factor(s) (12/57, 21%) received the most attention from the guidelines. The assessment of the methodological quality of 57 primary studies on the development of the guideline-recommended models revealed that only 40% of the modeling studies had a low risk of bias (ROB). The causes of high ROB were mainly in the analysis and participant domains.

Conclusions:

Global cardiovascular CPGs presented an unduly positive appraisal of the existing prediction models in terms of ROB, leading to stronger recommendations than were warranted. Future cardiovascular practice may benefit from well-established clinical prediction models with better methodological quality and extensive external validation.

1 Introduction

Cardiovascular diseases (CVDs) are major causes of mortality and a leading contributor to disability globally with the burden of disease continuing its decades-long rise in almost all countries (1, 2). In the United States of America, the estimated cost due to CVD for healthcare services, medications, and lost productivity was over $200 billion annually (3). This growing burden of CVD can be broadly attributed to the suboptimal implementation of prevention strategies and poor management of cardiovascular risk factors (3, 4). Hence, the assessment of CVD risk in the general population, risk stratification in suspected patients, and prevention of recurrence in patients suffering from CVD represent an opportunity for major public health gains (1, 5).

Multivariable clinical prediction models (also known as risk prediction/assessment tools or prediction rules) are usually developed based on person-level information, aiming to estimate the pretest probability or risk of an individual having (diagnosis) or developing (prognosis) a particular outcome (6–8). As they play an increasingly vital role in the screening, diagnosis, and prognosis management of conditions, prediction models have gradually received increased attention from healthcare service providers and creators of clinical practice guidelines (CPGs), especially in the cardiovascular domain (9, 10). Beyond their clinical utility, their economic value has also been demonstrated in a considerable number of pharmacoeconomics analyses (11). Currently, quite a few available prediction models have been recommended for clinical practice in cardiovascular CPGs, e.g., there are 14 models recommended for the primary and tertiary prevention of heart failure (HF) in the 2022 American Heart Association (AHA)/American College of Cardiology (ACC)/Heart Failure Society of America (HFSA) Guideline for the Management of HF (12). Furthermore, other well-known models, e.g., the Framingham Risk Score (FRS), Systematic Coronary Risk Estimation 2 (SCORE2), Pooled Cohort Equations (PCE), and the QRISK3 tool, are widely recommended for the assessment of 10-year CVD risk in European and American populations with or without risk factor(s) (13–16).

Despite the promising application prospects of the prediction models, the methodological quality of the primary studies on the development of the models has been roundly criticized in recent years. In the different medical domains, almost all the primary modeling studies are at high risk of bias (ROB), whether the models were developed based on traditional regression methods (e.g., Logistic regression) or emerging artificial intelligence (AI) technology (e.g., machine learning algorithms) (17–20). The same is true without discrepancy in the cardiovascular domain (21–23). Given the enormous impact of CPGs on clinical practice (24), an assessment of the methodological quality of primary modeling studies related to guideline-recommended models appears to be necessary.

Prior reports mainly reviewed and summarized several prediction models for cardiovascular risk classification in primary prevention, with limitations in their application to other clinical settings and target populations/conditions (10, 15, 25). To date, the recommendations for prediction models in cardiovascular practice guidelines have not been adequately synthesized, nor has methodological quality assessment been systematically conducted for primary modeling studies. Thus, we performed a systematic literature search aiming to (1) identify and provide an overview of the recommendations for prediction models in cardiovascular CPGs, (2) summarize the characteristics of the involved models in clinical practice, and (3) assess the ROB of the primary studies on the development of guideline-recommended models.

2 Materials and methods

This study was reported following the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) (Supplementary Table 1) (26).

2.1 Identification of eligible guidelines

We focused on searching the CPGs that recommend any CVD prediction models for clinical practice. Relevant guidelines published in English were searched in the following electronic databases on 28 December 2023: Medline (PubMed), EMBASE, and Web of Science. Considering the impact of the timeliness of the CPGs on their clinical utility, the publication date was restricted to after 2018. The search strategy was developed by a combination of Medical Subject Heading terms as well as text words related to the concepts of “cardiovascular diseases” and “practice guideline” (Supplementary Table 2). Furthermore, the following websites of governments or organizations were used for the supplementary search: World Health Organization (WHO, https://www.who.int/zh/publications), Guidelines International Network (GIN, https://g-i-n.net/), National Institute for Health and Clinical Excellence (NICE, https://www.nice.org.uk/), Scottish Intercollegiate Guidelines Network (SIGN, https://www.sign.ac.uk/), and Canadian Medical Association: Clinical Practice Guideline (CMACPG, https://www.cma.ca/). In addition, we also examined the references of the documents obtained through the aforementioned search strategies to further discover potentially eligible CPGs.

Eligible publications needed to provide a specific recommendation (whether in support or non-support) for at least one multivariable clinical prediction model that was defined as any combination of two or more predictors (variables, features) for estimating the probability or risk of an individual having (diagnosis) or developing (prognosis) a particular outcome in the cardiovascular domain. Guidelines were excluded if they were ongoing, publicly unpublished, or unavailable for full text. In addition, the older versions of guidelines developed by the same academic organization were also excluded in the screening stage. The procedure for identifying eligible guidelines was conducted according to the PRISMA flow diagram (27).

2.2 Screening process

The literature search was conducted by one reviewer (C-yJ) after consulting an experienced librarian who also assisted in developing the search strategy. All the search results were imported into Endnote V20.6 to manage the records and screen the eligible CPGs. Two reviewers (C-yJ and LZ) independently screened the titles and abstracts of all the available records. A list of potentially eligible publications was generated for the retrieval of the full texts. Thereafter, the same reviewers independently continued to assess the eligibility of these publications and determined the final included guidelines. The reasons for the exclusion of records were required to be noted. Throughout the process, any disagreement between the two reviewers was resolved by discussion until a consensus was reached or by consulting with an experienced third reviewer (XL).

While screening the guidelines, the primary studies on the development of the guideline-recommended prediction models were further identified through an examination of the references, and the full texts of the relevant studies were obtained by manual retrieval to gather the model details.

2.3 Data collection and synthesis

Based on our study objectives, we developed a data extraction form for the standard data collection procedure. The extraction form contained four sections: (1) general information of the guideline, including name, year developed or updated, country, location, and publication organization(s); (2) prediction model recommendations, including recommended model, overview of recommendations, classification of recommendations, level or quality of evidence, and criteria for recommendation; (3) characteristics of prediction model, including presentation, setting, predicted outcome(s), outcome type and target population or condition (Supplementary Table 3). A critical appraisal was conducted on all the recommendations in the CPGs and a narrative synthesis was performed to identify themes based on the extracted data.

2.4 Risk of bias assessment

For the guideline-recommended models, we used the Prediction model Risk Of Bias ASsessment Tool (PROBAST) to assess the ROB of the primary modeling studies (28, 29). PROBAST contains 20 signaling questions developed based on the Delphi method, which can assess the ROB according to four domains (participants, predictors, outcome, and analysis). The overall ROB was judged to be low, high, or unclear based on an assessment of all the signaling questions.

3 Results

A total of 15,296 potentially relevant records were identified from databases, 13,483 of which were screened based on titles and abstracts after removing 1,813 duplicates. We further identified 260 publications for final eligibility by assessing their full text. Of these, 37 guidelines (3, 12–14, 30–62) fulfill our predefined inclusion criteria while the other 223 publications were excluded for specific reasons (Supplementary Table 4). In addition, we included nine guidelines (16, 63–70) from 61 publications that were identified through the supplementary search. Figure 1 shows the PRISMA flow diagram for the entire CPG selection process.

Figure 1

3.1 Characteristics of the included CPGs

In the past 6 years, 46 cardiovascular CPGs with explicit recommendations for clinical prediction models were developed or updated globally. Except for one guideline that was developed by an international academic working group, the other 45 guidelines were published in Europe (19/45, 42%), North America (11/45, 24%), Asia (10/45, 22%), South America (3/45, 7%), and Oceania (2/45, 4%), encompassing 12 countries in total (Supplementary Table 5).

All of these CPGs were evidence-based guidelines developed based on six categories of criteria for forming recommendations. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach and a modified GRADE approach (17/46, 37%) were most widely adopted for assessing the quality of evidence and grading recommendations, with the European Society of Cardiology (ESC) approach (15/46, 33%) and the American College of Cardiology Foundation (ACCF)/AHA approach (11/46, 24%) following closely. The American Physical Therapy Association (APTH) approach, European Society for Medical Oncology (ESMO) approach, and Traffic light signaling system-based classification were used only once (1/46, 2%), respectively. The details of all the criteria are summarized in Supplementary Table 6.

3.2 Overview of recommendations on prediction models

A total of 80 eligible recommendations and 69 relevant prediction models were identified and extracted from the included CPGs. An overview of the recommendations is given in Supplementary Table 7. Of the 80 recommendations, 74 (93%) supported the application of an appropriate prediction model in clinical practice, while the other 6 (7%) explicitly put forward a disapproving opinion of the specific model. Figure 2 synthesizes and visualizes the recommendation classifications of the 69 models based on different criteria. Thus, the involved models could be divided into three categories: (1) 53 fully recommended models (only mentioned in the supportive recommendations); (2) 4 conditionally recommended models (mentioned in both the supportive and non-supportive recommendations); and (3) 12 non-recommended models (only mentioned in the non-supportive recommendations). More importantly, for recommendations regarding the use of prognostic models, 10 CPGs recommended 16 prognostic models for further clinical decision-making or treatment selection besides risk assessment.

Figure 2

Due to different evidence-grading criteria, there was wide variation in the level/quality of evidence for supporting the application of a model. However, almost all guideline-recommended (both fully recommended and conditionally recommended) models had a moderate to strong strength of recommendation, of which the conditionally recommended models were discouraged for populations with specific known risk factors (e.g., diabetes mellitus and familial hypercholesterolemia). Meanwhile, based on the low to moderate levels of evidence, the guidelines discouraged the use of all of the non-recommended models in unsuited target populations (e.g., pregnant women with suspected deep vein thrombosis) or conditions (e.g., in-hospital cardiac arrest). Figure 3 summarizes the recommended and non-recommended target populations/conditions in clinical practice of the three categories of models.

Figure 3

3.3 Attributes of the guideline-recommended models in cardiovascular practice

Table 1 shows the attributes of the guideline-recommended models in cardiovascular practice. Almost all the recommended models were prognostic models (53/57, 93%) and were focused on primary (16/57, 28%) and tertiary prevention (37/57, 65%), while the remaining models (4/57, 7%) were diagnostic models and were focused on secondary prevention.

A total of 10 conditions and seven types of target population were involved in the 57 models, where HF (14/57, 25%) and a general population with or without cardiovascular risk factor(s) (12/57, 21%) had received the most attention from the guidelines. Other target conditions included valvular heart disease (5/57, 9%), atrial fibrillation (4/57, 7%), cardiomyopathy (3/57, 5%), chronic coronary disease (3/57, 5%), acute coronary syndrome (2/57, 4%), long QT syndrome (2/57, 4%), hypertension (2/57, 4%), multivessel coronary artery disease (1/57, 2%), and chronic total occlusion lesions (1/57, 2%). Suspected venous thromboembolism patients (3/57, 5%); patients who had undergone coronary artery bypass grafting (CABG) (3/57, 5%) or non-cardiac surgery (1/57, 2%); or patients with cancer (5/57, 9%), reduced mobility (2/57, 4%), or an inherited clotting disorder (1/57, 2%) were also target populations.

The predicted outcome of the recommended models was mostly concerned with the risk of mortality (23/57, 40%), followed by the risk of incident fatal or non-fatal CVD (8/57, 14%) and the risk of incident venous thromboembolism (5/57, 9%). Others mainly focused on predicting the risk of incident cardiovascular complications and evaluating the feasibility/necessity of the interventions under consideration. A point scoring system (29/57, 51%) was the most common presentation format of the prediction models for use in clinical settings, followed by a website calculator (14/57, 25%). A nomogram (2/57, 4%) or graphical score chart (1/57, 2%) was relatively rare. In addition, the presentation of eight (14%) models combined two of the above four formats, while the remaining three (5%) models were unclear.

3.4 Risk of bias assessment of the primary modeling studies

Through the references provided in the guidelines and from manual retrieval, we further identified 57 primary studies on the development of the guideline-recommended models (Supplementary Table 8). Of the 57 primary studies on the guideline-recommended prediction models, 23 (40%) studies were rated as having a low ROB, 3 (4%) as having an unclear ROB, and the remaining 31 (56%) studies had a high ROB after PROBAST analysis (Figures 4, 5). The details of the ROB assessment of 20 signaling questions are shown in Supplementary Table 9.

Figure 4

Figure 5

The causes of high ROB were mainly in the analysis domain as 23 (40%) studies inappropriately evaluated the calibration of the relevant models (e.g., ignored the calibration or evaluated the calibration using the Hosmer–Lemeshow test only) and 17 (30%) studies did not account for the fitting of their models (e.g., a lack of internal validation or the dataset was randomly split into only derivation and validation sets). Other reasons for a high ROB included incorrect handling in continuous/categorical predictors and improper variable selection (i.e., based only on univariable analysis). A non-prospective data source was the major cause of high ROB in the participant domain, resulting in six (11%) studies being rated as having a high ROB. This also led to an unclear ROB in the blinding of predictor or outcome assessment with regard to the reporting of relevant missing content in such studies. Beyond that, there was no information about handling missing data and consistency in reporting the predictor weighting in 11 (19%) and 8 (14%) studies, respectively, leading to an unclear ROB for the relevant signaling questions.

4 Discussion

In this global survey of 46 qualified cardiovascular CPGs involving 80 recommendations for 69 relevant prediction models from 2018 to 2023, we found that 74 recommendations supported 57 models in cardiovascular practice with moderate to strong strength. Most of the guideline-recommended models were focused on predicting prognosis outcomes in primary and tertiary prevention, mainly concentrating on long-term risk stratification and prognosis management. Moreover, a further assessment of the methodological quality of the primary studies on the development of the guideline-recommended models revealed that only 40% of the modeling studies had a low risk of bias.

The findings of this study indicate that prediction models have been generally recommended for cardiovascular practice in global CPGs, although their primary application was concentrated in developed countries in Europe and the Americas, geographically. Given the abundance of developed prediction models and the rapidly increasing number of similar modeling studies, how to evaluate and recommend models with genuine clinical utility still remains a major dilemma for guideline creators and methodologists. Indeed, a phenomenon—the overall level of evidence for the recommendations of moderate to strong strength was moderate or even lower—emerged in this study. It is common for a recommendation of strong strength to only be supported by consensus/expert opinion-level evidence (41, 46, 50, 52). This may be attributed to the stability (via internal validation) and generalizability (via external validation) of the validated models despite the paucity of high-level evidence (71).

With regards to evidence-based CPGs, evidence is the cornerstone of formulating recommendations, and high-level evidence is universally defined as results from high-quality randomized controlled trials (RCTs) or meta-analyses (24). However, when conducting RCTs or meta-analyses for prediction models that are treated as specific interventions, additional challenges need to be considered more carefully. First, the applicable study design for investigating the impact of models on cardiovascular practice is a cluster randomized trial (CRT) (72, 73). Compared to the conventional RCT design, a CRT requires a considerably larger sample size for a statistically significant effect size and this sometimes faces several particular medical ethical challenges, such as the plausibility of informed consent waivers and the possible neglect of vulnerable individuals (74, 75). Second, the availability and heterogeneity of the original data, mainly resulting from reporting and selection biases, may hinder the synthesis of evidence in meta-analyses (29). Third, there are no established indicators or criteria that can be used to make a fully objective distinction between the superiority or inferiority of a model based on the results of meta-analyses, despite the use of the most common methods of discrimination such as the area under the receiver operating characteristic curve (AUC). But in view of poor methodological quality and ubiquitous high ROB, almost all the systematic reviews with or without a meta-analysis have adopted a deliberative attitude toward the application of prediction models in cardiovascular practice, regardless of their discrimination (76, 77). Addressing the aforementioned challenges will be critical for obtaining more high-level evidence in the future. More importantly, in terms of the prediction models themselves, how and when to update the existing models have been problems in an increasingly complex real-world medical setting, especially for certain specific cardiovascular diseases where treatment modalities and drugs are rapidly evolving. Thus, for static models (also called time-independent models), it is crucial to address calibration drift to maintain effective and efficient performance when the treatment is part of the predictors (78). Correspondingly, due to the lag inherent to the CPGs themselves, it will be wise and prudent for recommendations to take into account information regarding the updating (including frequency, aim, and methods of updating) of the recommended prediction model.

We also observed that the same models were recommended for heterogeneous target populations or conditions in different cardiovascular CPGs. For example, in addition to the general population (14), cancer patients treated with endocrine therapies were another target population recommended for the application of SCORE2 in the 2022 ESC Guidelines on cardio-oncology (41). Similar examples included the FRS (for the general population or patients with hypertension) (13, 54), Khorana risk score (for patients with cancer or inherited clotting disorder) (34, 42), and Society of Thoracic Surgeons risk score (for patients undergoing CABG or with valvular heart disease) (40, 46). Such excellent generalizability via external validation indicates the great application potential of a single well-established prediction model, which may contribute to research resource savings by reducing superfluous modeling studies (79).

To the best of our knowledge, this study was the first to systematically synthesize the recommendations for prediction models in cardiovascular CPGs published in the last 5 years while also conducting a methodological quality assessment of the primary studies on the development of the guideline-recommended models. The results may be of referential value to both clinicians and guideline methodologists. For future studies, substantial efforts are still required to improve the methodological quality of clinical prediction models, as well as their extensive external validation.

The limitations of this study merit further consideration. First, in consideration of the particularity of CPGs in terms of timeliness, we only included cardiovascular CPGs from 2018 to 2023, possibly omitting qualified guidelines that were published before 2018 and had an update period longer than 6 years. As a result, the few early-developed prediction models still in use may not have been assessed in this study. Second, in the case of the primary studies on the development of the guideline-recommended models identified through the references provided in the guidelines and from manual retrieval, there was no guaranteed consistency in model version in the 57 primary studies used for the assessment of methodological quality. This is due to individual models being updated over time, with multiple versions of primary modeling studies. Third, we did not further explore the methodological differences between the recommended and non-recommended models as these were not a major factor that supported or discouraged their use to provide clinical decision-making support. Finally, given the heterogeneous criteria for forming recommendations, we did not analyze the relevance between the strength of recommendation and the overall ROB of the prediction models. Thus, it is difficult to determine what role the ROB plays in the process of recommending models for cardiovascular practice. In any case, a high ROB implies that the performance of these models in new samples will probably be worse than that in the modeling samples, which could lead to improper clinical decisions and thus compromise the benefit for patients in healthcare settings (18, 80).

5 Conclusion

Global cardiovascular CPGs presented an unduly positive appraisal of the existing prediction models in terms of ROB, leading to stronger recommendations than were warranted. Future cardiovascular practice may benefit from well-established clinical prediction models with better methodological quality and extensive external validation.

References

• 1.

  AbbafatiCAbbasKMAbbasiMAbbasifardMAbbasi-KangevariMAbbastabarHet alGlobal burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet. (2020) 396(10258):1204–22. 10.1016/S0140-6736(20)30925-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  RothGAMensahGAJohnsonCOAddoloratoGAmmiratiEBaddourLMet alGlobal burden of cardiovascular diseases and risk factors, 1990–2019. J Am Coll Cardiol. (2020) 76(25):2982–3021. 10.1016/j.jacc.2020.11.010

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  ArnettDKBlumenthalRSAlbertMABurokerABGoldbergerZDHahnEJet al2019 ACC/AHA guideline on the primary prevention of cardiovascular disease. J Am Coll Cardiol. (2019) 74(10):e177–232. 10.1016/j.jacc.2019.03.010

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  MurrayCJLAravkinAYZhengPAbbafatiCAbbasKMAbbasi-KangevariMet alGlobal burden of 87 risk factors in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet. (2020) 396(10258):1223–49. 10.1016/S0140-6736(20)30752-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  TsaoCWAdayAWAlmarzooqZIAndersonCAMAroraPAveryCLet alHeart disease and stroke statistics—2023 update: a report from the American Heart Association. Circulation. (2023) 147(8):e93–621. 10.1161/cir.0000000000001123

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  BouwmeesterWZuithoffNPMallettSGeerlingsMIVergouweYSteyerbergEWet alReporting and methods in clinical prediction research: a systematic review. PLoS Med. (2012) 9(5):1–12. 10.1371/journal.pmed.1001221

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  SteyerbergEWMoonsKGvan der WindtDAHaydenJAPerelPSchroterSet alPrognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. (2013) 10(2):e1001381. 10.1371/journal.pmed.1001381

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  MoonsKGRoystonPVergouweYGrobbeeDEAltmanDG. Prognosis and prognostic research: what, why, and how?Br Med J. (2009) 338:b375. 10.1136/bmj.b375

  • CrossRef
  • Google Scholar

• 9.

  JensenJK. Risk predictionCirculation. (2016) 134(19):1441–3. 10.1161/CIRCULATIONAHA.116.024941

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  RosselloXDorresteijnJAJanssenALambrinouEScherrenbergMBonnefoy-CudrazEet alRisk prediction tools in cardiovascular disease prevention: a report from the ESC prevention of CVD programme led by the European Association of Preventive Cardiology (EAPC) in collaboration with the Acute Cardiovascular Care Association (ACCA) and the Association of Cardiovascular Nursing and Allied Professions (ACNAP). Eur J Prev Cardiol. (2019) 26(14):1534–44. 10.1177/2047487319846715

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  StevanovicJPostmaMJPechlivanoglouP. A systematic review on the application of cardiovascular risk prediction models in pharmacoeconomics, with a focus on primary prevention. Eur J Prev Cardiol. (2012) 19(2 Suppl):42–53. 10.1177/2047487312448994

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  HeidenreichPABozkurtBAguilarDAllenLAByunJJColvinMMet al2022 AHA/ACC/HFSA guideline for the management of heart failure. J Am Coll Cardiol. (2022) 79(17):e263–421. 10.1016/j.jacc.2021.12.012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  JainRStoneJAAgarwalGAndradeJGBaconSLBajajHSet alCanadian cardiovascular harmonized national guideline endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2022 update. CMAJ. (2022) 194(43):E1460–80. 10.1503/cmaj.220138

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  VisserenFLJMachFSmuldersYMCarballoDKoskinasKCBäckMet al2021 ESC guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J. (2021) 42(34):3227–337. 10.1093/eurheartj/ehab484

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  GoffDCJrLloyd-JonesDMBennettGCoadySD’AgostinoRBGibbonsRet al2013 ACC/AHA guideline on the assessment of cardiovascular risk. Circulation. (2014) 129(25 Suppl 2):S49–73. 10.1161/01.cir.0000437741.48606.98

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  National Institute for Health and Clinical Excellence. Cardiovascular Disease: Risk Assessment and Reduction, Including Lipid Modification (NG238). United Kingdom (2023). Available online at:https://www.nice.org.uk/guidance/cg181(accessed December 28, 2023).

  • Google Scholar

• 17.

  Andaur NavarroCLDamenJAATakadaTNijmanSWJDhimanPMaJet alRisk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. Br Med J. (2021) 375:n2281. 10.1136/bmj.n2281

  • CrossRef
  • Google Scholar

• 18.

  WynantsLVan CalsterBCollinsGSRileyRDHeinzeGSchuitEet alPrediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. Br Med J. (2020) 369:m1328. 10.1136/bmj.m1328

  • CrossRef
  • Google Scholar

• 19.

  TanJMaCZhuCWangYZouXLiHet alPrediction models for depression risk among older adults: systematic review and critical appraisal. Ageing Res Rev. (2023) 83:101803. 10.1016/j.arr.2022.101803

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  KareemiHVaillancourtCRosenbergHFournierKYadavK. Machine learning versus usual care for diagnostic and prognostic prediction in the emergency department: a systematic review. Acad Emerg Med. (2021) 28(2):184–96. 10.1111/acem.14190

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  de WinterMAvan EsNBüllerHRVisserenFLJNijkeuterM. Prediction models for recurrence and bleeding in patients with venous thromboembolism: a systematic review and critical appraisal. Thromb Res. (2021) 199:85–96. 10.1016/j.thromres.2020.12.031

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  CaiRWuXLiCChaoJ. Prediction models for cardiovascular disease risk in the hypertensive population: a systematic review. J Hypertens. (2020) 38(9):1632–9. 10.1097/HJH.0000000000002442

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  KeeOTHarunHMustafaNAbdul MuradNAChinSFJaafarRet alCardiovascular complications in a diabetes prediction model using machine learning: a systematic review. Cardiovasc Diabetol. (2023) 22(1):13. 10.1186/s12933-023-01741-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  Institute of Medicine (US). Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press (2011).

  • Google Scholar

• 25.

  PetrettaMCuocoloA. Prediction models for risk classification in cardiovascular disease. Eur J Nucl Med Mol Imaging. (2012) 39(12):1959–69. 10.1007/s00259-012-2254-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  von ElmEAltmanDGEggerMPocockSJGøtzschePCVandenbrouckeJP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. (2008) 61(4):344–9. 10.1016/j.jclinepi.2007.11.008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  PageMJMcKenzieJEBossuytPMBoutronIHoffmannTCMulrowCDet alThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews. J Clin Epidemiol. (2021) 134:178–89. 10.1016/j.jclinepi.2021.03.001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  WolffRFMoonsKGMRileyRDWhitingPFWestwoodMCollinsGSet alPROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. (2019) 170(1):51–8. 10.7326/M18-1376

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  MoonsKGMWolffRFRileyRDWhitingPFWestwoodMCollinsGSet alPROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. (2019) 170(1):W1–33. 10.7326/M18-1377

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30.

  ViraniSSNewbyLKArnoldSVBittnerVBrewerLCDemeterSHet al2023 AHA/ACC/ACCP/ASPC/NLA/PCNA guideline for the management of patients with chronic coronary disease: a report of the American Heart Association/American College of Cardiology joint committee on clinical practice guidelines. Circulation. (2023) 148(9):e9–119. 10.1161/CIR.0000000000001168

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  UengKCChiangCEChaoTHWuYWLeeWLLiYHet al2023 guidelines of the Taiwan Society of Cardiology on the diagnosis and management of chronic coronary syndrome. Acta Cardiologica Sinica. (2023) 39(1):4–96. 10.6515/ACS.202301_39(1).20221103A

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32.

  TakaseBIkedaTShimizuWAbeHAibaTChinushiMet alJCS/JHRS 2022 guideline on diagnosis and risk assessment of arrhythmia. Circ J. (2024) 88(9):1509–95. 10.1253/circj.CJ-22-0827

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  RajajeeVMuehlschlegelSWartenbergKEAlexanderSABuslKMChouSHYet alGuidelines for neuroprognostication in comatose adult survivors of cardiac arrest. Neurocrit Care. (2023) 38(3):533–63. 10.1007/s12028-023-01688-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  KeyNSKhoranaAAKudererNMBohlkeKLeeAYYArcelusJIet alVenous thromboembolism prophylaxis and treatment in patients with cancer: ASCO guideline update. J Clin Oncol. (2023) 41(16):3063–71. 10.1200/JCO.23.00294

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  HiraokaETanabeKIzutaSKubotaTKohsakaSKozukiAet alJCS 2022 guideline on perioperative cardiovascular assessment and management for non-cardiac surgery. Circ J. (2023) 87(9):1253–337. 10.1253/circj.CJ-22-0609

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36.

  FalangaAAyCDi NisioMGerotziafasGJara-PalomaresLLangerFet alVenous thromboembolism in cancer patients: ESMO clinical practice guideline. Ann Oncol. (2023) 34(5):452–67. 10.1016/j.annonc.2022.12.014

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  ByrneRARosselloXCoughlanJJBarbatoEBerryCChieffoAet al2023 ESC guidelines for the management of acute coronary syndromes. Eur Heart J. (2023) 44(38):3720–826. 10.1093/eurheartj/ehad191

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  ArbeloEProtonotariosAGimenoJRArbustiniEArbeloEBarriales-VillaRet al2023 ESC guidelines for the management of cardiomyopathies. Eur Heart J. (2023) 44(37):3503–626. 10.1093/eurheartj/ehad194

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  ZeppenfeldKTfelt-HansenJDe RivaMWinkelBGBehrERBlomNAet al2022 ESC guidelines for the management of patients with ventricular arrhythmias and the prevention of sudden cardiac death. Eur Heart J. (2022) 43(40):3997–4126. 10.1093/eurheartj/ehac262

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40.

  NakamuraMYakuHAkoJAraiHAsaiTChikamoriTet alJCS/JSCVS 2018 guideline on revascularization of stable coronary artery disease. Circ J. (2022) 86(3):477–588. 10.1253/circj.CJ-20-1282

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41.

  LyonARLópez-FernándeTCouchLSAsteggianoRAznarMCBergler-KleiJet al2022 ESC guidelines on cardio-oncology developed in collaboration with the European Hematology Association (EHA), the European Society for Therapeutic Radiology and Oncology (ESTRO) and the International Cardio-Oncology Society (IC-OS). Eur Heart J. (2022) 43(41):4229–361. 10.1093/eurheartj/ehac244

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42.

  HillegassELukaszewiczKPuthoffM. Role of physical therapists in the management of individuals at risk for or diagnosed with venous thromboembolism: evidence-based clinical practice guideline 2022. Phys Ther. (2022) 102(8):pzac057. 10.1093/ptj/pzac057

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43.

  FargeDFrereCConnorsJMKhoranaAAKakkarAAyCet al2022 international clinical practice guidelines for the treatment and prophylaxis of venous thromboembolism in patients with cancer, including patients with COVID-19. Lancet Oncol. (2022) 23(7):e334–47. 10.1016/S1470-2045(22)00160-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  AlbrickerACLFreireCMVDos SantosSNDe AlcantaraMLSalehMHCantisanoALet alJoint guideline on venous thromboembolism, 2022. Arq Bras Cardiol. (2022) 118(4):797–857. 10.36660/abc.20220213

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  TsutsuiHIdeTItoHKiharaYKinugawaKKinugawaSet alJCS/JHFS 2021 guideline focused update on diagnosis and treatment of acute and chronic heart failure. J Card Fail. (2021) 27(12):1404–44. 10.1016/j.cardfail.2021.04.023

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46.

  OttoCMNishimuraRABonowROCarabelloBAErwinJPGentileFet al2020 ACC/AHA guideline for the management of patients with valvular heart disease: a report of the American College of Cardiology/American Heart Association joint committee on clinical practice guidelines. Circulation. (2021) 143(5):E72–227. 10.1161/CIR.0000000000000923

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 47.

  LawtonJSTamis-HollandJEBangaloreSBatesERBeckieTMBischoffJMet al2021 ACC/AHA/SCAI guideline for coronary artery revascularization: a report of the American College of Cardiology/American Heart Association joint committee on clinical practice guidelines. Circulation. (2022):CIR0000000000001038. 10.1161/CIR.0000000000001038

  • CrossRef
  • Google Scholar

• 48.

  KakkosSKGohelMBaekgaardNBauersachsRBellmunt-MontoyaSBlackSAet alEditor’s choice—European Society for Vascular Surgery (ESVS) 2021 clinical practice guidelines on the management of venous thrombosis. Eur J Vasc Endovasc Surg. (2021) 61(1):9–82. 10.1016/j.ejvs.2020.09.023

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 49.

  HindricksGPotparaTDagresNArbeloEBaxJJBlomström-LundqvistCet al2020 ESC guidelines for the diagnosis and management of atrial fibrillation developed in collaboration with the European association for cardio-thoracic surgery (EACTS). Eur Heart J. (2021) 42(5):373–498. 10.1093/eurheartj/ehaa612

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 50.

  MachFBaigentCCatapanoALKoskinasKCCasulaMBadimonLet al2019 ESC/EAS guidelines for the management of dyslipidaemias: lipid modification to reduce cardiovascular risk. Eur Heart J. (2020) 41(1):111–88. 10.1093/eurheartj/ehz455

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 51.

  LiaoPDChenKJGeJBZhangMZ. Clinical practice guideline of integrative Chinese and western medicine for acute myocardial infarction. Chin J Integr Med. (2020) 26(7):539–51. 10.1007/s11655-019-3154-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52.

  KnuutiJWijnsWSarasteACapodannoDBarbatoEFunck-BrentanoCet al2019 ESC guidelines for the diagnosis and management of chronic coronary syndromes. Eur Heart J. (2020) 41(3):407–77. 10.1093/eurheartj/ehz425

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 53.

  HajjarLADa CostaIBSSLopesMACQHoffPMGDizMDPEFonsecaSMRet alDiretriz Brasileira de cardio-oncologia—2020. Arq Bras Cardiol. (2020) 115(5):1006–43. 10.36660/abc.20201006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 54.

  TykarskiAFilipiakKJJanuszewiczALitwinMNarkiewiczKPrejbiszAet al2019 guidelines for the management of hypertension—part 1–7. Arterial Hypertension (Poland). (2019) 23(2):41–87. 10.5603/AH.a2019.0008

  • CrossRef
  • Google Scholar

• 55.

  TranHAGibbsHMerrimanECurnowJLYoungLBennettAet alNew guidelines from the thrombosis and haemostasis society of Australia and New Zealand for the diagnosis and management of venous thromboembolism. Med J Aust. (2019) 210(5):227–35. 10.5694/mja2.50004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 56.

  Sousa-UvaMNeumannFJAhlssonAAlfonsoFBanningAPBenedettoUet al2018 ESC/EACTS guidelines on myocardial revascularization. Eur J Cardiothorac Surg. (2019) 55(1):4–90. 10.1093/ejcts/ezy289

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 57.

  KimuraKKimuraTIshiharaMNakagawaYNakaoKMiyauchiKet alJCS 2018 guideline on diagnosis and treatment of acute coronary syndrome. Circ J. (2019) 83(5):1085–196. 10.1253/circj.CJ-19-0133

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58.

  Feitosa-FilhoGSPeixotoJMPinheiroJESNetoAAde AlbuquerqueALTCattaniÁCet alUpdated geriatric cardiology guidelines of the Brazilian society of cardiology—2019. Arq Bras Cardiol. (2019) 112(5):649–705. 10.5935/abc.20190086

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 59.

  ChantraratTKunjara Na AyudhyaRPhrominthikulAAriyachaipanichAKrittayaphongR. Heart failure council of Thailand (HFCT) 2019 heart failure guideline: atrial fibrillation in heart failure guidelines. J Med Assoc Thail. (2019) 102(4):513–7.

  • Google Scholar

• 60.

  LiYHWangYCWangYCLiuJCLeeCHChenCCet al2018 guidelines of the Taiwan Society of Cardiology, Taiwan Society of Emergency Medicine and Taiwan Society of Cardiovascular Interventions for the management of non ST-segment elevation acute coronary syndrome. J Formos Med Assoc. (2018) 117(9):766–90. 10.1016/j.jfma.2018.06.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 61.

  JoungBLeeJMLeeKHKimTHChoiEKLimWHet al2018 Korean guideline of atrial fibrillation management. Korean Circ J. (2018) 48(12):1033–80. 10.4070/kcj.2018.0339

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 62.

  BriegerDAmerenaJAttiaJBajorekBChanKHConnellCet alNational Heart Foundation of Australia and the Cardiac Society of Australia and New Zealand: Australian clinical guidelines for the diagnosis and management of atrial fibrillation 2018. Heart Lung Circ. (2018) 27(10):1209–66. 10.1016/j.hlc.2018.06.1043

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 63.

  National Institute for Health and Clinical Excellence. Familial Fypercholesterolaemia: Identification and Management (CG71). United Kingdom. (2019). Available online at:https://www.nice.org.uk/guidance/cg71(accessed December 28, 2023).

  • Google Scholar

• 64.

  National Institute for Health and Clinical Excellence. Abdominal Aortic Aneurysm: Diagnosis and Management (NG156). United Kingdom. (2020). Available online at:https://www.nice.org.uk/guidance/ng156(accessed December 28, 2023).

  • Google Scholar

• 65.

  National Institute for Health and Clinical Excellence. Venous Thromboembolic Diseases: Diagnosis, Management and Thrombophilia Testing (NG158). United Kingdom. (2023). Available online at:https://www.nice.org.uk/guidance/ng158(accessed December 28, 2023).

  • Google Scholar

• 66.

  National Institute for Health and Clinical Excellence. Acute Coronary Syndromes (NG185). (2020). United Kingdom. Available online at:https://www.nice.org.uk/guidance/ng185(accessed December 28, 2023).

  • Google Scholar

• 67.

  National Institute for Health and Clinical Excellence. Atrial Fibrillation: Diagnosis and Management (NG196). United Kingdom. (2021). Available online at:https://www.nice.org.uk/guidance/ng196 (accessed December 28, 2023).

  • Google Scholar

• 68.

  American Diabetes Association, European Society of Endocrinology. Primary Prevention of ASCVD and T2DM in Patients at Metabolic Risk: An Endocrine Society* Clinical Practice Guideline. United States. (2019). Available online at:https://guidelines.ebmportal.com/primary-prevention-ascvd-and-t2dm-patients-metabolic-risk-endocrine-societyclinical-practice(accessed December 28, 2023).

  • Google Scholar

• 69.

  European Society of Endocrinology. Lipid Management in Patients with Endocrine Disorders: An Endocrine Society Clinical Practice Guideline. United States. (2020). Available online at:https://guidelines.ebmportal.com/lipid-management-patients-endocrinedisorders-endocrine-society-clinical-practice-guideline(accessed December 28, 2023).

  • Google Scholar

• 70.

  Guidelines and Protocols and Advisory Committee. Cardiovascular Disease—Primary Prevention. (2021). Canada. Available online at:https://www2.gov.bc.ca/assets/gov/health/practitioner-pro/bc-guidelines/gpac_cardiovasculardisease_full_guideline_dec2021_updated_aug2023.pdf(accessed December 28, 2023).

  • Google Scholar

• 71.

  SteyerbergEWVergouweY. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. (2014) 35(29):1925–31. 10.1093/eurheartj/ehu207

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 72.

  VagholkarSZwarNJayasingheUWDenney-WilsonEPatelACampbellTet alInfluence of cardiovascular absolute risk assessment on prescribing of antihypertensive and lipid-lowering medications: a cluster randomized controlled trial. Am Heart J. (2014) 167(1):28–35. 10.1016/j.ahj.2013.10.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 73.

  SvendsenKJacobsDRMorch-ReiersenLTGarstadKWHenriksenHBTelle-HansenVHet alEvaluating the use of the heart age tool in community pharmacies: a 4-week cluster-randomized controlled trial. Eur J Public Health. (2020) 30(6):1139–45. 10.1093/eurpub/ckaa048

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 74.

  McRaeADWeijerCBinikAGrimshawJMBoruchRBrehautJCet alWhen is informed consent required in cluster randomized trials in health research?Trials. (2011) 12:202. 10.1186/1745-6215-12-202

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 75.

  WeijerCGrimshawJMEcclesMPMcRaeADWhiteABrehautJCet alThe Ottawa statement on the ethical design and conduct of cluster randomized trials. PLoS Med. (2012) 9(11):e1001346. 10.1371/journal.pmed.1001346

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 76.

  SoaresCKwokMBoucherKAHajiMEchouffo-TcheuguiJBLongeneckerCTet alPerformance of cardiovascular risk prediction models among people living with HIV: a systematic review and meta-analysis. JAMA Cardiol. (2023) 8(2):139–49. 10.1001/jamacardio.2022.4873

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 77.

  KaboréEGMacdonaldCKaboréADidierRArveuxPMedaNet alRisk prediction models for cardiotoxicity of chemotherapy among patients with breast cancer: a systematic review. JAMA Netw Open. (2023) 6(2):e230569. 10.1001/jamanetworkopen.2023.0569

  • CrossRef
  • Google Scholar

• 78.

  DavisSELaskoTAChenGSiewEDMathenyME. Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc. (2017) 24(6):1052–61. 10.1093/jamia/ocx030

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 79.

  GlasziouPAltmanDGBossuytPBoutronIClarkeMJuliousSet alReducing waste from incomplete or unusable reports of biomedical research. Lancet. (2014) 383(9913):267–76. 10.1016/S0140-6736(13)62228-X

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 80.

  WallaceESmithSMPerera-SalazarRVaucherPMcCowanCCollinsGet alFramework for the impact analysis and implementation of clinical prediction rules (CPRs). BMC Med Inform Decis Mak. (2011) 11:62. 10.1186/1472-6947-11-62

  • Pubmed Abstract
  • CrossRef
  • Google Scholar