Identification of high-risk patients for referral through machine learning assisting the decision making to manage minor ailments in community pharmacies

Background: Data analysis techniques such as machine learning have been used for assisting in triage and the diagnosis of health problems. Nevertheless, it has not been used yet to assist community pharmacists with services such as the Minor Ailment Services These services have been implemented to reduce the burden of primary care consultations in general medical practitioners (GPs) and to allow a better utilization of community pharmacists’ skills. However, there is a need to refer high-risk patients to GPs. Aim: To develop a predictive model for high-risk patients that need referral assisting community pharmacists’ triage through a minor ailment service. Method: An ongoing pragmatic type 3 effectiveness-implementation hybrid study was undertaken at a national level in Spanish community pharmacies since October 2020. Pharmacists recruited patients presenting with minor ailments and followed them 10 days after the consultation. The main outcome measured was appropriate medical referral (in accordance with previously co-designed protocols). Nine machine learning models were tested (three statistical, three black box and three tree models) to assist pharmacists in the detection of high-risk individuals in need of referral. Results: Over 14′000 patients were included in the study. Most patients were female (68.1%). With no previous treatment for the specific minor ailment (68.0%) presented. A percentage of patients had referral criteria (13.8%) however, not all of these patients were referred by the pharmacist to the GP (8.5%). The pharmacists were using their clinical expertise not to refer these patients. The primary prediction model was the radial support vector machine (RSVM) with an accuracy of 0.934 (CI95 = [0.926,0.942]), Cohen’s kappa of 0.630, recall equal to 0.975 and an area under the curve of 0.897. Twenty variables (out of 61 evaluated) were included in the model. radial support vector machine could predict 95.2% of the true negatives and 74.8% of the true positives. When evaluating the performance for the 25 patient’s profiles most frequent in the study, the model was considered appropriate for 56% of them. Conclusion: A RSVM model was obtained to assist in the differentiation of patients that can be managed in community pharmacy from those who are at risk and should be evaluated by GPs. This tool potentially increases patients’ safety by increasing pharmacists’ ability to differentiate minor ailments from other medical conditions.


Introduction
Community pharmacists' scope of practice has been extending over time with the inclusion of services such as minor ailment services, medication therapy management or immunization are now under the pharmacists' responsibility (Adams and Weaver, 2021;DiPietro Mager and Bright, 2022).
In Spain different pharmacy services are nationally defined and protocolized (ForoAF-FC, 2022), although not all of them are implemented or remunerated by the national health system. The minor ailment service (MAS) has been defined as the one "provided upon patient request in the pharmacy, when unsure of which medicinal product to acquire and upon requesting that the pharmacist provide the most appropriate remedy for a specific health problem" (ForoAF-FC, 2022).
In many other countries, such as the United Kingdom and Canada, MASs have been implemented to increase the capacity of primary care (Aly et al., 2018). Their main objectives are to reduce the burden of primary care consultations with general medical practitioners (GPs) allowing them to focus on more complex patients (Aly et al., 2018), to assist patients in choosing the appropriate level of care and to allow a better utilization of community pharmacist skills which extends their role in primary care (Butterworth et al., 2017;Bradley et al., 2018). A systematic review including studies from studies the United Kingdom, Europe, Australia, Canada, and Singapore, concluded that triage services in community pharmacy reduce the burden on other healthcare services by diverting those seeking treatment for minor ailments (Curley et al., 2016).
MAS includes clinical protocols for pharmacists to assist in the triage of patients that can be managed in situ or to those at risk that require referral to other healthcare professionals. One of the main characteristics of MASs is a series of clinical management pathways, which include referral criteria and red flags for minor ailments (Curley et al., 2016). Over sixty different minor ailments are included in protocols in international MASs in countries such as Scotland (gov.scot. NHS Pharmacy First Scotland: information for patients, 2020), England (Boots. NHS Minor, 2022), Wales (NHS. Common, 2019), Canada (CPhA, 2022) and Spain (Amador-Fernández et al., 2018). The protocols, that were mutually agreed with GPs, reflect the potential of community pharmacies to assess common problems that can frequently lead to consultations with GPs or emergency departments (Fielding et al., 2015). MAS, with its underpinning clinical protocols, allows the differentiation of patients' demands from patients' needs in order to identify the most appropriate entry point into the healthcare system whilst ensuring patient safety (Robertson-Steel, 2006). There is evidence that pharmacist-led MASs improve healthcare outcomes (Amador-Fernández et al., 2019;Dineen-Griffin et al., 2021;Stämpfli et al., 2022).
More recently, technologies, such as machine learning, have been shown to be an important diagnostic tool assisting healthcare professionals (WHO, 2021). Machine learning is a field of artificial intelligence that focuses on the study of algorithms that learn from previous data (Mitchell, 1997). The World Health Organization has emphasized the future of artificial intelligence and its use in aiding decision-making by clinicians (WHO, 2021). Machine learning has already been used for aiding disease prognosis in cardiovascular diseases (Azmi et al., 2022), diagnosing neurodegenerative diseases (Myszczynska et al., 2020) and for classifying and predicting the progression of multiple sclerosis (Vázquez-Marrufo et al., 2022). The use of predictive models and their impact is said to be increasing exponentially (NCBI, 2016) and, although these models cannot replace GP diagnostic intelligence, it could augment it (Summerton and Cansdale, 2019).
Machine learning has been used in some pharmacy settings to detect adverse drug events (Bouzillé et al., 2019) and to evaluate patients' adherence (Lo-Ciganic et al., 2015;Koesmahargyo et al., 2020) but it has not yet been used to assist community pharmacists provide triage services to appropriately identify high-risk individuals in need of referral. Therefore, there is a need of using advance technologies to assist pharmacists in their daily tasks of triaging patients consulting or demanding a treatment for a minor ailment in community pharmacy.
The aim of this study was to develop a predictive model for highrisk patients that need referral assisting community pharmacists' triage through a MAS.

Study design
Data for the development of the predictive model was derived from a pragmatic type 3 effectiveness-implementation hybrid study Frontiers in Pharmacology frontiersin.org (Curran et al., 2012), which commenced in October 2020 and is ongoing, at a national level in Spanish community pharmacies. The study protocol was registered in ClinicalTrials.gov NCT05247333. A brief description of the study is as follows.

Study population
Six Pharmaceutical Associations and the Spanish Society of Clinical, Family and Community Pharmacy (SEFAC) invited their member community pharmacies/pharmacists to take part in the program through a number of channels. Patients included in the study were those who presented in the participating community pharmacies with symptoms or requested a medication for a minor ailment. The patient inclusion criteria were: aged ≥18 years, or younger if they were accompanied by a responsible adult, presenting one of the minor ailments listed below or other included at the discretion of the pharmacist. Clinical protocols were designed for the management of: upper respiratory tract related (nasal congestion, cold, cough); pain related (headache, joint and back pain, dental pain, sore throat, dysmenorrhea); gastroenterology (heartburn, flatulence, diarrhea, constipation, vomiting); dermatological (acne, mouth ulcers, dermatitis, soft tissue injuries, cold sore hyperhidrosis, bites and stings, athlete's foot, burns, rashes); other ailments (acute stress disorder, fever, hemorrhoids, insomnia, red eye, dry eye, vaginal candidiasis, varicose veins). These protocols were agreed through a co-design process between medical and pharmacy associations and were facilitated by researchers from the University of Granada.

Description of the intervention
The pharmacist-patient intervention is described using the Template for Intervention Description and Replication or TIDieR (de Barra et al., 2019) template (Supplementary Appendix SA1). It included a standardized consultation protocol incorporating: • A general procedure for the service (Pharmaceutical Care Forum in Community Pharmacy FA-F, 2019). • Management protocols co-developed between GPs and community pharmacists for each specific minor ailment, including referral criteria, and recommended pharmacological and non-pharmacological treatment (Amador-Fernández et al., 2018).

SEFAC, 2014) that guided pharmacists
Educational training was provided covering topics such as service provision, service protocols, communication skills with the patient and other health professionals, web-based software use and data collection. Change agents (CAs) were used to conduct the training as well as periodic follow-ups in order to provide feedback and advice to the pharmacists as well as checking the fidelity to the intervention. CAs also received monthly ongoing training, implementation strategies (Damschroder et al., 2009) and feedback by the research group.

Study outcomes
The outcome variables measured related to the impact of MAS were (Amador-Fernández et al., 2022): • Appropriate medical referral: patient referral by the pharmacist made in accordance with the co-designed protocols. • Modification of direct product request.
• Symptom resolution: relief of symptoms from 1"not at all" to 10"completely" at 10-day follow-up. • Re-consultation rate for the same minor ailment 10 days or after the pharmacist consultation.
This paper will report only on the medical referral model.

Ethics approval and consent to participate
The study was approved by the University of Granada Ethics Committee (registration number 253c23c678ee33a14d3dc16e585baf6a5a3da056).
Participating pharmacists had to sign a commitment form to participate in the study. Patients or responsible adults were requested to provide written consent after being informed of the study.

Machine learning model
Data used for obtaining the model was extracted from the webbased consultation program. For the model development, the outcome was appropriate medical referral defined as a dichotomous qualitative variable (yes/no).
There were thirteen patient-related covariates: gender, age, province, other health problems, other treatments, minor ailment consulted, previous treatment of the minor ailment, minor ailment duration, referral criteria (due to the patient's age, pregnancy/ breastfeeding, red flags, symptom duration, other treatments, health problems and others), consultation type (presenting in CP with a consultation/requesting a medication for a minor ailment), quality of life (visual analogue scale from 0 to 100), physiological status (pregnancy/breastfeeding) and day of the consultation.
Pharmacists-related covariates included: gender, age, nationality and role in the CP. Pharmacies-related covariates were location (rural/ urban), location size and number of pharmacists working in the CP.
Statistical analysis and design of the models Quantitative covariates were described using the arithmetic mean, standard deviation, median, maximum, and minimum as appropriate. Near zero variance predictors were removed. Furthermore, two non-relevant variables with more than 65% of missing values were eliminated (Anatomical Therapeutic Chemical or ATC classification of the recommended treatments and whether the treatment requested was a prescription treatment) (Kang, 2013). Remaining variables were subjected to missing value imputation using k-Nearest Neighbors. Outliers were identified and treated Frontiers in Pharmacology frontiersin.org using the Tukey's boxplot method for the covariate symptom duration and Winsorization for existing underlying health problems and the number of treatments being used (Aguinis et al., 2013). In addition, a Yeo-Johnson transformation was used to stabilize variance for the three covariates (symptom duration, underlying health problems and the number of treatments being used) and additionally for EQVAS and number of pharmacists per community pharmacy (Yeo and Johnson, 2000). Finally, correlation between covariates was analyzed to avoid multicollinearity problems. 70.0% of the database was used for obtaining the models as a training cohort and 30.0% as a test cohort (Lantz, 2019).
For the main prediction model, an additive selection technique with three feature selection methods: Kohonen's Learning Vector Quantization (LVQ) algorithm (Kohonen, 1995), Random Forest and RFE (recursive feature elimination) was used to select the number of predictors that should be included in the model (Kuhn and Johnson, 2013;Kuhn and Johnson, 2019).
Due to the high-class imbalance in the referral outcome variable (no: 91.5%, yes: 8.5%), the SMOTE (Synthetic Minority Oversampling Technique) was applied to the training dataset to reduce the class imbalance (Chawla et al., 2002). Nine models were tested for the outcome "referral to the GP": three statistical models (k-Nearest Neighbor or kNN, Naive Bayes or NB, Logistic Regression or RegLog), three black box models (Artificial Neural Network or ANN, Support Vector Machine Lineal or SVM. L, Support Vector Machine Radial or SVM. R) and three tree models (C5.0 Decision Tree or C5DT, Random Forest or RF, XGBoost or XGB).
To assess the primary prediction model, its performance was evaluated for the 25 most frequent patient profiles and for the 25 most frequent pharmacists' profiles found in the study. Accuracy, Cohen's kappa and recall were evaluated for each profile.
The software used for the analysis was R v4.2.2 with the package RStudio 2022.12.0 build 353.

Results
14,083 patients were included in the study with patient characteristics presented in Table 1. Most participant patients were female (68.1%, n = 9594), with a consultation about the minor ailment (84.6%, n = 11911), with no previous treatment for the minor ailment consulted (68.0%, n = 9588) and with no referral criteria (86.2%, n = 12146).
Most patients did not present any referral criteria (86.2%, n = 12,116). Those with referral criteria presented a mean of 1.02 (SD = 0.56) criteria with a maximum of three criteria. Red flags were the most common referral criteria (39.5%, n = 846) (Table 1). Higher number of patients presented referral criteria (13.8%, n = 1937) than those who were ultimately referred (8.5%, n = 1,194). A synthetic minority oversampling technique (SMOTE) was used to solve the class imbalance.
For obtaining the machine learning model, 70.0% (n = 9,839) of the data set was used as a training cohort and 30.0% (n = 4,218) as a test cohort. Correlation between covariates was analyzed without finding any high correlations. A moderate correlation was found between the patients' health problems and their medication (0.29); nevertheless, no additional correction was made due to the correlation being lower than 0.5. The primary prediction model (Figure 1) was the radial support vector machine (  Twenty variables (out of 61 covariates evaluated) were finally included in the model, i.e., the variables that showed statistically significant difference (p ≤ 0.01) when tested compared to a random variable using a random forest model. These variables were listed according to relevance (Figure 3): referral criteria (red flags, symptom duration, health problems, treatments, patients' age), symptom duration, CP population served, pharmacist's age, physiological status, number of pharmacists working in the CP, CP's location, pharmacist's role in the CP, pharmacist's gender, quality of life, previous treatment of the minor ailment, minor ailment consulted, month of the consultation and patient's age.
The first six covariates included in the primary prediction model were related to the patient presenting with criteria that might lead to referral (Figure 3). The importance of these referral criteria was

Discussion
Machine learning technologies are increasingly being developed and used as a diagnostic tool assisting healthcare professionals. Nevertheless, its use is not extended in community pharmacy, and it has never been developed for assisting pharmacists while triaging patients with minor ailments. This study developed a machine learning model (RSVM) to assist community pharmacists when triaging patients to better identify high-risk patients in order to differentiate those that may be managed by a community pharmacist from those who should be referred and evaluated by a GP. The model highlighted the most important patient characteristics that could predict which patients have a high probability of referral (prediction of 95.2% of the true negatives and 74.8% of the true positives). It also allowed the detection of certain pharmacists' profiles that would benefit from supplementary training.
No similar studies have been carried out in community pharmacy. Nevertheless, studies in healthcare have used machine learning techniques in other settings to assist diagnosis. A systematic review that compared machine learning techniques for cardiovascular disease prediction found models with an accuracy between 81% and 100% concluding that the random forest model produced the most significant outcomes in terms of accuracy (Azmi et al., 2022). Our RSVM model obtained better results in terms of accuracy, area under the curve and recall than those studies for the prediction of medication adherence (Koesmahargyo et al., 2020). The model included 20 variables that showed statistically significant difference when tested, similarly to those included in a model used to predict asthma exacerbation in an upcoming year (Jiao et al., 2022).

Implications for research and practice
Although, studies have reported that patients can be appropriately triaged in community pharmacy to GPs (Dineen-Griffin et al., 2020;Amador-Fernández et al., 2022); the use of predictive modelling at the time of the consultation would suggest, although not proven, a contribution to the increase in patient safety, clinical decision-making and governance. Future research should focus on developing artificial intelligence techniques to help clinicians in daily practice, always taking into consideration the legal and ethical aspects.
There are two main implications for practice: an increase in patients' safety and improved community pharmacy practice. Interestingly, a study found that pharmacist self-perception to differentiate minor ailments from similar conditions has been found to have a low score when evaluating competencies to manage minor ailments such as recommendation of appropriate over the counter medicines (Makhlouf et al., 2021). Clinical protocols and guidelines together with machine learning models may increase pharmacists' ability to differentiate minor ailments from other medical conditions. Secondly, a key element when triaging and/or diagnosing is not only the patients' characteristics but also the health providers' traits and their abilities to perform clinically. The model detected certain pharmacists' profiles on which the CAs could focus their interventions, intensify training and perform more frequent follow-ups with the ultimate aim of improving patients' clinical results.

Strengths and limitations
To our knowledge, this is the first study using machine learning to develop a predictive model that can assist community pharmacists delivering MASs to triage patients. A major strength is that all data used to construct the model emanated from data derived from actual practices. New data will be used for the continuing learning of the model, permitting further improvement of the predictive ability of the model (now it was considered appropriate for 56% of the 25 most common patients' profiles included in the study).
One of the main limitations of the study was the lack of availability of individual patient clinical and medical history, as community pharmacies do not have access to patient clinical history. The model was therefore restricted to the data obtained during the pharmacistpatient consultation, reflecting usual practice. Model performance could potentially be improved by adding more specific biomarkers that allow patient differentiation, which currently is not feasible in real practice for community pharmacy.

Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Requests to access these datasets should be directed to noelia.af@outlook.com.

Ethics statement
The studies involving human participants were reviewed and approved by the University of Granada Ethics Committee. The patients/participants provided their written informed consent to participate in this study.

Funding
The funding for the study was provided by a national organization, the Spanish Society of Clinical, Family and Community Pharmacy and the Pharmaceutical Associations of Valencia, Madrid, Gipuzkoa, Malaga, Castellon and Valladolid. It was a contract signed by the University of Granada and the above name associations. The funders had no direct role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. They had no role in deciding to publish this information or the analysis of the data. Open access funding by University of Lausanne.