<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Hum. Neurosci.</journal-id>
<journal-title>Frontiers in Human Neuroscience</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Hum. Neurosci.</abbrev-journal-title>
<issn pub-type="epub">1662-5161</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fnhum.2021.716643</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Human Neuroscience</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Machine Learning for Subtyping Concussion Using a Clustering Approach</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Rosenblatt</surname> <given-names>Cirelle K.</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<xref ref-type="corresp" rid="c001"><sup>&#x0002A;</sup></xref>
<xref ref-type="author-notes" rid="fn002"><sup>&#x02020;</sup></xref>
<xref ref-type="author-notes" rid="fn003"><sup>&#x02021;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1346941/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Harriss</surname> <given-names>Alexandra</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
</contrib>
<contrib contrib-type="author">
<name><surname>Babul</surname> <given-names>Aliya-Nur</given-names></name>
<xref ref-type="aff" rid="aff3"><sup>3</sup></xref>
<xref ref-type="author-notes" rid="fn002"><sup>&#x02020;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1399504/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Rosenblatt</surname> <given-names>Samuel A.</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
</contrib>
</contrib-group>
<aff id="aff1"><sup>1</sup><institution>Advance Concussion Clinic Inc.</institution>, <addr-line>Vancouver, BC</addr-line>, <country>Canada</country></aff>
<aff id="aff2"><sup>2</sup><institution>Division of Sport &#x00026; Exercise Medicine, Department of Family Practice, Faculty of Medicine, University of British Columbia</institution>, <addr-line>Vancouver, BC</addr-line>, <country>Canada</country></aff>
<aff id="aff3"><sup>3</sup><institution>Department of Astronomy, Columbia University</institution>, <addr-line>New York, NY</addr-line>, <country>United States</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Carol A. DeMatteo, McMaster University, Canada</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Thomas Edward Doyle, McMaster University, Canada; Robert L. Kane, Self-Employed, Washington, DC, United States</p></fn>
<corresp id="c001">&#x0002A;Correspondence: Cirelle K. Rosenblatt <email>crosenblatt&#x00040;advanceconcussion.com</email></corresp>
<fn fn-type="other" id="fn001"><p>This article was submitted to Brain Health and Clinical Neuroscience, a section of the journal Frontiers in Human Neuroscience</p></fn>
<fn fn-type="equal" id="fn002"><p>&#x02020;These authors share first authorship</p></fn>
<fn fn-type="equal" id="fn003"><p>&#x02021;Senior author</p></fn></author-notes>
<pub-date pub-type="epub">
<day>30</day>
<month>09</month>
<year>2021</year>
</pub-date>
<pub-date pub-type="collection">
<year>2021</year>
</pub-date>
<volume>15</volume>
<elocation-id>716643</elocation-id>
<history>
<date date-type="received">
<day>28</day>
<month>05</month>
<year>2021</year>
</date>
<date date-type="accepted">
<day>31</day>
<month>08</month>
<year>2021</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2021 Rosenblatt, Harriss, Babul and Rosenblatt.</copyright-statement>
<copyright-year>2021</copyright-year>
<copyright-holder>Rosenblatt, Harriss, Babul and Rosenblatt</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license>
</permissions>
<abstract>
<p><bold>Background:</bold> Concussion subtypes are typically organized into commonly affected symptom areas or a combination of affected systems, an approach that may be flawed by bias in conceptualization or the inherent limitations of interdisciplinary expertise.</p>
<p><bold>Objective:</bold> The purpose of this study was to determine whether a bottom-up, unsupervised, machine learning approach, could more accurately support concussion subtyping.</p>
<p><bold>Methods:</bold> Initial patient intake data as well as objective outcome measures including, the Patient-Reported Outcomes Measurement Information System (PROMIS), Dizziness Handicap Inventory (DHI), Pain Catastrophizing Scale (PCS), and Immediate Post-Concussion Assessment and Cognitive Testing Tool (ImPACT) were retrospectively extracted from the Advance Concussion Clinic&#x00027;s database. A correlation matrix and principal component analysis (PCA) were used to reduce the dimensionality of the dataset. Sklearn&#x00027;s agglomerative clustering algorithm was then applied, and the optimal number of clusters within the patient database were generated. Between-group comparisons among the formed clusters were performed using a Mann-Whitney U test.</p>
<p><bold>Results:</bold> Two hundred seventy-five patients within the clinics database were analyzed. Five distinct clusters emerged from the data when maximizing the Silhouette score (0.36) and minimizing the Davies-Bouldin score (0.83). Concussion subtypes derived demonstrated clinically distinct profiles, with statistically significant differences (<italic>p</italic> &#x0003C; 0.05) between all five clusters.</p>
<p><bold>Conclusion:</bold> This machine learning approach enabled the identification and characterization of five distinct concussion subtypes, which were best understood according to levels of complexity, ranging from Extremely Complex to Minimally Complex. Understanding concussion in terms of Complexity with the utilization of artificial intelligence, could provide a more accurate concussion classification or subtype approach; one that better reflects the true heterogeneity and complex system disruptions associated with mild traumatic brain injury.</p>
</abstract>
<kwd-group>
<kwd>concussion</kwd>
<kwd>artificial intelligence</kwd>
<kwd>cluster analysis</kwd>
<kwd>interdisciplinary</kwd>
<kwd>rehabilitation</kwd>
<kwd>mild traumatic brain injury</kwd>
<kwd>complexity</kwd>
</kwd-group>
<counts>
<fig-count count="4"/>
<table-count count="1"/>
<equation-count count="0"/>
<ref-count count="40"/>
<page-count count="10"/>
<word-count count="5809"/>
</counts>
</article-meta>
</front>
<body>
<sec sec-type="intro" id="s1">
<title>Introduction</title>
<p>Mild traumatic brain injury (mTBI) is a growing public healthcare concern, and presents a substantial burden to patients, families and health care systems (Langer et al., <xref ref-type="bibr" rid="B27">2020</xref>). The incidence rate of this silent epidemic has significantly increased over the past decade, and accounts for the majority of all reported traumatic brain injury cases (Rao et al., <xref ref-type="bibr" rid="B33">2017</xref>). While the majority of patients recover within 3 months, up to 30% of patients experience persistent concussion symptoms, affecting their ability to return to school, work, and activities of daily living (Dennis et al., <xref ref-type="bibr" rid="B10">2019</xref>; Permenter et al., <xref ref-type="bibr" rid="B31">2021</xref>). Given the significant increase in concussion and economic burden to health care systems, there is a need for effective and efficient evaluation of these injuries by healthcare professionals to implement accurate and timely management strategies.</p>
<p>Concussion reflects in a variety of affected systems and areas of co-occurring disruption that requires interdisciplinary management, an approach that is universally recommended in consensus statements and best-practice clinical guidelines (Collins et al., <xref ref-type="bibr" rid="B7">2016</xref>; McCrory et al., <xref ref-type="bibr" rid="B29">2017</xref>; Schneider et al., <xref ref-type="bibr" rid="B35">2019</xref>). It is the heterogeneous nature of concussion, from its etiology and pathophysiology, to its individual clinical presentation and variable recovery trajectories, that has made its management especially challenging for the clinician or primary care physician. Moreover, in regions or health care systems where interdisciplinary care falls outside insured or core medical services, translating concussion best-practices in the application of the interdisciplinary treatment model represents a further barrier toward effective concussion management and meaningful progress in addressing the concussion epidemic.</p>
<p>Growing evidence of the complexity of concussion has given rise to the development of clinical subtypes, with steady empirical support (Collins et al., <xref ref-type="bibr" rid="B8">2014</xref>; Ellis et al., <xref ref-type="bibr" rid="B12">2016</xref>, <xref ref-type="bibr" rid="B11">2018</xref>; Kontos and Collins, <xref ref-type="bibr" rid="B24">2018</xref>; Lumba-Brown et al., <xref ref-type="bibr" rid="B28">2020</xref>), and is considered a valuable tool toward informing clinical decision making, treatment planning and the conceptualization of targeted rehabilitation pathways. Similar to Collins et al. (<xref ref-type="bibr" rid="B8">2014</xref>) early delineation of six distinct sport-related concussion subtypes based on patient-reported symptoms following one-week post-injury, subsequent subtypes, or post-concussion disorders, have been organized according to symptoms, impairments or a combination of affected systems.</p>
<p>However, these subtypes may be vulnerable to bias in their conceptualization (Langdon et al., <xref ref-type="bibr" rid="B26">2020</xref>), or inherently limited by the application of discipline specific expertise. Certainly, there is a disproportionate body of research on sport related concussion with notable gaps in our understanding of mTBI associated with motor vehicle accidents and other causes. Moreover, the complexity of concussion and its multi-factorial nature does not align with the approaches that assimilate the dimensionality of concussion into its basic units of impairment. A reductionism approach&#x02013;that isolates a single factor, or combination of factors, and assumes them to be the cause of injury or impairment (Hulme and Finch, <xref ref-type="bibr" rid="B18">2015</xref>)&#x02013;may be useful in understanding causal, linear relationships (Bittencourt et al., <xref ref-type="bibr" rid="B4">2016</xref>), however the non-linear, multifaceted entity that is concussion may require more sophisticated methods to capture the complex determinants that influence outcomes. As Langer et al. (<xref ref-type="bibr" rid="B27">2020</xref>) suggested, such a model requires &#x0201C;testing in an overview of empirical evidence focusing on data driven clustering of symptoms&#x0201D; into concussion subtypes. Kontos and Collins (<xref ref-type="bibr" rid="B24">2018</xref>), identified the need for mapping symptom clusters across various domains that quantified symptom clusters of concussion into objective deficits in functional outcome domain measures.</p>
<p>Analyses that integrate Artificial Intelligence (AI) and embrace a systems approach to concussion, with inherent non-linearity and complex, dynamic interactions, may improve our ability to identify patterns of system disruption in such multidimensional injuries as concussion. Recent AI work has used machine learning to predict symptom resolution following sport-related concussion (Bergeron et al., <xref ref-type="bibr" rid="B3">2019</xref>). While another study used a clustering approach on vestibular and balance diagnostic data, and demonstrated two clinically distinct groups, patients with prominent vestibular disorders and others with no clear vestibular or balance impairment (Visscher et al., <xref ref-type="bibr" rid="B40">2019</xref>). Specifically in consideration of the heterogeneity of concussion and noting the ways in which this complicates research efforts, Kenzie et al. (<xref ref-type="bibr" rid="B23">2018</xref>) utilized causal loop diagramming to visualize relationships between concussion injury factors, including pathophysiology, deficits, symptom persistence and recovery trajectories.</p>
<p>Advances in machine learning can provide a distinct practical advantage to healthcare providers (Davenport and Kalakota, <xref ref-type="bibr" rid="B9">2019</xref>). The advent of using machine learning in addressing complex healthcare questions is already underway, demonstrating promise in automating and assisting in clinical diagnoses and treatment response (Garcia-Vidal et al., <xref ref-type="bibr" rid="B14">2019</xref>; Nakata, <xref ref-type="bibr" rid="B30">2019</xref>; Stevens et al., <xref ref-type="bibr" rid="B37">2019</xref>). Compared to traditional statistics, machine learning can identify non-linear relationships and high-order interactions between multiple variables, where traditional statistics fall short (Bergeron et al., <xref ref-type="bibr" rid="B3">2019</xref>). Thus, machine learning could more appropriately address multifaceted and complicated human health conditions, such as concussion.</p>
<p>Hierarchical agglomerative clustering, is a unsupervised, bottom-up, machine learning approach that can identify subgroups from complex data and provide an opportunity to classify clinical patterns as well as create novel representations of clinical profiles (Hassan et al., <xref ref-type="bibr" rid="B16">2020</xref>). There are substantial implications for research on concussion subtypes, but also the utilization of machine learning to help in interpreting overall assessment results and summarizing multiple parameters to identify which features or combination of features discriminate between clinical profiles. This approach aligns with the heterogeneity, complexity, and diversity of concussion.</p>
<p>The purpose of this study was to determine whether a bottom-up, unsupervised, machine learning approach, could provide insight into different concussion clinical profiles by using objective outcome measures, including the Patient-Reported Outcomes Measurement Information System (PROMIS), Dizziness Handicap Inventory (DHI), Pain Catastrophizing Scale (PCS), and Immediate Post-Concussion Assessment and Cognitive Testing Tool (ImPACT). Utilization of self-administered, objective outcome measures, without the costly barriers of interdisciplinary concussion assessment were prioritized for this study.</p></sec>
<sec sec-type="materials and methods" id="s2">
<title>Materials and Methods</title>
<sec>
<title>Participants</title>
<p>In this retrospective study design, a cluster analysis was used on 275 patients in the initial intake database of the Advance Concussion Clinic, in Vancouver, British Columbia, Canada. Patients who were 18 years of age and older as well as completed the PROMIS (Version 2.1), DHI, PCS and ImPACT between January 2018 to December 2019 were included in the analysis. Patients were excluded from the analysis if, &#x0003E;5% of the patient&#x00027;s data were missing from the database, the patient did not allow the use of their data for research purposes, the patient did not complete the objective outcome measures at their initial assessment.</p></sec>
<sec>
<title>Database</title>
<p>The Advance Concussion Clinic initial intake database consists of patient reported outcome measures obtained from PROMIS, DHI, PCS, and ImPACT. PROMIS is set of person-centered measures that evaluates and monitors domains of pain interference, fatigue, depression, anxiety, sleep disturbance, cognitive concerns and abilities, physical function and social function on a one to five numeric rating scale, as well as an average pain intensity score, on a zero to 10 numeric rating scale (Cella et al., <xref ref-type="bibr" rid="B6">2010</xref>). The DHI is a 25-item form that evaluates a patients self-perceived handicapping effects imposed by vestibular dysfunction (Jacobson and Newman, <xref ref-type="bibr" rid="B21">1990</xref>). The PCS is a 13-item scale that assesses three aspects of catastrophizing: helplessness, rumination and magnification (Sullivan et al., <xref ref-type="bibr" rid="B38">1995</xref>). ImPACT is a computerized neurocognitive testing measure, which consists of six cognitive test modules. These six modules are utilized to generate four composite scores: verbal memory, visual memory, visual-motor processing speed, and reaction time. A number of studies have reported on the test&#x00027;s validity and utility in identifying subtle cognitive changes associated with concussion (Schatz et al., <xref ref-type="bibr" rid="B34">2006</xref>; Van Kampen et al., <xref ref-type="bibr" rid="B39">2006</xref>; Broglio et al., <xref ref-type="bibr" rid="B5">2007</xref>; Alsalaheen et al., <xref ref-type="bibr" rid="B2">2016</xref>). The Acute Concussion Evaluation (Gioia et al., <xref ref-type="bibr" rid="B15">2008</xref>) and Concussion Grading Scale (CGS) were also extracted from the initial assessment. The CGS is a 21-item self-report measure that records symptom severity using a 7-point Linkert scale. Studies demonstrate that the CGS scale is able to discriminate between concussed and non-concussed patients (Schatz et al., <xref ref-type="bibr" rid="B34">2006</xref>; Broglio et al., <xref ref-type="bibr" rid="B5">2007</xref>).</p></sec>
<sec>
<title>Data Analysis</title>
<p>Data extracted from the Acute Concussion Evaluation were generated using descriptive statistics in SPSS statistical software (IBM Corp, <xref ref-type="bibr" rid="B19">2017</xref>) and are reported as mean and standard deviation. For the cluster analysis, participant data were imported into Python (Python, Wilmington, DE: Python Software Foundation) and analyses were carried out using the scikit-learn toolkit (Abraham et al., <xref ref-type="bibr" rid="B1">2014</xref>). Patient data were stored into a single matrix, where each row represented one patient and each column one variable. As a first step to reduce multicollinearity, a correlation matrix was used and redundant features were discarded. Moderate correlation areas (<italic>r</italic> &#x0003E; 0.55) were reduced to the element with the highest inter-patient variability (Schober et al., <xref ref-type="bibr" rid="B36">2018</xref>). A principal component analysis (PCA) was then applied to reduce the dimensionality of the dataset (Jolliffe and Cadima, <xref ref-type="bibr" rid="B22">2016</xref>). A PCA extracts information needed to explain the highest amount of variance within the dataset and in turn produces a set of new orthogonal variables called principal components (Feeny et al., <xref ref-type="bibr" rid="B13">2020</xref>). Following, sklearn&#x00027;s agglomerative clustering algorithm was used on the determined principal components.</p>
<p>Agglomerative clustering is a hierarchical bottom up clustering approach which groups objects into clusters based on n their similarity. It is particularly suited for datasets where clusters maybe unevenly shaped, of unequal size and unequally distributed across parameter space (Hirano et al., <xref ref-type="bibr" rid="B17">2004</xref>). In this hierarchical cluster analysis, the model is initialized by assuming that each datapoint is an individual cluster (Feeny et al., <xref ref-type="bibr" rid="B13">2020</xref>; Hassan et al., <xref ref-type="bibr" rid="B16">2020</xref>). Similarity and linkage are the two parameters of greatest importance for agglomerative clustering. Similarity was calculated using the euclidean distance between two data-points in PCA space while linkage was measured as the variance of each cluster (ward linkage).</p>
<p>Ward linkage was chosen to minimize the variance of each cluster ensuring that assessment scores of patients in each cluster were maximally uniform (Hirano et al., <xref ref-type="bibr" rid="B17">2004</xref>). For the remaining parameters we used the default sklearn. At each iteration of the algorithm, the clusters with the shortest distance merge. The distance between clusters containing multiple data values is calculated using the minimum distance between a point in cluster <italic>x</italic> and a point in cluster <italic>y</italic>. The algorithm will continue merging clusters until stopped or there is only one large cluster. Since agglomerative clustering begins by assigning each datapoint a cluster, very few assumptions are made about the data. This is one of the strengths of agglomerative clustering, compared to other clustering methods, such as k-means (Hassan et al., <xref ref-type="bibr" rid="B16">2020</xref>).</p>
<p>To determine the optimal number of clusters, the Silhouette and Davies-Bouldin scores were used (Feeny et al., <xref ref-type="bibr" rid="B13">2020</xref>; Hassan et al., <xref ref-type="bibr" rid="B16">2020</xref>). The Silhouette score is used to determine the separation distance between the resulting clusters (i.e., measures how similar an object is to its own cluster, compared to other clusters). In contrast, the Davies-Bouldin score is a measure of similarity between each cluster. In this analysis, we aimed to minimize the Davies-Bouldin score to ensure each cluster is maximally different from the other clusters and maximize the Silhouette score to maintain maximal uniformity between values within a given cluster. <xref ref-type="fig" rid="F1">Figure 1</xref> shows the Silhouette score as a function of the number of clusters. To improve the stability of the cluster outcomes, the agglomerative clustering model started with 100 patients to capture the initial cluster properties. Following, an additional 100 patients were added to ensure that both the properties and profiles of each cluster did not change. Finally, all remaining participants were added to ensure cluster stability in that profiles and properties remained unchanged.</p>
<fig id="F1" position="float">
<label>Figure 1</label>
<caption><p>The Silhouette score as a function of the number of clusters.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fnhum-15-716643-g0001.tif"/>
</fig>
<p>A Mann Whitney U test was used to determine significant differences between each assessment in each cluster against the same assessment in another cluster (i.e., pain interference in cluster one against pain interference in cluster two). A Bonferroni Correction was used on multiple comparisons (Ranstam, <xref ref-type="bibr" rid="B32">2016</xref>). For all analyses, a statistical significance was assessed using <italic>p</italic> &#x0003C; 0.05 and confidence interval of 95% (<xref ref-type="table" rid="T1">Table 1</xref>).</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p>Means and standards deviations of the five determined clusters.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th valign="top" align="left"><bold>Cluster</bold></th>
<th valign="top" align="center" colspan="14" style="border-bottom: thin solid #000000;"><bold>Outcome measures</bold></th>
</tr>
<tr>
<th/>
<th valign="top" align="center" colspan="9" style="border-bottom: thin solid #000000;"><bold>PROMIS</bold></th>
<th valign="top" align="center" colspan="2" style="border-bottom: thin solid #000000;"><bold>DHI</bold></th>
<th valign="top" align="center" colspan="3" style="border-bottom: thin solid #000000;"><bold>ImPACT</bold></th>
</tr>
<tr>
<th/>
<th valign="top" align="center"><bold>Pain interference percentile</bold></th>
<th valign="top" align="center"><bold>Pain intensity</bold></th>
<th valign="top" align="center"><bold>Physical function and mobility percentile</bold></th>
<th valign="top" align="center"><bold>Anxiety percentile</bold></th>
<th valign="top" align="center"><bold>Depression percentile</bold></th>
<th valign="top" align="center"><bold>Sleep disturbance percentile</bold></th>
<th valign="top" align="center"><bold>Ability to participate in social roles percentile</bold></th>
<th valign="top" align="center"><bold>Cognitive function percentile</bold></th>
<th valign="top" align="center"><bold>Fatigue percentile</bold></th>
<th valign="top" align="center"><bold>DHI total</bold></th>
<th valign="top" align="center"><bold>DHI functional</bold></th>
<th valign="top" align="center"><bold>Visual motor speed composite</bold></th>
<th valign="top" align="center"><bold>Reaction time composite score</bold></th>
<th valign="top" align="center"><bold>Memory composite (verbal) score</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">2</td>
<td valign="top" align="center">50.97 (9.24)</td>
<td valign="top" align="center">0.22 (0.19)</td>
<td valign="top" align="center">50.80 (6.96)</td>
<td valign="top" align="center">48.6 (8.05)</td>
<td valign="top" align="center">46.04 (8.06)</td>
<td valign="top" align="center">43.69 (7.88)</td>
<td valign="top" align="center">51.63 (8.80)</td>
<td valign="top" align="center">32.86 (6.46)</td>
<td valign="top" align="center">48.27 (9.64)</td>
<td valign="top" align="center">5.91 (7.51)</td>
<td valign="top" align="center">2.52 (3.28)</td>
<td valign="top" align="center">37.54 (7.68)</td>
<td valign="top" align="center">0.69 (0.10)</td>
<td valign="top" align="center">89.83 (9.33)</td>
</tr>
<tr>
<td valign="top" align="left">3</td>
<td valign="top" align="center">60.23 (7.61)</td>
<td valign="top" align="center">0.43 (0.21)</td>
<td valign="top" align="center">42.82 (7.09)</td>
<td valign="top" align="center">60.22 (8.87)</td>
<td valign="top" align="center">55.71 (7.56)</td>
<td valign="top" align="center">54.27 (7.11)</td>
<td valign="top" align="center">39.95 (7.98)</td>
<td valign="top" align="center">43.39 (7.33)</td>
<td valign="top" align="center">61.20 (7.45)</td>
<td valign="top" align="center">22.70 (12.56)</td>
<td valign="top" align="center">9.63 (5.77)</td>
<td valign="top" align="center">38.62 (7.77)</td>
<td valign="top" align="center">0.67 (0.13)</td>
<td valign="top" align="center">90.20 (6.92)</td>
</tr>
<tr>
<td valign="top" align="left">1</td>
<td valign="top" align="center">61.42 (5.99)</td>
<td valign="top" align="center">0.46 (0.20)</td>
<td valign="top" align="center">40.82 (7.01)</td>
<td valign="top" align="center">57.71 (6.89)</td>
<td valign="top" align="center">55.17 (7.99)</td>
<td valign="top" align="center">55.01 (6.97)</td>
<td valign="top" align="center">40.03 (7.52)</td>
<td valign="top" align="center">42.86 (5.38)</td>
<td valign="top" align="center">59.94 (5.92)</td>
<td valign="top" align="center">32.77 (14.05)</td>
<td valign="top" align="center">12.85 (6.36)</td>
<td valign="top" align="center">28.37 (6.90)</td>
<td valign="top" align="center">0.86 (0.18)</td>
<td valign="top" align="center">62.45 (10.68)</td>
</tr>
<tr>
<td valign="top" align="left">0</td>
<td valign="top" align="center">67.76 (5.80)</td>
<td valign="top" align="center">0.62 (0.18)</td>
<td valign="top" align="center">36.35 (5.23)</td>
<td valign="top" align="center">66.65 (7.81)</td>
<td valign="top" align="center">62.23 (8.30)</td>
<td valign="top" align="center">58.48 (7.96)</td>
<td valign="top" align="center">34.12 (5.92)</td>
<td valign="top" align="center">49.57 (6.49)</td>
<td valign="top" align="center">68.24 (6.35)</td>
<td valign="top" align="center">60.94 (11.58)</td>
<td valign="top" align="center">24.97 (5.09)</td>
<td valign="top" align="center">30.50 (7.13)</td>
<td valign="top" align="center">0.80 (0.17)</td>
<td valign="top" align="center">79.63 (9.77)</td>
</tr>
<tr>
<td valign="top" align="left">4</td>
<td valign="top" align="center">69.94 (8.99)</td>
<td valign="top" align="center">0.73 (0.24)</td>
<td valign="top" align="center">33.98 (4.22)</td>
<td valign="top" align="center">65.88 (6.88)</td>
<td valign="top" align="center">63.46 (5.43)</td>
<td valign="top" align="center">66.87 (8.28)</td>
<td valign="top" align="center">30.41 (4.45)</td>
<td valign="top" align="center">51.69 (7.43)</td>
<td valign="top" align="center">69.46 (6.02)</td>
<td valign="top" align="center">71.75 (9.66)</td>
<td valign="top" align="center">30.13 (3.69)</td>
<td valign="top" align="center">20.53 (6.61)</td>
<td valign="top" align="center">1.10 (0.24)</td>
<td valign="top" align="center">47.19 (10.72)</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>Mann Whitney U tests determined statistically significant in-between differences across all assessments in each of the clusters (p &#x0003C; 0.05)</italic>.</p>
</table-wrap-foot>
</table-wrap></sec></sec>
<sec sec-type="results" id="s3">
<title>Results</title>
<sec>
<title>Participant Characteristics</title>
<p>The mean age of participants was 37.84 (SD = 12.50 years of age). Females represented 51% (<italic>n</italic> = 139) of the study population, males 48% (<italic>n</italic> = 133), and 1% (<italic>n</italic> = 3) did not disclose this information. The majority of concussions occurred following a motor vehicle accident (<italic>n</italic> = 148, 54%), and remaining mechanisms of injury were sport (<italic>n</italic> = 70, 25%), falls (<italic>n</italic> = 22, 8%), and other (<italic>n</italic> = 35, 13%). Loss of consciousness (LOC) was reported in 73 participants (27%), with 202 (73%) denying or not reporting LOC. The mean total for the CGS was 52.8 &#x000B1; 29.8. Time since injury and litigation weren&#x00027;t specifically calculated.</p></sec>
<sec>
<title>Outcome of Clustering Procedure</title>
<p>The correlation matrix extracted redundant features (<italic>r</italic> &#x0003E; 0.55), that were subsequently removed from the analysis (<xref ref-type="fig" rid="F2">Figure 2</xref>). PROMIS parameters retained by the correlation matrix were Pain Interference, Fatigue, Depression, Anxiety, Sleep Disturbance, Physical Function and Mobility, Ability to Participate in Social Roles, and Cognitive Function as well as Pain Intensity on a numeric score. From DHI, the total score as well as the DHI Functional subgroup were retained. Lastly, ImPACT scores for reaction time, visual motor processing speed, and verbal memory were also used.</p>
<fig id="F2" position="float">
<label>Figure 2</label>
<caption><p>Results of correlation matrix between patient baseline characteristics and objective outcome measures including, PROMIS, CGS DHI, PCS, and ImPACT.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fnhum-15-716643-g0002.tif"/>
</fig>
<p>Principal component analysis further reduced the dimensionality of the dataset from 14 features to two features. Using sklearn&#x00027;s agglomerative clustering algorithm on the two principal components, the maximum silhouette score was 0.36 and minimum Davies Bouldin score was 0.83. Therefore, it was determined that the optimal number of clusters was five (<xref ref-type="fig" rid="F3">Figure 3</xref>). Since all patients in the study presented with a concussion, the range of values for any given assessment was not very large. In addition, all patients suffered from pain and movement related symptoms so it was expected the Davies Bouldin score would lie closer to 1, indicating similarities between clusters. shows the clinically distinct five clusters. Mann Whitney U tests determined statistically significant in-between differences across all assessments in each of the clusters (<italic>p</italic> &#x0003C; 0.05).</p>
<fig id="F3" position="float">
<label>Figure 3</label>
<caption><p>Scatterplot of the formed five clusters.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fnhum-15-716643-g0003.tif"/>
</fig></sec></sec>
<sec sec-type="discussion" id="s4">
<title>Discussion</title>
<p>Concussion is only seeming to grow in complexity as research and evidence advance our understanding of its heterogenous nature. From its etiology and pathophysiology, to its individual clinical presentation and variable recovery trajectories, clinical management has been especially challenging for the clinician or primary care physician. Research has varied widely in how it has dealt with this increasing complexity, with some seeming to lean into the complexity by highlighting systems approaches and recursive modeling (Schneider et al., <xref ref-type="bibr" rid="B35">2019</xref>), while others argue that its heterogeneity is associated with mutually reinforcing biopsychosocial symptoms rather than a single entity (Iverson, <xref ref-type="bibr" rid="B20">2019</xref>).</p>
<p>Clinically, management of the concussion patient has continued to grow in complexity as well, with increasing challenge for the primary care doctor or individual clinician, particularly in context of westernized medical frameworks and reimbursement systems. Multiple risk factors have been identified for prolonged recovery from concussion such as previous concussion history, developmental delay, headache history and psychiatric history, all of which require consideration together with a multitude of other variables that can influence outcomes. Clinical assessment of the concussion patient requires consideration of the above as well as potential interactions between them, all of which form the unique patterns of presentation in the individual concussion patient. AI is an ideal, and perhaps even necessary, partner, that can best support our ability to manage the complexity of concussion, toward gaining a better understanding of patterns that may improve our ability to diagnose and treat mTBI.</p>
<p>This study evaluated the use of an unsupervised, bottom-up, machine learning approach to analyze the clinical profiles of patients attending a private concussion clinic. Utilization of objective outcome measures provided a unique opportunity to engage machine learning with various concussion and non-concussion specific evidence-based metrics. Findings revealed five statistically significant and distinct clusters, each with unique patterns of system disruption.</p>
<p>Across each of these five clusters, 14 features were retained, which combined outcomes from PROMIS, DHI, and ImPACT. These 14 features were: Pain Interference, Pain Intensity, Physical Function and Mobility, Anxiety, Depression, Sleep Disturbance, Ability to Participate in Social Roles, Cognitive Function, Fatigue, DHI Total, DHI Functional, Visual Motor Speed, Reaction Time, and Verbal Memory. Reliable and validated self-administered outcome measures were specifically utilized to ensure the accessibility and affordability of this approach.</p>
<p>The authors considered each of five clusters, or concussion subtypes to be best understood in terms of Complexity, with the following classification system suggested for each concussion subtype: Minimally Complex (Cluster 2), Mildly Complex (Cluster 3), Moderately Complex (Cluster 1), Highly Complex (Cluster 0), Extremely Complex (Cluster 4) (<xref ref-type="fig" rid="F4">Figure 4</xref>). Level of complexity was ranked according to number and relative severity of arenas affected. &#x0201C;Complexity&#x0201D; was considered to better align with the nature and particular features of concussive injury, informing specific and treatable features that support clinical decision making. The language of &#x0201C;complexity&#x0201D; was furthermore preferred to &#x0201C;severity&#x0201D; to promote perceived control, in both practitioner and patient, while avoiding the catastrophizing that might arise with a severity approach.</p>
<fig id="F4" position="float">
<label>Figure 4</label>
<caption><p>Concussion subtypes (clusters) according to complexity.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fnhum-15-716643-g0004.tif"/>
</fig>
<p>Historically, concussion classification approaches have grouped subtypes according to particular symptoms or a combination of affected systems. From the concussion profiles suggested by Collins et al. (<xref ref-type="bibr" rid="B8">2014</xref>) to the most recent subtypes as proposed by Lumba-Brown et al. (<xref ref-type="bibr" rid="B28">2020</xref>) and Kontos et al. (<xref ref-type="bibr" rid="B25">2020</xref>), the approach to subtyping has been grounded in heuristics derived from clinical experience and patient presentation or reporting.</p>
<p>The utilization of AI offered an opportunity to engage machine learning pattern recognition to explore what clinical concussion data can inform regarding concussion profiles. Indeed, this analysis revealed five unique and distinct subtypes with various <italic>patterns</italic> of system disruption, across multiple symptoms areas. This machine learning approach may be most appropriate for capturing the complexity and heterogeneity of the dynamic injury that is concussion, and in doing so, maximizing the potential of these subtypes to support and improve clinical decision making in concussions.</p>
<p>Reliable and validated self-administered outcome measures were specifically utilized to ensure the accessibility and affordability of this approach, offering a data-driven method toward enhancing clinical judgement and decision making without requiring commonly utilized clinicaly administered measures. The utilization of self-administered objective measures is of notable value in countries and regions that are more dispersed with less access to the recommended interdisciplinary concussion team otherwise needed to assess or screen the range of clinical areas affected in concussion.</p>
<p>Moreover, the Coronavirus pandemic emphasized the need for virtual tools to support and inform concussion care when in-person options are not available. The self-administered nature of the tools utilized in this analysis, as well the interdisciplinary value of the information provided, may well reduce costs or make affordable a reliable screening for those in a fee-for-service environment, as well as for the health systems that don&#x00027;t include interdisciplinary care among their insured services.</p>
<sec>
<title>Limitations and Future Directions</title>
<p>This clustering approach was selected to maximize the value of clinically meaningful data points that were derived from a large concussion population data set comprised of evidence-based outcome measures. This novel representation of concussion subtypes may help to guide interdisciplinary management, though further study is needed to assess its value in estimating recovery timelines and response to specific treatments. Future work should focus on a evaluating a wider variety of clustering algorithms to determine if they reveal further insights into the data; however, the uniformity in the clusters generated by the agglomerative clustering algorithm provide a first insight into the interconnectedness of systems affected by concussion. Further study is needed to understand the clinical application of these profiles and to explore the utility and feasibility of these subtypes within a clinical setting.</p>
<p>As the subtypes derived were based only on patients specifically presenting with symptoms associated with concussion, certainty regarding its diagnostic value cannot be ascertained. Extension of this cluster analysis to include healthy controls would support the validation of the current clinical profiles and aid in a diagnostic context.</p>
<p>Notably, many concussion subtypes have been developed based upon sport related concussions (SRC), while this research offers evidence of concussion subtyping more broadly applicable to concussions associated with other causes that are non-sports related. While the current sample represents a majority associated with motor vehicle accidents, results may be more generalizable than research that has focused on SRC alone. Further research would be needed to confirm the generalizability of results between these, and perhaps other, distinct groups.</p></sec>
<sec>
<title>Conclusion</title>
<p>Classification of concussion according to subtypes may become a useful, if not essential, tool to support clinical diagnosis and treatment planning and is useful to concussion practitioners in directing and coordinating care, and in evaluating progress toward recovery. It stands to reason that as our knowledge and understanding of the complexity surrounding concussion grows, a more comprehensive approach is warranted.</p>
<p>This study demonstrated the novel opportunity of using AI to gain insight into the complex clinical profiles of concussion. By systematically analyzing evidence-based metrics, five clusters emerged that were not only clinically distinct, but could be used to develop a novel view of concussion complexity that better approximates the true heterogeneity of the injury. In turn, this could help inform as well as support clinical decision making, and interdisciplinary involvement more readily. Given the interdisciplinary nature of concussion assessment, and the importance of the interdisciplinary teams&#x00027; findings in treatment planning and providing Clearance&#x02014;both to learn and to sport&#x02014;AI&#x00027;s work in automating, if not its diagnosis, it&#x00027;s management, would be useful to most primary care physicians and other clinicians involved in its management.</p>
<p>Furthermore, healthcare providers without specific training in concussion, and in those parts of the world where this expertise may be largely unavailable would also benefit. With the right collaboration and balance, the integration of AI with concussion subtyping can optimize otherwise elusive or incomplete concussion recoveries.</p></sec></sec>
<sec sec-type="data-availability" id="s5">
<title>Data Availability Statement</title>
<p>The datasets presented in this article are not readily available because these data are clinical information obtained from a private healthcare database. Requests to access the datasets should be directed to <email>crosenblatt&#x00040;advanceconcussion.com</email>.</p></sec>
<sec id="s6">
<title>Ethics Statement</title>
<p>Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.</p></sec>
<sec id="s7">
<title>Author Contributions</title>
<p>CKR conceived the investigation with contribution by SAR. CKR and ANB designed the study. CKR, ANB, and SAR prepared the data. ANB trained and tested the algorithm with contribution by CKR. CKR and ANB interpreted the results. AH contributed to interpretation. AH and CKR drafted the paper and then substantially revised the paper with contributions by ANB. All authors contributed to the article and approved the submitted version.</p></sec>
<sec sec-type="COI-statement" id="conf1">
<title>Conflict of Interest</title>
<p>The authors declare the potential following conflicts of interest with respect to the research, authorship, and/or publication of this article: AH, ANB, and SAR are employed by Advance Concussion Clinic. CKR is the Founder and Clinical Director of Advance Concussion Clinic.</p></sec>
<sec sec-type="disclaimer" id="s8">
<title>Publisher&#x00027;s Note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p></sec>
</body>
<back>
<ack><p>The authors would like to thank Julia Dahlby, Paddy O&#x00027;Flaherty, Benny Freedman, and Danit Macklin, along with the entire clinical and administrative staff at Advance Concussion Clinic.</p>
</ack>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Abraham</surname> <given-names>A.</given-names></name> <name><surname>Pedregosa</surname> <given-names>F.</given-names></name> <name><surname>Eickenberg</surname> <given-names>M.</given-names></name> <name><surname>Gervais</surname> <given-names>P.</given-names></name> <name><surname>Mueller</surname> <given-names>A.</given-names></name> <name><surname>Kossaifi</surname> <given-names>J.</given-names></name> <etal/></person-group>. (<year>2014</year>). <article-title>Machine learning for neuroimaging with scikit-learn</article-title>. <source>Front. Neuroinform.</source> <volume>8</volume>:<fpage>14</fpage>. <pub-id pub-id-type="doi">10.3389/fninf.2014.00014</pub-id><pub-id pub-id-type="pmid">24600388</pub-id></citation></ref>
<ref id="B2">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Alsalaheen</surname> <given-names>B.</given-names></name> <name><surname>Stockdale</surname> <given-names>K.</given-names></name> <name><surname>Pechumer</surname> <given-names>D.</given-names></name> <name><surname>Broglio</surname> <given-names>S. P.</given-names></name></person-group> (<year>2016</year>). <article-title>Validity of the Immediate Post Concussion Assessment and Cognitive Testing (ImPACT)</article-title>. <source>Sports Med.</source> <volume>46</volume>, <fpage>1487</fpage>&#x02013;<lpage>1501</lpage>. <pub-id pub-id-type="doi">10.1007/s40279-016-0532-y</pub-id><pub-id pub-id-type="pmid">27585635</pub-id></citation></ref>
<ref id="B3">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bergeron</surname> <given-names>M. F.</given-names></name> <name><surname>Landset</surname> <given-names>S.</given-names></name> <name><surname>Maugans</surname> <given-names>T. A.</given-names></name> <name><surname>Williams</surname> <given-names>V. B.</given-names></name> <name><surname>Collins</surname> <given-names>C. L.</given-names></name> <name><surname>Wasserman</surname> <given-names>E. B.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Machine learning in modeling high school sport concussion symptom resolve</article-title>. <source>Med. Sci. Sports Exerc.</source> <volume>51</volume>, <fpage>1362</fpage>&#x02013;<lpage>1371</lpage>. <pub-id pub-id-type="doi">10.1249/MSS.0000000000001903</pub-id><pub-id pub-id-type="pmid">30694980</pub-id></citation></ref>
<ref id="B4">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bittencourt</surname> <given-names>N. F. N.</given-names></name> <name><surname>Meeuwisse</surname> <given-names>W. H.</given-names></name> <name><surname>Mendon&#x000E7;a</surname> <given-names>L. D.</given-names></name> <name><surname>Nettel-Aguirre</surname> <given-names>A.</given-names></name> <name><surname>Ocarino</surname> <given-names>J. M.</given-names></name> <name><surname>Fonseca</surname> <given-names>S. T.</given-names></name></person-group> (<year>2016</year>). <article-title>Complex systems approach for sports injuries: moving from risk factor identification to injury pattern recognition&#x02014;narrative review and new concept</article-title>. <source>Br. J. Sports Med.</source> <volume>50</volume>, <fpage>1309</fpage>&#x02013;<lpage>1314</lpage>. <pub-id pub-id-type="doi">10.1136/bjsports-2015-095850</pub-id><pub-id pub-id-type="pmid">27445362</pub-id></citation></ref>
<ref id="B5">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Broglio</surname> <given-names>S. P.</given-names></name> <name><surname>Macciocchi</surname> <given-names>S. N.</given-names></name> <name><surname>Ferrara</surname> <given-names>M. S.</given-names></name></person-group> (<year>2007</year>). <article-title>Sensitivity of the concussion assessment battery</article-title>. <source>Neurosurgery</source> <volume>60</volume>, <fpage>1050</fpage>&#x02013;<lpage>1058</lpage>. <pub-id pub-id-type="doi">10.1227/01.NEU.0000255479.90999.C0</pub-id><pub-id pub-id-type="pmid">17538379</pub-id></citation></ref>
<ref id="B6">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cella</surname> <given-names>D.</given-names></name> <name><surname>Riley</surname> <given-names>W.</given-names></name> <name><surname>Stone</surname> <given-names>A.</given-names></name> <name><surname>Rothrock</surname> <given-names>N.</given-names></name> <name><surname>Reeve</surname> <given-names>B.</given-names></name> <name><surname>Yount</surname> <given-names>S.</given-names></name> <etal/></person-group>. (<year>2010</year>). <article-title>The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005&#x02013;2008</article-title>. <source>J. Clin. Epidemiol.</source> <volume>63</volume>, <fpage>1179</fpage>&#x02013;<lpage>1194</lpage>. <pub-id pub-id-type="doi">10.1016/j.jclinepi.2010.04.011</pub-id><pub-id pub-id-type="pmid">20685078</pub-id></citation></ref>
<ref id="B7">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Collins</surname> <given-names>M. W.</given-names></name> <name><surname>Kontos</surname> <given-names>A. P.</given-names></name> <name><surname>Okonkwo</surname> <given-names>D. O.</given-names></name> <name><surname>Almquist</surname> <given-names>J.</given-names></name> <name><surname>Bailes</surname> <given-names>J.</given-names></name> <name><surname>Barisa</surname> <given-names>M.</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>Statements of Agreement From the Targeted Evaluation and Active Management (TEAM) approaches to treating concussion meeting held in Pittsburgh, October 15-16, 2015</article-title>. <source>Neurosurgery</source> <volume>79</volume>, <fpage>912</fpage>&#x02013;<lpage>929</lpage>. <pub-id pub-id-type="doi">10.1227/NEU.0000000000001447</pub-id><pub-id pub-id-type="pmid">27861416</pub-id></citation></ref>
<ref id="B8">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Collins</surname> <given-names>M. W.</given-names></name> <name><surname>Kontos</surname> <given-names>A. P.</given-names></name> <name><surname>Reynolds</surname> <given-names>E.</given-names></name> <name><surname>Murawski</surname> <given-names>C. D.</given-names></name> <name><surname>Fu</surname> <given-names>F. H.</given-names></name></person-group> (<year>2014</year>). <article-title>A comprehensive, targeted approach to the clinical care of athletes following sport-related concussion</article-title>. <source>Knee Surg. Sports Traumatol. Arthrosc.</source> <volume>22</volume>, <fpage>235</fpage>&#x02013;<lpage>246</lpage>. <pub-id pub-id-type="doi">10.1007/s00167-013-2791-6</pub-id><pub-id pub-id-type="pmid">24337463</pub-id></citation></ref>
<ref id="B9">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Davenport</surname> <given-names>T.</given-names></name> <name><surname>Kalakota</surname> <given-names>R.</given-names></name></person-group> (<year>2019</year>). <article-title>The potential for artificial intelligence in healthcare</article-title>. <source>Future Healthc J.</source> <volume>6</volume>, <fpage>94</fpage>&#x02013;<lpage>98</lpage>. <pub-id pub-id-type="doi">10.7861/futurehosp.6-2-94</pub-id><pub-id pub-id-type="pmid">31363513</pub-id></citation></ref>
<ref id="B10">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dennis</surname> <given-names>J.</given-names></name> <name><surname>Yengo-Kahn</surname> <given-names>A. M.</given-names></name> <name><surname>Kirby</surname> <given-names>P.</given-names></name> <name><surname>Solomon</surname> <given-names>G. S.</given-names></name> <name><surname>Cox</surname> <given-names>N. J.</given-names></name> <name><surname>Zuckerman</surname> <given-names>S. L.</given-names></name></person-group> (<year>2019</year>). <article-title>Diagnostic algorithms to study post-concussion syndrome using electronic health records: validating a method to capture an important patient population</article-title>. <source>J. Neurotrauma</source> <volume>36</volume>, <fpage>2167</fpage>&#x02013;<lpage>2177</lpage>. <pub-id pub-id-type="doi">10.1089/neu.2018.5916</pub-id><pub-id pub-id-type="pmid">30773988</pub-id></citation></ref>
<ref id="B11">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ellis</surname> <given-names>M. J.</given-names></name> <name><surname>Leddy</surname> <given-names>J.</given-names></name> <name><surname>Cordingley</surname> <given-names>D.</given-names></name> <name><surname>Willer</surname> <given-names>B.</given-names></name></person-group> (<year>2018</year>). <article-title>A physiological approach to assessment and rehabilitation of acute concussion in collegiate and professional athletes</article-title>. <source>Front. Neurol.</source> <volume>9</volume>:<fpage>1115</fpage>. <pub-id pub-id-type="doi">10.3389/fneur.2018.01115</pub-id><pub-id pub-id-type="pmid">30619068</pub-id></citation></ref>
<ref id="B12">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ellis</surname> <given-names>M. J.</given-names></name> <name><surname>Leddy</surname> <given-names>J.</given-names></name> <name><surname>Willer</surname> <given-names>B.</given-names></name></person-group> (<year>2016</year>). <article-title>Multi-disciplinary management of athletes with post-concussion syndrome: an evolving pathophysiological approach</article-title>. <source>Front. Neurol.</source> <volume>7</volume>:<fpage>136</fpage>. <pub-id pub-id-type="doi">10.3389/fneur.2016.00136</pub-id><pub-id pub-id-type="pmid">27605923</pub-id></citation></ref>
<ref id="B13">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Feeny</surname> <given-names>A. K.</given-names></name> <name><surname>Chung</surname> <given-names>M. K.</given-names></name> <name><surname>Madabhushi</surname> <given-names>A.</given-names></name> <name><surname>Attia</surname> <given-names>Z. I.</given-names></name> <name><surname>Cikes</surname> <given-names>M.</given-names></name> <name><surname>Firouznia</surname> <given-names>M.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Artificial intelligence and machine learning in arrhythmias and cardiac electrophysiology</article-title>. <source>Circ Arrhythm. Electrophysiol.</source> <volume>13</volume>:<fpage>e007952</fpage>. <pub-id pub-id-type="doi">10.1161/CIRCEP.119.007952</pub-id><pub-id pub-id-type="pmid">32628863</pub-id></citation></ref>
<ref id="B14">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Garcia-Vidal</surname> <given-names>C.</given-names></name> <name><surname>Sanjuan</surname> <given-names>G.</given-names></name> <name><surname>Puerta-Alcalde</surname> <given-names>P.</given-names></name> <name><surname>Moreno-Garc&#x000ED;a</surname> <given-names>E.</given-names></name> <name><surname>Soriano</surname> <given-names>A.</given-names></name></person-group> (<year>2019</year>). <article-title>Artificial intelligence to support clinical decision-making processes</article-title>. <source>EBioMed.</source> <volume>46</volume>, <fpage>27</fpage>&#x02013;<lpage>29</lpage>. <pub-id pub-id-type="doi">10.1016/j.ebiom.2019.07.019</pub-id><pub-id pub-id-type="pmid">31303500</pub-id></citation></ref>
<ref id="B15">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gioia</surname> <given-names>G. A.</given-names></name> <name><surname>Collins</surname> <given-names>M.</given-names></name> <name><surname>Isquith</surname> <given-names>P. K.</given-names></name></person-group> (<year>2008</year>). <article-title>Improving identification and diagnosis of mild traumatic brain injury with evidence: psychometric support for the acute concussion evaluation</article-title>. <source>J. Head Trauma Rehabil.</source> <volume>23</volume>, <fpage>230</fpage>&#x02013;<lpage>242</lpage>. <pub-id pub-id-type="doi">10.1097/01.HTR.0000327255.38881.ca</pub-id><pub-id pub-id-type="pmid">18650767</pub-id></citation></ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hassan</surname> <given-names>S. I.</given-names></name> <name><surname>Samad</surname> <given-names>A.</given-names></name> <name><surname>Ahmad</surname> <given-names>O.</given-names></name> <name><surname>Alam</surname> <given-names>A.</given-names></name></person-group> (<year>2020</year>). <article-title>Partitioning and hierarchical based clustering: a comparative empirical assessment on internal and external indices, accuracy, and time</article-title>. <source>Int. J. Inf. Tecnol.</source> <volume>12</volume>, <fpage>1377</fpage>&#x02013;<lpage>1384</lpage>. <pub-id pub-id-type="doi">10.1007/s41870-019-00406-7</pub-id></citation>
</ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hirano</surname> <given-names>S.</given-names></name> <name><surname>Sun</surname> <given-names>X.</given-names></name> <name><surname>Tsumoto</surname> <given-names>S.</given-names></name></person-group> (<year>2004</year>). <article-title>Comparison of clustering methods for clinical databases</article-title>. <source>Inf. Sci.</source> <volume>159</volume>, <fpage>155</fpage>&#x02013;<lpage>165</lpage>. <pub-id pub-id-type="doi">10.1016/j.ins.2003.03.011</pub-id></citation>
</ref>
<ref id="B18">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hulme</surname> <given-names>A.</given-names></name> <name><surname>Finch</surname> <given-names>C. F.</given-names></name></person-group> (<year>2015</year>). <article-title>From monocausality to systems thinking: a complementary and alternative conceptual approach for better understanding the development and prevention of sports injury</article-title>. <source>Inj. Epidemiol.</source> <volume>2</volume>:<fpage>31</fpage>. <pub-id pub-id-type="doi">10.1186/s40621-015-0064-1</pub-id><pub-id pub-id-type="pmid">26691678</pub-id></citation></ref>
<ref id="B19">
<citation citation-type="book"><person-group person-group-type="author"><collab>IBM Corp</collab></person-group> (<year>2017</year>). <source>IBM SPSS Statistics for Windows</source>. <publisher-loc>Armonk, NY</publisher-loc>: <publisher-name>IBM Corp</publisher-name>.</citation>
</ref>
<ref id="B20">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Iverson</surname> <given-names>G. L.</given-names></name></person-group> (<year>2019</year>). <article-title>Network analysis and precision rehabilitation for the post-concussion syndrome</article-title>. <source>Front. Neurol.</source> <volume>10</volume>:<fpage>489</fpage>. <pub-id pub-id-type="doi">10.3389/fneur.2019.00489</pub-id><pub-id pub-id-type="pmid">31191426</pub-id></citation></ref>
<ref id="B21">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jacobson</surname> <given-names>G. P.</given-names></name> <name><surname>Newman</surname> <given-names>C. W.</given-names></name></person-group> (<year>1990</year>). <article-title>The development of the dizziness handicap inventory</article-title>. <source>Arch. Otolaryngol. Head Neck Surg.</source> <volume>116</volume>, <fpage>424</fpage>&#x02013;<lpage>427</lpage>. <pub-id pub-id-type="doi">10.1001/archotol.1990.01870040046011</pub-id><pub-id pub-id-type="pmid">32009340</pub-id></citation></ref>
<ref id="B22">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jolliffe</surname> <given-names>I. T.</given-names></name> <name><surname>Cadima</surname> <given-names>J.</given-names></name></person-group> (<year>2016</year>). <article-title>Principal component analysis: a review and recent developments</article-title>. <source>Phil. Trans. R. Soc. A.</source> <volume>374</volume>:<fpage>20150202</fpage>. <pub-id pub-id-type="doi">10.1098/rsta.2015.0202</pub-id><pub-id pub-id-type="pmid">26953178</pub-id></citation></ref>
<ref id="B23">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kenzie</surname> <given-names>E. S.</given-names></name> <name><surname>Parks</surname> <given-names>E. L.</given-names></name> <name><surname>Bigler</surname> <given-names>E. D.</given-names></name> <name><surname>Wright</surname> <given-names>D. W.</given-names></name> <name><surname>Lim</surname> <given-names>M. M.</given-names></name> <name><surname>Chestnutt</surname> <given-names>J. C.</given-names></name> <etal/></person-group>. (<year>2018</year>). <article-title>The dynamics of concussion: mapping pathophysiology, persistence, and recovery with causal-loop diagramming</article-title>. <source>Front. Neurol.</source> <volume>9</volume>:<fpage>203</fpage>. <pub-id pub-id-type="doi">10.3389/fneur.2018.00203</pub-id><pub-id pub-id-type="pmid">29670568</pub-id></citation></ref>
<ref id="B24">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kontos</surname> <given-names>A. P.</given-names></name> <name><surname>Collins</surname> <given-names>M. W.</given-names></name></person-group> (<year>2018</year>). <source>Concussion: A Clinical Profile Approach to Assessment and Treatment</source>. <publisher-loc>Washington, DC</publisher-loc>: <publisher-name>American Psychological Association</publisher-name>, <fpage>23</fpage>&#x02013;<lpage>58</lpage>.</citation>
</ref>
<ref id="B25">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kontos</surname> <given-names>A. P.</given-names></name> <name><surname>Elbin</surname> <given-names>R. J.</given-names></name> <name><surname>Trbovich</surname> <given-names>A.</given-names></name> <name><surname>Womble</surname> <given-names>M.</given-names></name> <name><surname>Said</surname> <given-names>A.</given-names></name> <name><surname>Sumrok</surname> <given-names>V. F.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Concussion Clinical Profiles Screening (CP Screen) tool: preliminary evidence to inform a multidisciplinary approach</article-title>. <source>Neurosurgery</source> <volume>87</volume>, <fpage>348</fpage>&#x02013;<lpage>356</lpage>. <pub-id pub-id-type="doi">10.1093/neuros/nyz545</pub-id><pub-id pub-id-type="pmid">31950187</pub-id></citation></ref>
<ref id="B26">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Langdon</surname> <given-names>S.</given-names></name> <name><surname>K&#x000F6;nigs</surname> <given-names>M.</given-names></name> <name><surname>Adang</surname> <given-names>E. a. M. C.</given-names></name> <name><surname>Goedhart</surname> <given-names>E.</given-names></name> <name><surname>Oosterlaan</surname> <given-names>J.</given-names></name></person-group> (<year>2020</year>). <article-title>Subtypes of sport-related concussion: a systematic review and meta-cluster analysis</article-title>. <source>Sports Med.</source> <volume>50</volume>, <fpage>1829</fpage>&#x02013;<lpage>1842</lpage>. <pub-id pub-id-type="doi">10.1007/s40279-020-01321-9</pub-id><pub-id pub-id-type="pmid">32720230</pub-id></citation></ref>
<ref id="B27">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Langer</surname> <given-names>L.</given-names></name> <name><surname>Levy</surname> <given-names>C.</given-names></name> <name><surname>Bayley</surname> <given-names>M.</given-names></name></person-group> (<year>2020</year>). <article-title>Increasing incidence of concussion: true epidemic or better recognition?</article-title> <source>J. Head Trauma Rehabil.</source> <volume>35</volume>, <fpage>E60</fpage>&#x02013;<lpage>E66</lpage>. <pub-id pub-id-type="doi">10.1097/HTR.0000000000000503</pub-id><pub-id pub-id-type="pmid">31246881</pub-id></citation></ref>
<ref id="B28">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lumba-Brown</surname> <given-names>A.</given-names></name> <name><surname>Teramoto</surname> <given-names>M.</given-names></name> <name><surname>Bloom</surname> <given-names>O. J.</given-names></name> <name><surname>Brody</surname> <given-names>D.</given-names></name> <name><surname>Chesnutt</surname> <given-names>J.</given-names></name> <name><surname>Clugston</surname> <given-names>J. R.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Concussion guidelines step 2: evidence for subtype classification</article-title>. <source>Neurosurgery</source> <volume>86</volume>, <fpage>2</fpage>&#x02013;<lpage>13</lpage>. <pub-id pub-id-type="doi">10.1093/neuros/nyz332</pub-id><pub-id pub-id-type="pmid">31432081</pub-id></citation></ref>
<ref id="B29">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>McCrory</surname> <given-names>P.</given-names></name> <name><surname>Meeuwisse</surname> <given-names>W.</given-names></name> <name><surname>Dvorak</surname> <given-names>J.</given-names></name> <name><surname>Aubry</surname> <given-names>M.</given-names></name> <name><surname>Bailes</surname> <given-names>J.</given-names></name> <name><surname>Broglio</surname> <given-names>S.</given-names></name> <etal/></person-group>. (<year>2017</year>). <article-title>Consensus statement on concussion in sport&#x02014;the 5th international conference on concussion in sport held in Berlin, October 2016</article-title>. <source>Br. J. Sports Med.</source> <volume>51</volume>, <fpage>838</fpage>&#x02013;<lpage>847</lpage>. <pub-id pub-id-type="doi">10.1136/bjsports-2017-097699</pub-id><pub-id pub-id-type="pmid">28446457</pub-id></citation></ref>
<ref id="B30">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Nakata</surname> <given-names>N.</given-names></name></person-group> (<year>2019</year>). <article-title>Recent technical development of artificial intelligence for diagnostic medical imaging</article-title>. <source>Jpn. J. Radiol.</source> <volume>37</volume>, <fpage>103</fpage>&#x02013;<lpage>108</lpage>. <pub-id pub-id-type="doi">10.1007/s11604-018-0804-6</pub-id><pub-id pub-id-type="pmid">30706381</pub-id></citation></ref>
<ref id="B31">
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Permenter</surname> <given-names>C. M.</given-names></name> <name><surname>Fern&#x000E1;ndez-de Thomas</surname> <given-names>R. J.</given-names></name> <name><surname>Sherman</surname> <given-names>A.</given-names></name></person-group> (<year>2021</year>). <article-title>Postconcussive syndrome</article-title>, in <source>StatPearls</source> (<publisher-loc>Treasure Island, FL</publisher-loc>: <publisher-name>StatPearls Publishing</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>11</lpage>. Available online at: <ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/books/NBK534786/">http://www.ncbi.nlm.nih.gov/books/NBK534786/</ext-link> (accessed May 25, 2021).</citation>
</ref>
<ref id="B32">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ranstam</surname> <given-names>J.</given-names></name></person-group> (<year>2016</year>). <article-title>Multiple P-values and Bonferroni correction</article-title>. <source>Osteoarthr. Cartil.</source> <volume>24</volume>, <fpage>763</fpage>&#x02013;<lpage>764</lpage>. <pub-id pub-id-type="doi">10.1016/j.joca.2016.01.008</pub-id><pub-id pub-id-type="pmid">26802548</pub-id></citation></ref>
<ref id="B33">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rao</surname> <given-names>D. P.</given-names></name> <name><surname>McFaull</surname> <given-names>S.</given-names></name> <name><surname>Thompson</surname> <given-names>W.</given-names></name> <name><surname>Jayaraman</surname> <given-names>G. C.</given-names></name></person-group> (<year>2017</year>). <article-title>Trends in self-reported traumatic brain injury among Canadians, 2005-2014: a repeated cross-sectional analysis</article-title>. <source>CMAJ Open</source> <volume>5</volume>, <fpage>E301</fpage>&#x02013;<lpage>E307</lpage>. <pub-id pub-id-type="doi">10.9778/cmajo.20160115</pub-id></citation>
</ref>
<ref id="B34">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Schatz</surname> <given-names>P.</given-names></name> <name><surname>Pardini</surname> <given-names>J. E.</given-names></name> <name><surname>Lovell</surname> <given-names>M. R.</given-names></name> <name><surname>Collins</surname> <given-names>M. W.</given-names></name> <name><surname>Podell</surname> <given-names>K.</given-names></name></person-group> (<year>2006</year>). <article-title>Sensitivity and specificity of the ImPACT Test Battery for concussion in athletes</article-title>. <source>Arch. Clin. Neuropsychol.</source> <volume>21</volume>, <fpage>91</fpage>&#x02013;<lpage>99</lpage>. <pub-id pub-id-type="doi">10.1016/j.acn.2005.08.001</pub-id><pub-id pub-id-type="pmid">16143492</pub-id></citation></ref>
<ref id="B35">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Schneider</surname> <given-names>K. J.</given-names></name> <name><surname>Emery</surname> <given-names>C. A.</given-names></name> <name><surname>Black</surname> <given-names>A.</given-names></name> <name><surname>Yeates</surname> <given-names>K. O.</given-names></name> <name><surname>Debert</surname> <given-names>C. T.</given-names></name> <name><surname>Lun</surname> <given-names>V.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Adapting the dynamic, recursive model of sport injury to concussion: an individualized approach to concussion prevention, detection, assessment, and treatment</article-title>. <source>J. Orthop. Sports Phys. Ther.</source> <volume>49</volume>, <fpage>799</fpage>&#x02013;<lpage>810</lpage>. <pub-id pub-id-type="doi">10.2519/jospt.2019.8926</pub-id><pub-id pub-id-type="pmid">31610760</pub-id></citation></ref>
<ref id="B36">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Schober</surname> <given-names>P.</given-names></name> <name><surname>Boer</surname> <given-names>C.</given-names></name> <name><surname>Schwarte</surname> <given-names>L. A.</given-names></name></person-group> (<year>2018</year>). <article-title>Correlation coefficients: appropriate use and interpretation</article-title>. <source>Anesth. Analg.</source> <volume>126</volume>, <fpage>1763</fpage>&#x02013;<lpage>1768</lpage>. <pub-id pub-id-type="doi">10.1213/ANE.0000000000002864</pub-id><pub-id pub-id-type="pmid">29481436</pub-id></citation></ref>
<ref id="B37">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Stevens</surname> <given-names>E.</given-names></name> <name><surname>Dixon</surname> <given-names>D. R.</given-names></name> <name><surname>Novack</surname> <given-names>M. N.</given-names></name> <name><surname>Granpeesheh</surname> <given-names>D.</given-names></name> <name><surname>Smith</surname> <given-names>T.</given-names></name> <name><surname>Linstead</surname> <given-names>E.</given-names></name></person-group> (<year>2019</year>). <article-title>Identification and analysis of behavioral phenotypes in autism spectrum disorder via unsupervised machine learning</article-title>. <source>Int. J. Med. Inform.</source> <volume>129</volume>, <fpage>29</fpage>&#x02013;<lpage>36</lpage>. <pub-id pub-id-type="doi">10.1016/j.ijmedinf.2019.05.006</pub-id><pub-id pub-id-type="pmid">31445269</pub-id></citation></ref>
<ref id="B38">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sullivan</surname> <given-names>M. J. L.</given-names></name> <name><surname>Bishop</surname> <given-names>S. R.</given-names></name> <name><surname>Pivik</surname> <given-names>J.</given-names></name></person-group> (<year>1995</year>). <article-title>The pain catastrophizing scale: development and validation</article-title>. <source>Psychol. Assess.</source> <volume>7</volume>, <fpage>524</fpage>&#x02013;<lpage>532</lpage>. <pub-id pub-id-type="doi">10.1037/1040-3590.7.4.524</pub-id><pub-id pub-id-type="pmid">28616005</pub-id></citation></ref>
<ref id="B39">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Van Kampen</surname> <given-names>D. A.</given-names></name> <name><surname>Lovell</surname> <given-names>M. R.</given-names></name> <name><surname>Pardini</surname> <given-names>J. E.</given-names></name> <name><surname>Collins</surname> <given-names>M. W.</given-names></name> <name><surname>Fu</surname> <given-names>F. H.</given-names></name></person-group> (<year>2006</year>). <article-title>The &#x0201C;Value Added&#x0201D; of neurocognitive testing after sports-related concussion</article-title>. <source>Am. J. Sports Med.</source> <volume>34</volume>, <fpage>1630</fpage>&#x02013;<lpage>1635</lpage>. <pub-id pub-id-type="doi">10.1177/0363546506288677</pub-id><pub-id pub-id-type="pmid">16816151</pub-id></citation></ref>
<ref id="B40">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Visscher</surname> <given-names>R. M. S.</given-names></name> <name><surname>Feddermann-Demont</surname> <given-names>N.</given-names></name> <name><surname>Romano</surname> <given-names>F.</given-names></name> <name><surname>Straumann</surname> <given-names>D.</given-names></name> <name><surname>Bertolini</surname> <given-names>G.</given-names></name></person-group> (<year>2019</year>). <article-title>Artificial intelligence for understanding concussion: Retrospective cluster analysis on the balance and vestibular diagnostic data of concussion patients</article-title>. <source>PLoS ONE</source> <volume>14</volume>:<fpage>e0214525</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pone.0214525</pub-id><pub-id pub-id-type="pmid">30939164</pub-id></citation></ref>
</ref-list> 
</back>
</article>