The Italian telephone-based Verbal Fluency Battery (t-VFB): standardization and preliminary clinical usability evidence

Abstract

Background:

This study aimed at standardizing and providing preliminary evidence on the clinical usability of the Italian telephone-based Verbal Fluency Battery (t-VFB), which includes phonemic (t-PVF), semantic (t-SVF) and alternate (t-AVF) verbal fluency tasks.

Methods:

Three-hundred and thirty-five Italian healthy participants (HPs; 140 males; age range = 18–96 years; education range = 4–23 years) and 27 individuals with neurodegenerative or cerebrovascular diseases were administered the t-VFB. Switch number and cluster size were computed via latent semantic analyses. HPs underwent the telephone-based Mental State Examination (MMSE) and Backward Digit Span (BDS). Construct validity, factorial structure, internal consistency, test-retest and inter-rater reliability and equivalence with the in-person Verbal Fluency tasks were assessed. Norms were derived via Equivalent Scores. Diagnostic accuracy against clinical populations was assessed.

Results:

The majority of t-VFB scores correlated among each other and with the BDS, but not with the MMSE. Switch number correlated with t-PVF, t-SVF, t-AVF scores, whilst cluster size with the t-SVF and t-AVF scores only. The t-VFB was underpinned by a mono-component structure and was internally consistent (Cronbach’s α = 0.91). Test-retest (ICC = 0.69–0.95) and inter-rater reliability (ICC = 0.98–1) were optimal. Each t-VFB test was statistically equivalent to its in-person version (equivalence bounds yielding a p < 0.05). Education predicted all t-VFB scores, whereas age t-SVF and t-AVF scores and sex only some t-SVF scores. Diagnostic accuracy against clinical samples was optimal (AUC = 0.81–0.86).

Discussion:

The t-VFB is a valid, reliable and normed telephone-based assessment tool for language and executive functioning, equivalent to the in-person version; results show promising evidence of its diagnostic accuracy in neurological populations.

Introduction

Verbal fluency tasks are renowned measures of both language (i.e., lexical retrieval) and executive functioning (i.e., executive control over selective attention, inhibition, set-shifting and self-monitoring). These tests usually require to produce, within a given time (usually of 60 s), as many words as possible starting with a given letter (phonemic verbal fluency, PFV) or belonging to a given category (semantic verbal fluency, SVF) (Whiteside et al., 2016; Aita et al., 2018). There is also an alternate verbal fluency test (AVF), which requires to generate words by continuously alternating between phonemically and semantically cued ones, thus assessing set-shifting abilities (Costa et al., 2014).

As not requiring either visual or physical supports, verbal fluency tasks have been widely adopted within telephone-based cognitive assessment (Rapp et al., 2012; Bunker et al., 2017; Marceaux et al., 2019), both as screening measures and within comprehensive neuropsychological batteries (Carlew et al., 2020).

Telephone-based cognitive assessment is relevant to clinical (e.g., for not-viable/-preferable access to clinics due to logistical, geographical, economical or health safety reasons; Christodoulou et al., 2016; Caze et al., 2020; De Cola et al., 2020; Soldati et al., 2021) and experimental telemedicine (e.g., population-based/epidemiological studies, decentralized clinical trials and prevention campaigns; Crooks et al., 2005; Yaari et al., 2006; Herr and Ankri, 2013). However, function-specific telephone-based tests often do not meet rigorous statistical-methodological standards, especially within the Italian context (Zanin et al., 2022). Moreover, so far, standardized telephone-based cognitive assessment tools in Italy are limited to screening tests for the assessment of global cognitive status (Aiello et al., 2022a,d,e).

Telephone-based adaptations of existing tests require ad hoc norms, psychometrics and diagnostics even if their administration and scoring are comparable to their in-person versions, since unknown sources of systematic error variance may affect scores due to the different administration setting (Parlar et al., 2020; Fox-Fuller et al., 2021; Postal et al., 2021; Zanin et al., 2022). More specifically, threats to the equivalence between in-person and telephone-based measures have been identified in a lack of control over the testing environment (e.g., due to sources of distractions or facilitations), technical issues (e.g., telephone line instability) and interpersonal aspects (e.g., the absence of a face-to-face interaction), all these factors being leading confounders especially in examinees with sensory-motor, cognitive or behavioral disorders (Martin-Khan et al., 2010; Binng et al., 2020; Booth et al., 2021; Hunter et al., 2021). Moreover, the lack of ad hoc standardizations has been highlighted also by clinicians as a barrier to remote cognitive assessment procedures (Rochette et al., 2021).

Such issues likewise apply to telephone-based verbal fluency tasks, which have been recognized as not completely comparable to in-person ones (Hunter et al., 2021). Nevertheless, no ad hoc standardizations of telephone-based verbal fluency tasks have been provided so far.

Given the above premises, the present study aimed at standardizing, for the Italian population, a telephone-based Verbal Fluency Battery (t-VFB) comprising PVF, SVF, and AVF tasks, as well as at preliminarily assessing its clinical usability in individuals with cerebrovascular or neurodegenerative diseases.

Materials and methods

Participants

Three-hundred and thirty-five Italian healthy participants (HPs) from different regions of Italy represented the normative sample. HPs were recruited through both Authors’ personal acquaintances and advertising at the University of Milano-Bicocca and University of Padova. HPs had no history of (1) neurological/psychiatric disorders, (2) active psychotropic medications, (3) uncompensated/severe general-medical conditions and (4) uncorrected hearing deficits.

Twenty-seven outpatients with neurological diseases were consecutively recruited at two neuropsychology services in Northern Italy. A clinical neurological diagnosis formulated by a neurologist constituted the inclusion criteria for clinical groups. Diagnoses were posed according to current diagnostic criteria and based on neurological, neuroradiological and neuropsychological examinations. Exclusion criteria for patients were: (1) severe medical-general conditions in the acute phase; (2) severe behavioral impairment that would have undermined compliance; (3) uncorrected hearing deficits. Ten participants had ischemic/hemorrhagic stroke causing unilateral, cortical or subcortical hemispheric lesion (5 with right-sided hemispheric lesion, and 5 with left-sided lesion). Six individuals presented with hypokinetic, extra-pyramidal disorders: 3 had Parkinson’s disease (Postuma et al., 2015), whereas 3 had atypical parkinsonisms (Levin et al., 2016). Six individuals had small vessel disease (Shi and Wardlaw, 2016). Four had a mixed, atrophic-vascular dementia (Zekry et al., 2002). Diagnoses of stroke, small vessel disease and mixed dementia were supported by computerized tomography or magnetic resonance imaging, whereas those of extra-pyramidal disorders by single-photon emission computerized tomography of the dopamine transporter. All patients underwent, for clinical purposes, a neuropsychological battery comprising measures of overall cognitive efficiency (Mini-Mental State Examination (Measso et al., 1993), attention (digit cancellation test, Spinnler and Tognoni, 1987; Trail-Making Test, Siciliano et al., 2019), executive functions (Frontal Assessment Battery Appollonio et al., 2005; Raven Colored Progressive Matrices Basso et al., 1987; phonemic and alternate verbal fluency Costa et al., 2014; Stroop test Caffarra et al., 2002), language (Token Test, Spinnler and Tognoni, 1987; Boston Naming Test, Goodglass et al., 1983; semantic verbal fluency, Costa et al., 2014), verbal and visuo-spatial memory (Babcock test, Novelli et al., 1986; forward/backward digit and Corsi span, Monaco et al., 2013) and visuo-spatial/constructional abilities (Clock Drawing Test, Caffarra et al., 2011; design copy, Spinnler and Tognoni, 1987). Additionally, stroke patients with a left-sided hemispheric lesion underwent a comprehensive language (Esame Neuropsicologico per l’Afasia; Capasso and Miceli, 2001) and limb apraxia assessment (ideomotor praxis, De Renzi et al., 1980; ideative praxis, De Renzi and Lucchelli, 1988), whereas in those with a right-sided damage extra-personal and personal hemispatial neglect was assessed (Bells test, Vallar et al., 1994; letter cancellation test, Vallar et al., 1994; Apple Cancellation Test, Mancuso et al., 2015; line bisection test, Nichelli et al., 1989; Fluff test, Cocchini et al., 2001; Comb and Razor test, Zoccolotti et al., 1992).

Medical history for all participants was collected through a semi-structured interview.

The study received ethical approval by the Committees of the University of Milano-Bicocca, Milano (ID: RM-2021-382, 19/02/2021), the University of Padova, Padua (ID: 4107, 19/02/2021) and IRCCS Istituto Auxologico Italiano, Milano (ID: 25C122, 18/05/2021). Informed consent was acquired from every participant. Data collection for HPs started in March 2021 and ended in January 2022, whereas that for clinical populations started in July 2021 and ended in January 2022.

Materials

The t-VFB comprises the telephone-based versions of 3 verbal fluency subtests: the t-PVF, t-SVF, and t-AVF subtests standardized by Costa et al.’s (2014), which were administered over the telephone. The t-PVF requires to generate as many words as possible beginning with letters “F,” “A,” and “S,” within a 60”-timespan each. The t-SVF require to generate semantic category exemplars: “Color,” “Animal,” and “Fruit” (60 s for each category). The t-AVF requires to continuously alternate letter-cued words with category-cued words as follows: “A/Color,” “F/Animal,” “S/Fruit”; again, each trial lasted 60 s. Following the administration procedure for the in-person version of Costa et al.’s (2014) battery, the three verbal fluency subtests were administered consecutively in the following order: t-PVF, t-SVF, and t-AVF; according to the original work, PVF and SVF tasks must precede the AVF subtest in order to calculate the Composite Shifting Index (CSI), i.e., a measure of the cost of shifting from phonemic/semantic (single-cued) to alternating (double-cued) fluency. As in the original work by Costa et al. (2014), participants were instructed not to produce proper nouns, place names, numbers or inflected words with the same suffix. For each trial, the number of words generated in 60 s was recorded. Performance score in each subtest is computed as the sum of the number of words generated in all trials belonging to that specific subtest. A t-CSI was also computed as follows: t-AVF/[(t-PVF + t-SVF)/2]. The t-CSI thus addresses the words generated in all three subtests, and it reflects the shifting cost for passing from the single fluency subtests to the alternate one.

The protocol of the t-VFB is reported in Supplementary Material 1. The t-VFB protocol was conceptualized and approved without reservation by an Author board comprising a neurologist (IA), six researchers with expertise in neuropsychology and psychometrics (SZ, SM, NB, ENA, VP, and LD).

To assess convergent validity, HPs were remotely administered the Italian telephone-based Mini-Mental State Examination (Itel-MMSE) (Metitieri et al., 2001; Aiello et al., 2022b) and a telephone version of the backward digit span (BDS) task (Monaco et al., 2013). The BDS encompasses both the longer sequence recalled, measuring working memory capacity (BDS-WM), and the total number of recalled sequences (BDS-T), measuring sustained attention (Pasotti et al., 2022).

Procedures

Before telephone-based testing, all participants underwent a detailed sound-check to ensure a good quality of the call, as well as and a brief training focused on those actions required for the executions of telephone-based tasks (Supplementary Material 2). For each participant, a caregiver/cohabitee was required to preliminarily ensure that no distractions or facilitations were present within the administration setting. The same person was asked to confirm the correctness of the address information provided by the participant during the spatial orientation task of the Itel-MMSE.

To test between-modality equivalence, a subgroup of 47 HPs were also administered in-person verbal fluency tasks either before (N = 23) or after (N = 24) a 48-h distance from telephonic testing, to rule out possible carry-over effects.

Eighteen HPs were re-tested with the t-VFB after 14 days from the baseline for test-retest reliability, whereas 27 protocols were simultaneously scored by two independent raters for inter-rater reliability.

All the above sub-samples for validity/reliability analyses were randomly selected from the whole normative sample.

Data were collected by either licensed or trainee psychologists who first underwent a thorough training.

Statistical analyses

SPSS 27 (IBM Corp., 2020), R 4.1.0¹ and jamovi 1.6.23² were adopted to analyze data.

Convergent validity was explored through either Pearson’s or Spearman correlations for normally and non-normally distributed data, respectively (skewness and kurtosis values < |1| and | 3|, respectively; Kim, 2013).

Intra-class correlations were adopted to examine test-retest and inter-rater reliability. Internal consistency and factorial structure were tested via Cronbach’s α and a Principal Component Analysis (PCA), respectively.

Verbal fluency performance relies on both language (i.e., lexical-semantic abilities) and executive functioning (Whiteside et al., 2016; Aita et al., 2018). According to Troyer et al. (1997), a valid measure of lexical-semantic integrity derived from verbal fluency is the number of circumscribable clusters of semantically-related words (cluster size), while a measure of executive functioning is the frequency of transitions between these clusters (number of switches). Accordingly, to provide further construct validity evidence, the number of switches and cluster size were computed relying on measures of semantic relatedness derived via a Latent Semantic Analysis, which is a natural language processing statistical model (Landauer and Dumais, 1997).

A two one-sided test (TOST) procedure for dependent samples (Lakens, 2017) was adopted for testing the equivalence between telephone-based and in-person verbal fluency tasks. The TOST procedure regards a between-mean effect size as equivalent to 0 if it falls within the upper and lower equivalence bounds and distances from them at p < 0.05.

Norms were derived through the Equivalent Score (ES) approach (Capitani and Laiacona, 2017; Aiello and Depaoli, 2022). Accordingly, clinical judgments were allowed by identifying outer and inner tolerance limits (oTL; iTL), as well as ES thresholds, on stepwise regression-adjusted scores. The ES scales allows to draw clinical judgments as follows: ES = 0 (adjusted scores ≤ oTL) → impaired; ES = 1→ borderline; ES = 2→ “low-end” normal; ES = 3/4→ normal.

Diagnostic accuracy of the t-VFB was tested via receiver-operating characteristics analyses by addressing scores of the whole clinical group against the whole normative sample.

Power analyses

The minimum sample size for reliability and validity analyses in HPs were set at N≈20 and N≈80, respectively, pursuantly to Hobart et al.’s (2012) recommendations on psychometric measurement in neurology.

According to qualitative guidelines delivered by Kyriazos (2018), a sample of 100 subjects was deemed as sufficient for running the PCA.

Sample size estimation for the TOST procedure, as run through the R package TOSTER³ (Lakens, 2017), yielded a minimum of N = 44 pairs with a 90% power, α = 0.05 and upper and lower equivalence bounds of -0.5 and 0.5, respectively.

For norm derivation, an estimated N = 287 was deemed sufficient to detect a small-to-medium effect size (f² = 0.05) with a 90% power and a type-I error rate of 5% within a multiple regression model (df_numerator = 3) through the R package pwr⁴.

The minimum sample sizes for the normative and clinical groups for diagnostic accuracy analyses were set at N = 19 and N = 190, respectively, in accordance with Obuchowski’s (2005) procedures for a single-test ROC analysis and via easyROC⁵, by addressing a case-control allocation ratio of 10, AUC = 0.7, 1-β = 0.9 and α = 0.05.

Results

Table 1 shows the stratification of the normative sample, and Table 2 summarizes HPs’ demographic data and test scores.

Acceptability rate was 100%.

Convergence against the other telephone-based measures (Itel-MMSE and BDS) is reported in Supplementary Table 1. At α_adjusted = 0.001, the vast majority of t-VFB measures were associated with BDS-T and BDS-WM scores; by contrast, the Itel-MMSE was only associated with the t-AVF and t-CSI, but not to t-PVF and t-SVF scores.

When putting into relation all t-VFB measures among each other, significant, moderate-to-high correlations were detected [0.33 ≤ r_s(335) ≤ 0.92; all ps < 0.001] among t-PVF, t-SVF, and t-AVF, whereas, as to the t-CSI, significant associations where only detected with t-AVF scores [r_s(335) ≥ 0.54; all ps < 0.001]. At α_adjusted = 0.006, the number of switches was strongly related with total t-PVF, t-SVF and t-AVF scores [0.71 ≤ r_s(138) ≤ 0.74; all ps < 0.001], whereas cluster size was related to the t-SVF and t-AVF [both r_s(138) = -0.32, p < 0.001], but not to t-PVF [r_s(138) = -0.16, p = 0.067]. The t-CSI was not associated with either the number switches or cluster size (p ≥ 0.009).

The PCA revealed a mono-component structure underpinning t-PVF, t-SVF, and t-AVF scores (59.7% of variance explained; loading range = 0.65–0.83), here named “linguistically mediated executive functioning.” Internal consistency for such scores was excellent (Cronbach’s α = 0.91).

Inter-rater reliability was excellent for all t-VFB total scores: t-PVF: ICC = 1; t-SVF: ICC = 0.98; t-AVF: ICC = 0.98. Test-retest reliability was optimal for the t-PVF (ICC = 0.95) and t-AVF (ICC = 0.89), and good for the t-SVF (ICC = 0.69).

The TOST procedure revealed across-modality equivalence for all verbal fluency total scores: PVF: telephone-based: 42.6 ± 11.2 vs. in-person 41.7 ± 11, t(46) = 0.63, p = 0.533, upper and lower equivalence bounds both yielding a p ≤ 0.01; SVF: telephone-based: 55.9 ± 11.6 vs. in-person 53.7 ± 11.7, t(46) = -1.45, p = 0.154, upper and lower equivalence bounds both yielding a p < 0.05; AVF: telephone-based: 43.5 ± 11.1 vs. in-person 43.5 ± 12, t(46) = -0.01, p = 0.988, upper and lower equivalence bounds both yielding a p ≤ 0.001; CSI: telephone-based:0.89 ± 0.17 vs. in-person 0.91 ± 0.2, t(46) = 0.68, p = 0.501, upper and lower equivalence bounds both yielding a p ≤ 0.01.

Education predicted all t-VFB measures (ps ≤ 0.029), age the t-SVF, t-AVF and t-CSI (p ≤ 0.001), whereas sex predicted only t-SVF-colors (p = 0.006) and t-SVF-fruits scores (p < 0.001), with females performing better than males. Adjustment equations, TLS and ES thresholds are reported in Table 3; adjustment grids are reported in Supplementary Tables 2–4, whereas an automated adjusted sheet in Supplementary Material 3.

Table 4 reports demographic data and test scores of individuals with neurological diseases. When comparing the whole clinical group against the normative sample, each t-VFB task yielded excellent accuracy: t-PVF: AUC = 0.81, SE = 0.05, CI 95% [0.71, 0.91]; t-SVF: AUC = 0.81, SE = 0.04, CI 95% [0.73, 0.9]; t-AVF: AUC = 0.86, SE = 0.03, CI 95% [0.8, 0.93].

Discussion

The present study provides Italian practitioners and researchers with a standardized, telephone-based set of verbal fluency tests (t-VFB), along with clinical usability evidence in participants affected with neurological conditions of different etiologies. As providing modality-specific norms, psychometrics and diagnostics of verbal fluency tasks to be administered over the telephone, this work is unprecedented both within the Italian and international literature, enriching the range of telephone-based cognitive tests available in Italy (Aiello et al., 2022a,d,c) by meeting statistical-methodological requirements suggested for telephone-based tools-related standardizations (Zanin et al., 2022). In this last respect, it is remarkable that the present telephone-based version of verbal fluency tests showed to be statistically equivalent to the in-person versions by Costa et al. (2014), this supporting, within longitudinal assessments, a flexible use of the t-VFB in combination with the in-person version normed by Costa et al. (2014). However, this finding should not lead users to address, for the t-VFB, the same norms, psychometrics and diagnostics of in-person verbal fluency tests. Indeed, in order to avoid distortion in test scores, such information needs to be derived through ad hoc standardization studies specific to the telephonic modality, consistently to international guidelines on remote cognitive testing (Postal et al., 2021; Zanin et al., 2022). Users of the t-VFB could nonetheless safely compare, within longitudinal assessments, the ESs yielded from its administration to those yielded by the administration of Costa et al.’s (2014) battery, since the 5-points ES scale allows to compare score at different tests (Capitani and Laiacona, 2017; Aiello and Depaoli, 2022).

As coming with norms separately for each task and subtask, the t-VFB represents a flexible tool as to its administration. Moreover, the t-VFB presents with a solid factorial structure, optimal construct validity, internal consistency, test-retest, and inter-rater reliability. This work also provides promising, albeit preliminary, diagnostic accuracy evidence of the t-VFB in individuals with cerebrovascular and neurodegenerative diseases. The construct validity of the t-VFB is also supported by its association with renowned, fine-grained measures of both language- and executive-related verbal fluency underpinnings, i.e., the cluster size and the number of switches, respectively (Troyer et al., 1997). Indeed, all t-VFB measures were related to the number of switches, i.e., the executive component common to all verbal fluency tasks (Whiteside et al., 2016), whereas only the t-SVF and t-AVF, which rely more on language functions as compared to PVF, correlated with cluster size, that is a proper language measure reflecting the integrity of the lexical-semantic component (Shao et al., 2014; Aita et al., 2018).

Data on demographic effects are overall in line with those reported by Costa et al. (2014): t-SVF, t-AVF, and t-CSI scores were predicted by both age and education, whereas the t-PVF only by the latter. However, in Costa et al. (2014), sex predicted all PVF, SVF, and AVF scores, whilst, within the present work, it was only predictive of certain t-SVF subtest scores. This discrepancy is surprising, considering that the samples of both studies had a similar number of females and males, i.e., 56% of females in Costa et al. (2014), whereas 58% in the present study. Sex differences in healthy individuals on verbal fluency have been previously reported, although current evidence is still unclear and varied from report to report (Heister, 1982; Gauthier et al., 2009). Nevertheless, previous findings appear to sufficiently converge as to a slight female advantage at least for PVF (Capitani et al., 1998).

As to the limitations, it should be noted that the present findings about the clinical usability of the t-VFB refer to a small sample of participants with neurological diseases of different etiology. Future studies should extend and verify the present results in larger and homogeneous clinical cohorts featured by executive and/or language disorders that could influence verbal fluency performance (Suárez-González et al., 2021). Within such clinical usability studies, post-test probabilities (positive and negative predictive values and likelihood ratios), responsiveness and reliable change need also to be assessed (Aiello et al., 2022c). Moreover, equivalence of the t-VFB with its in-person version needs to be documented also in neurological populations.

In conclusions, the t-VFB shows to be a valid, reliable, diagnostically accurate and normed telephone-based cognitive tool for the assessment of linguistically mediated executive functions. The t-VFB is equivalent to in-person verbal fluency tasks and shows promising evidence of clinical usability in individuals with cerebrovascular or neurodegenerative diseases.

References

• 1

  AielloE. N.DepaoliE. G. (2022). Norms and standardizations in neuropsychology via equivalent scores: software solutions and practical guides.Neurol. Sci.43961–966. 10.1007/s10072-021-05374-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  AielloE. N.EspositoA.GiannoneI.DianaL.AppollonioI.BologniniN. (2022a). Telephone Interview for Cognitive Status (TICS): Italian adaptation, psychometrics and diagnostics.Neurol. Sci.433071–3077. 10.1007/s10072-021-05729-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  AielloE. N.EspositoA.GiannoneI.DianaL.WoolleyS.MurphyJ.et al (2022d). ALS Cognitive Behavioral Screen-Phone Version (ALS-CBS™-PhV): norms, psychometrics, and diagnostics in an Italian population sample.Neurol. Sci.432571–2578. 10.1007/s10072-021-05636-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  AielloE. N.PucciV.DianaL.NiangA.PretiA. N.Delli PontiA.et al (2022e). Telephone-based Frontal Assessment Battery (t-FAB): standardization for the Italian population and clinical usability in neurological diseases.Aging Clin. Exp. Res.341635–1644. 10.1007/s40520-022-02155-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  AielloE. N.EspositoA.PucciV.MondiniS.BologniniN.AppollonioI. (2022b). Italian telephone-based Mini-Mental State Examination (Itel-MMSE): item-level psychometric properties.Aging Clin. Exp. Res.341259–1265. 10.1007/s40520-021-02041-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  AielloE. N.RimoldiS.BologniniN.AppollonioI.ArcaraG. (2022c). Psychometrics and diagnostics of Italian cognitive screening tests: a systematic review.Neurol. Sci.43821–845. 10.1007/s10072-021-05683-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  AitaS. L.BeachJ. D.TaylorS. E.BorgognaN. C.HarrellM. N.HillB. D. (2018). Executive, language, or both? An examination of the construct validity of verbal fluency measures.Appl. Neuropsychol.26441–451. 10.1080/23279095.2018.1439830

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  AppollonioI.LeoneM.IsellaV.PiamartaF.ConsoliT.VillaM. L.et al (2005). The Frontal Assessment Battery (FAB): normative values in an Italian population sample.Neurol. Sci.26108–116. 10.1007/s10072-005-0443-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  BassoA.CapitaniE.LaiaconaM. (1987). Raven’s coloured progressive matrices: normative values on 305 adult normal controls.Funct. Neurol.2189–194.

  • Google Scholar
  • View reference in article

• 10

  BinngD.SplonskowskiM.JacovaC. (2020). Distance assessment for detecting cognitive impairment in older adults: a systematic review of psychometric evidence.Dement. Geriatr. Cogn. Disord.49456–470. 10.1159/000511945

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  BoothT.MurrayA.Muniz-TerreraG. (2021). Are we measuring the same thing? Psychometric and research considerations when adopting new testing modes in the time of COVID-19.Alzheimers Dement.17251–254. 10.1002/alz.12197

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  BunkerL.HshiehT. T.WongB.SchmittE. M.TravisonT.YeeJ.et al (2017). The SAGES telephone neuropsychological battery: correlation with in-person measures.Int. J. Geriatr. Psychiatry32991–999. 10.1002/gps.4558

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  CaffarraP.GardiniS.ZonatoF.ConcariL.DieciF.CopelliS.et al (2011). Italian norms for the Freedman version of the Clock Drawing Test.J. Clin. Exp. Neuropsychol.33982–988. 10.1080/13803395.2011.589373

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  CaffarraP.VezzadiniG.DieciF.ZonatoF.VenneriA. (2002). Una versione abbreviata del test di Stroop: dati normativi nella popolazione italiana.Nuova Riv. Neurol.12111–115.

  • Google Scholar
  • View reference in article

• 15

  CapassoR.MiceliG. (2001). Esame Neuropsicologico per l’Afasia.Berlin: Springer Science.

  • Google Scholar
  • View reference in article

• 16

  CapitaniE.LaiaconaM. (2017). Outer and inner tolerance limits: their usefulness for the construction of norms and the standardization of neuropsychological tests.Clin. Neuropsychol.311219–1230. 10.1080/13854046.2017.1334830

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  CapitaniE.LaiaconaM.BassoA. (1998). Phonetically cued word-fluency, gender differences and aging: A reappraisal.Cortex34779–783. 10.1016/S0010-9452(08)70781-0

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  CarlewA. R.FatimaH.LivingstoneJ. R.ReeseC.LacritzL.PendergrassC.et al (2020). Cognitive assessment via telephone: a scoping review of instruments.Arch. Clin. Neuropsychol.351215–1233. 10.1093/arclin/acaa096

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  CazeT.DorsmanK. A.CarlewA. R.DiazA.BaileyK. C. (2020). Can You Hear Me Now? Telephone-Based Teleneuropsychology Improves Utilization Rates in Underserved Populations.Arch. Clin. Neuropsychol.351234–1239. 10.1093/arclin/acaa098

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  ChristodoulouG.GenningsC.HupfJ.Factor-LitvakP.MurphyJ.GoetzR. R.et al (2016). Telephone based cognitive-behavioral screening for frontotemporal changes in patients with amyotrophic lateral sclerosis (ALS).Amyotroph. Later. Scler. Frontotemp. Degenerat.17482–488. 10.3109/21678421.2016.1173703

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  CocchiniG.BeschinN.JehkonenM. (2001). The Fluff Test: A simple task to assess body representation neglect.Neuropsychol. Rehabil.1117–31. 10.1080/09602010042000132

  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  CostaA.BagojE.MonacoM.ZabberoniS.De RosaS.PapantonioA. M.et al (2014). Standardization and normative data obtained in the Italian population for a new verbal fluency instrument, the phonemic/semantic alternate fluency test.Neurol. Sci.35365–372. 10.1007/s10072-013-1520-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  CrooksV. C.ClarkL.PetittiD. B.ChuiH.ChiuV. (2005). Validation of multi-stage telephone-based identification of cognitive impairment and dementia.BMC Neurology5:8. 10.1186/1471-2377-5-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  De ColaM. C.TrigliaG.CameraM.CoralloF.Di CaraM.BramantiP.et al (2020). Effect of neurological screening on early dementia detection in southern Italy.J. Int. Med. Res.48:0300060520949763. 10.1177/0300060520949763

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  De RenziE.LucchelliF. (1988). Ideational apraxia.Brain1111173–1185. 10.1093/brain/111.5.1173

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  De RenziE.MottiF.NichelliP. (1980). Imitating gestures: a quantitative approach to ideomotor apraxia.Arch. Neurol.376–10. 10.1001/archneur.1980.00500500036003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 27

  Fox-FullerJ. T.RizerS.AndersenS. L.SunderaramanP. (2021). Survey Findings About the Experiences, Challenges, and Practical Advice/Solutions Regarding Teleneuropsychological Assessment in Adults.Arch. Clin. Neuropsychol.37274–291. 10.1093/arclin/acab076

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  GauthierC. T.DuymeM.ZancaM.CapronC. (2009). Sex and performance level effects on brain activation during a verbal fluency task: a functional magnetic resonance imaging study.Cortex45164–176. 10.1016/j.cortex.2007.09.006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  GoodglassH.KaplanE.WeintraubS. (1983). Boston naming test.Philadelphia, PA: Lea and Febiger.

  • Google Scholar
  • View reference in article

• 30

  HeisterG. (1982). Sex differences in verbal fluency: a short note.Curr. Psychol.2257–260. 10.1007/BF03186768

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  HerrM.AnkriJ. (2013). A critical review of the use of telephone tests to identify cognitive impairment in epidemiology and clinical research.J. Telemed. Telecare1945–54. 10.1177/1357633X12474962

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  HobartJ. C.CanoS. J.WarnerT. T.ThompsonA. J. (2012). What sample sizes for reliability and validity studies in neurology?J. Neurol.2592681–2694. 10.1007/s00415-012-6570-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  HunterM. B.JenkinsN.DolanC.PullenH.RitchieC.Muniz-TerreraG. (2021). Reliability of telephone and videoconference methods of cognitive assessment in older adults with and without dementia.J. Alzheimers Dis.811625–1647. 10.3233/JAD-210088

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  KimH. Y. (2013). Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis.Restorat. Dentist. Endodont.3852–54. 10.5395/rde.2013.38.1.52

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  KyriazosT. A. (2018). Applied psychometrics: sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general.Psychology9:2207. 10.4236/psych.2018.98126

  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  LakensD. (2017). Equivalence tests: a practical primer for t tests, correlations, and meta-analyses.Soc. Psychol. Pers. Sci.8355–362. 10.1177/1948550617697177

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  LandauerT. K.DumaisS. T. (1997). A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge.Psychol. Rev.104:211. 10.1037/0033-295X.104.2.211

  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  LevinJ.KurzA.ArzbergerT.GieseA.HöglingerG. U. (2016). The differential diagnosis and treatment of atypical parkinsonism.Dtsch. Ärztebl. Int.11361–69. 10.3238/arztebl.2016.0061

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  MancusoM.RosadoniS.CapitaniD.BickertonW. L.HumphreysG. W.De TantiA.et al (2015). Italian standardization of the apples cancellation test.Neurol. Sci.361233–1240. 10.1007/s10072-015-2088-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  MarceauxJ. C.ProsjeM. A.McClureL. A.KanaB.CroweM.KisselaB.et al (2019). Verbal fluency in a national sample: telephone administration methods.Int. J. Geriatr. Psychiatry34578–587. 10.1002/gps.5054

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  Martin-KhanM.WoottonR.GrayL. (2010). A systematic review of the reliability of screening for cognitive impairment in older adults by use of standardised assessment tools administered via the telephone.J. Telemed. Telecare16422–428. 10.1258/jtt.2010.100209

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  MeassoG.CavarzeranF.ZappalàG.LebowitzB. D.CrookT. H.PirozzoloF. J.et al (1993). The mini-mental state examination: normative study of an Italian random sample.Dev. Neuropsychol.977–85. 10.1080/87565649109540545

  • CrossRef
  • Google Scholar
  • View reference in article

• 43

  MetitieriT.GeroldiC.PezziniA.FrisoniG. B.BianchettiA.TrabucchiM. (2001). The Itel-MMSE: an Italian telephone version of the Mini-Mental State Examination. Int. J. Geriatr. Psychiatry.16, 166–167. 10.1002/1099-1166(200102)16:2<166::aid-gps290>3.0.co;2-m

  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  MonacoM.CostaA.CaltagironeC.CarlesimoG. A. (2013). Forward and backward span for verbal and visuo-spatial data: standardization and normative data from an Italian adult population.Neurol. Sci.34749–754. 10.1007/s10072-012-1130-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 45

  NichelliP.RinaldiM.CubelliR. (1989). Selective spatial attention and length representation in normal subjects and in patients with unilateral spatial neglect.Brain Cogn.957–70. 10.1016/0278-2626(89)90044-4

  • CrossRef
  • Google Scholar
  • View reference in article

• 46

  NovelliG.PapagnoC.CapitaniE.LaiaconaM. (1986). Tre test clinici di memoria verbale a lungo termine: taratura su soggetti normali.Arch. Psicol. Neurol. Psichiatr.47278–296.

  • Google Scholar
  • View reference in article

• 47

  ObuchowskiN. A. (2005). ROC analysis.Am. J. Roentgenol.184364–372. 10.2214/ajr.184.2.01840364

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 48

  ParlarM. E.SpilkaM. J.Wong GonzalezD.BallantyneE. C.DoolC.GojmeracC.et al (2020). “You can’t touch this”: delivery of inpatient neuropsychological assessment in the era of COVID-19 and beyond.Clin. Neuropsychol.341395–1410. 10.1080/13854046.2020.1810324

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 49

  PasottiF.De LucaG.AielloE. N.GramegnaC.Di GangiM.FoderaroG.et al (2022). A multi-component, adaptive Working Memory Assessment battery (WoMAB): validation and norms in an Italian population sample.Neurol. Sci.43958–992. 10.1007/s10072-021-05416-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 50

  PostalK. S.BilderR. M.LancaM.AaseD. M.BarisaM.HollandA. A.et al (2021). Inter Organizational Practice Committee guidance/recommendation for models of care during the novel coronavirus pandemic.Arch. Clin. Neuropsychol.3617–28. 10.1093/arclin/acaa073

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 51

  PostumaR. B.BergD.SternM.PoeweW.OlanowC. W.OertelW.et al (2015). MDS clinical diagnostic criteria for Parkinson’s disease.Move. Disord.301591–1601. 10.1002/mds.26424

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 52

  RappS. R.LegaultC.EspelandM. A.ResnickS. M.HoganP. E.CokerL. H.et al (2012). Validation of a cognitive assessment battery administered over the telephone.J. Am. Geriatr. Soc.601616–1623. 10.1111/j.1532-5415.2012.04111.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 53

  RochetteA. D.Rahman-FilipiakA.SpencerR. J.MarshallD.StelmokasJ. E. (2021). Teleneuropsychology practice survey during COVID-19 within the United States.Appl. Neuropsychol.10.1080/23279095.2021.1872576[Epub ahead of print].

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 54

  ShaoZ.JanseE.VisserK.MeyerA. S. (2014). What do verbal fluency tasks measure? Predictors of verbal fluency performance in older adults.Front. Psychol.5:772. 10.3389/fpsyg.2014.00772

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 55

  ShiY.WardlawJ. M. (2016). Update on cerebral small vessel disease: a dynamic whole-brain disease.Stroke Vasc. Neurol.183–92. 10.1136/svn-2016-000035

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 56

  SicilianoM.ChiorriC.BattiniV.Sant’EliaV.AltieriM.TrojanoL.et al (2019). Regression-based normative data and equivalent scores for Trail Making Test (TMT): an updated Italian normative study.Neurol. Sci.40469–477. 10.1007/s10072-018-3673-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 57

  SoldatiA. B.AlmeidaC.LimaM.AraujoA.Araujo-LeiteM. A.SilvaM. T. T. (2021). Telephone screening of cognitive status (TICS) in severe COVID-19 patients: Utility in the era of social isolation.Eneurologicalsci22:100322. 10.1016/j.ensci.2021.100322

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 58

  SpinnlerH.TognoniG. (1987). Standardizzazione e taratura italiana di test neuropsicologici.Neurological Sciences61–120.

  • Google Scholar
  • View reference in article

• 59

  Suárez-GonzálezA.CassaniA.GopalanR.StottJ.SavageS. (2021). When it is not primary progressive aphasia: a scoping review of spoken language impairment in other neurodegenerative dementias.Alzheimers Dement.7:e12205. 10.1002/trc2.12205

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 60

  TroyerA. K.MoscovitchM.WinocurG. (1997). Clustering and switching as two components of verbal fluency: evidence from younger and older healthy adults. Neuropsychology11, 138–146. 10.1037//0894-4105.11.1.138

  • CrossRef
  • Google Scholar
  • View reference in article

• 61

  VallarG.RusconiM. L.FontanaS.MusiccoM. (1994). Tre test di esplorazione visuo-spaziale: taratura su 212 soggetti normali.Arch. Psicol. Neurol. Psichiatria55827–841.

  • Google Scholar
  • View reference in article

• 62

  WhitesideD. M.KealeyT.SemlaM.LuuH.RiceL.BassoM. R.et al (2016). Verbal fluency: Language or executive function measure?Appl. Neuropsychol.2329–34. 10.1080/23279095.2015.1004574

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 63

  YaariR.FleisherA. S.GamstA. C.BagwellV. P.ThalL. J. (2006). Utility of the telephone interview for cognitive status for enrollment in clinical trials.Alzheimers Dement.2104–109. 10.1016/j.jalz.2006.02.004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 64

  ZaninE.AielloE. N.DianaL.FusiG.BonatoM.NiangA.et al (2022). Tele-neuropsychological assessment tools in Italy: a systematic review on psychometric properties and usability.Neurol. Sci.43125–138. 10.1007/s10072-021-05719-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 65

  ZekryD.HauwJ. J.GoldG. (2002). Mixed dementia: epidemiology, diagnosis, and treatment.J. Am. Geriatr. Soc.501431–1143. 10.1046/j.1532-5415.2002.50367.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 66

  ZoccolottiP.AntonucciG.JudicaA. (1992). Psychometric characteristics of two semi-structured scales for the functional evaluation of hemi-inattention in extrapersonal and personal space.Neuropsychol. Rehabil.2179–191. 10.1080/09602019208401407

  • CrossRef
  • Google Scholar
  • View reference in article