Impact Factor 2.089
2017 JCR, Clarivate Analytics 2018

The world's most-cited Multidisciplinary Psychology journal

This article is part of the Research Topic

Language and Mild Cognitive Impairment

Original Research ARTICLE Provisionally accepted The full-text will be published soon. Notify me

Front. Psychol. | doi: 10.3389/fpsyg.2019.01020

Automatic scoring of semantic fluency

  • 1School of Computing, Information and Communications University, South Korea
  • 2School of Informatics, University of Edinburgh, United Kingdom
  • 3Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, United Kingdom

In neuropsychological assessment, semantic fluency is a widely accepted measure of executive function and access to semantic memory. While fluency scores are typically reported as the number of unique words produced, several alternative manual scoring methods have been proposed that provide additional insights into performance, such as clusters of semantically related items. Many automatic scoring methods yield metrics that are difficult to relate to the theories behind manual scoring methods, and most require manually-curated linguistic ontologies or large corpus infrastructure. In this paper, we propose a novel automatic scoring method based on Wikipedia, Backlink-VSM, which is easily adaptable to any of the 61 languages with more than 100k Wikipedia entries, can account for cultural differences in semantic relatedness, and covers a wide range of item categories. Our Backlink-VSM method combines relational knowledge as represented by links between Wikipedia entries (Backlink model) with a semantic proximity metric derived from distributional representations (vector space model; VSM). Backlink-VSM yields measures that approximate manual clustering and switching analyses, providing a straightforward link to the substantial literature that uses these metrics. We illustrate our approach with examples from two languages (English and Korean), and two commonly used categories of items (animals and fruits). For both Korean and English, we show that the measures generated by our automatic scoring procedure correlate well with manual annotations. We also successfully replicate findings that older adults produce significantly fewer switches compared to younger adults. Furthermore, our automatic scoring procedure outperforms the manual scoring method and a WordNet-based model in separating younger and older participants measured by binary classification accuracy for both English and Korean datasets. Our method also generalizes to a different category (fruit), demonstrating its adaptability.

Keywords: verbal fluency, semantic fluency, Executive Function, Semantic memory, word embeddings, Relation extraction, category fluency test

Received: 30 Nov 2018; Accepted: 17 Apr 2019.

Edited by:

Carlo Semenza, University of Padova, Italy

Reviewed by:

Zude Zhu, Jiangsu Normal University, China
Maria Montefinese, Department of General Psychology, University of Padova, Italy
Yves Joanette, Université de Montréal, Canada  

Copyright: © 2019 Kim, Kim, Wolters, MacPherson and Park. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Prof. Jong C. Park, School of Computing, Information and Communications University, Daejeon, South Korea, park@nlp.kaist.ac.kr