Original Research ARTICLE
Automatic scoring of semantic fluency
- 1School of Computing, Information and Communications University, South Korea
- 2School of Informatics, University of Edinburgh, United Kingdom
- 3Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, United Kingdom
In neuropsychological assessment, semantic fluency is a widely accepted measure of executive function and access to semantic memory. While fluency scores are typically reported as the number of unique words produced, several alternative manual scoring methods have been proposed that provide additional insights into performance, such as clusters of semantically related items. Many automatic scoring methods yield metrics that are difficult to relate to the theories behind manual scoring methods, and most require manually-curated linguistic ontologies or large corpus infrastructure. In this paper, we propose a novel automatic scoring method based on Wikipedia, Backlink-VSM, which is easily adaptable to any of the 61 languages with more than 100k Wikipedia entries, can account for cultural differences in semantic relatedness, and covers a wide range of item categories. Our Backlink-VSM method combines relational knowledge as represented by links between Wikipedia entries (Backlink model) with a semantic proximity metric derived from distributional representations (vector space model; VSM). Backlink-VSM yields measures that approximate manual clustering and switching analyses, providing a straightforward link to the substantial literature that uses these metrics. We illustrate our approach with examples from two languages (English and Korean), and two commonly used categories of items (animals and fruits). For both Korean and English, we show that the measures generated by our automatic scoring procedure correlate well with manual annotations. We also successfully replicate findings that older adults produce significantly fewer switches compared to younger adults. Furthermore, our automatic scoring procedure outperforms the manual scoring method and a WordNet-based model in separating younger and older participants measured by binary classification accuracy for both English and Korean datasets. Our method also generalizes to a different category (fruit), demonstrating its adaptability.
Keywords: verbal fluency, semantic fluency, Executive Function, Semantic memory, word embeddings, Relation extraction, category fluency test
Received: 30 Nov 2018;
Accepted: 17 Apr 2019.
Edited by:Carlo Semenza, University of Padova, Italy
Reviewed by:Zude Zhu, Jiangsu Normal University, China
Maria Montefinese, Department of General Psychology, University of Padova, Italy
Yves Joanette, Université de Montréal, Canada
Copyright: © 2019 Kim, Kim, Wolters, MacPherson and Park. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence: Prof. Jong C. Park, School of Computing, Information and Communications University, Daejeon, South Korea, email@example.com