AUTHOR=Fox Carly , Jones Sharad , Gillam Sandra Laing , Israelsen-Augenstein Megan , Schwartz Sarah , Gillam Ronald Bradley 

TITLE=Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA)

JOURNAL=Frontiers in Psychology

VOLUME=Volume 13 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2022.894478

DOI=10.3389/fpsyg.2022.894478

ISSN=1664-1078

ABSTRACT=Language sample analysis (LSA) is an important practice for providing a culturally sensitive and3
accurate assessment of a child’s language abilities. A child’s usage of literate language devices4
in narrative samples has been shown to be a critical target for evaluation. While automated5
scoring systems have begun to appear in the field, no such system exists for conducting progress-6
monitoring on literate language usage within narratives. The current study aimed to develop a7
hard-coded scoring system called the Literate Language Use in Narrative Assessment (LLUNA),8
to automatically evaluate six aspects of literate language in non-coded narrative transcripts.9
LLUNA was designed to individually score six literate language elements (e.g., coordinating and10
subordinating conjunctions, linguistic and mental verbs, adverbs, and elaborated noun phrases).11
The interrater reliability of LLUNA with an expert scorer, as well as its’ reliability compared to12
certified undergraduate scorers was calculated using a quadratic weighted kappa (Kqw). Results13
indicated that LLUNA met strong levels of interrater reliability with an expert scorer on all six14
elements. LLUNA also surpassed the reliability levels of certified, but non-expert scorers on four15
of the six elements and came close to matching reliability levels on the remaining two. LLUNA16
shows promise as means for automating the scoring of literate language in LSA and narrative17
samples for the purpose of assessment and progress-monitoring.