We envisage a novel cognitive intervention program where a robot constantly supports clinicians in evaluating/monitoring the cognitive state of a patient and in making decisions about the intervention plan to follow. The general structure is depicted in Figure 1. The process fosters a continuous “feedback loop” between the robot and the clinician. It interleaves patient–robot interactions (i.e., steps 1.1, 2.1, and 3.1 in Figure 1) with direct clinician validation and involvement (i.e., steps 1.2, 2.2, and 3.2 in Figure 1). The interleaving of steps performed by the two actors aimed at achieving a fruitful synergy combining the computational capabilities of the robot with the analytical capabilities of the clinician.

It is worth noticing that the clinician is constantly involved in the decisional process and maintains control over the decisions made by the robot, validating them. Each cycle consists of a number of human–robot interaction steps aiming at 1) profiling the health state of a person (steps 1.1 and 1.2), 2) defining an intervention plan suitable for the specific health needs of a patient (steps 2.1 and 2.2) and, 3) executing the plan by administrating exercises within a certain temporal horizon (e.g., a week or a month) and evaluating outcomes (steps 3.1 and 3.2).

The cyclic repetition of these phases allows a clinician to continuously monitor and assess the evolving cognitive state of a patient with the support of a robot. Two “feedback chains” are considered as shown in Figure 1. One feedback chain assesses and (if necessary) updates the profile of the user at the end of each cycle. In this way, it is possible to keep track of the outcomes of synthesized plans, keep track of changes in the health state of a person, and adapt the next cycle accordingly. Another feedback chain concerns the continuous construction of a dataset containing information about the evolution of user profiles and the related outcomes of the cognitive interventions. This information would, in particular, allow the clinician to analyze the evolution over time of the state of a user and thus make better decisions about next steps.

To implement the process of Figure 1, an assistive robot should be able to reason about the health state/conditions of a patient and autonomously make suitable decisions. In particular, a robot needs a number of properly designed and integrated cognitive capabilities in order to contextualize assistive behaviors and effectively support both patients and clinicians. Taking inspiration from cognitive architectures (Langley et al., 2009; Kotseruba and Tsotsos, 2020), we focus on the development and integration of AI-based technologies supporting knowledge representation and reasoning and decision making and problem solving. The integration of knowledge representation and reasoning with automated planning has been shown to be effective for the synthesis of flexible robot behaviors. They are particularly crucial to realize advanced (cognitive) controllers capable of (autonomously) personalize and adapt robot behaviors to the specific features of different application scenarios, for example, service robots (Awaad et al., 2015; Tenorth and Beetz, 2015; Porzel et al., 2020), daily assistance (Umbrico et al., 2020a; Cortellessa et al., 2021), and manufacturing (Borgo et al., 2019).

Figure 2 provides an overview of modules developed to support the considered capabilities and their integration within a “cognitive loop.” On the one hand, an ontology-based representation and reasoning module allows an assistive robot to internally represent cognitive and physical information about an assisted person and contextualize its interaction and intervention capabilities accordingly. Pursuing a foundational approach (Guarino, 1998), we defined a domain ontology based on the ICF classification to represent user profiles and reason on the health state of a person. It relies on DOLCE ² as a theoretical foundation, and was written in OWL (Antoniou and Harmelen, 2009) using Protégé ³ . The robot knowledge and related knowledge-reasoning modules have been developed in Java using the open-source library Apache Jena ⁴ . On the other hand, a decision making and problem solving module allows an assistive robot to synthesize and execute intervention plans, personalized according to the “recommendations” extracted from the robot knowledge. The synthesis and execution of such plans rely on PLATINUm (Umbrico et al., 2017), a timeline-based planning and execution framework deployed in assistive scenarios (Umbrico et al., 2020a) and concrete human–robot collaboration manufacturing scenarios (Pellegrinelli et al., 2017). The contribution of this work specifically focuses on the developed ontology-based representation and reasoning capabilities.

There are several factors that can be considered when modeling users. Different choices would support different robot behaviors and different levels/types of adaptation. Broadly speaking, the creation of a complete and effective user model is crucial to realize human–robot interactions characterized by adaptability, trust building, effective communication, and explainability (Tabrez et al., 2020). In the context of cognitive assessment with assistive robots, many works tend to focus on personality, emotions, and engagement as aspects of the user profile that the robot should take into account (Sorrentino et al., 2021). For example, Tapus et al. (2008) described a socially assisted robot therapist designed to monitor, assist, encourage, and socially interact with post-stroke users engaged in rehabilitation exercises. This work investigated the role of the robot’s personality (i.e., introvert–extrovert) in the therapy process taking into account the personality traits of a user. Similarly, (Rossi et al. (2018) investigated the influence of the user’s personality traits on the perception of the Pepper robot, administrating a cognitive test. Their results suggested that the usage of a robot in this context improved socialization among the participants. On the other hand, the works of Desideri et al. (2019) and (Pino et al. (2020) showed that the usage of a robotic platform for cognitive stimulation engaged more participants to the therapy. In the mentioned works, the influence of each aspect was mostly investigated offline and it was mostly related to the occurred quality of the interaction. In addition, the robotic platform was adopted as a medium for the administration of the clinical protocol, without providing any cues on how the information collected by the robot could be used for planning future interventions. The assumption behind this work is that the robot should be able to adapt its intervention, by focusing on the quality of the interaction, but also on the user cognitive profile.

Concerning our contribution, other works have used the ICF framework to characterize cognitive and physical conditions of users. The work (Kostavelis et al., 2019) introduced a novel robot-based assessment methodology of users’ skills to characterize the needed level of daily assistance. The work (Filippeschi et al., 2018) used ICF to characterize cognitive and physical skills of users and accordingly represent the outcomes of the implemented robot-based assessment procedures. Similarly, the work (García-Betances et al., 2016) used ICF to represent needs and requirements of different types of users and support a user-centered design of ICT technologies. This work integrates an ontological model of ICF into the cognitive architecture ACT-R (Anderson et al., 1997) to simulate the behaviors of different types of user. Nevertheless, these works present a “rigid” and static representation as they usually do not rely on a well-structured ontological formalism to dynamically contextualize knowledge about users (i.e., user profiles) in different situations. Such works usually do not integrate online reasoning mechanisms to allow assistive robots to autonomously reason about the specific needs of a user and autonomously (or partially autonomously) decide the kind of intervention that best fit such needs. Conversely, our approach pursues a highly flexible solution implementing the cognitive capabilities needed to understand health conditions of users and (autonomously) personalize assistance accordingly, under the supervision of a human expert.

The ICF classification aimed at organizing and documenting information on functioning and disability. It pursues the interpretation of functioning as a dynamic interaction among health conditions of a person, environmental factors, and personal factors. Each defined concept characterizes a specific aspect concerning the physical or cognitive functioning of a person. The level of functioning of each physical/cognitive aspect is represented by the following scale: 1) the value 0 denotes no impairment, 2) the value 1 denotes soft impairment, 3) the value 2 denotes medium impairment, 4) the value 3 denotes serious impairment, and 5) the value 4 denotes full impairment.

The ICF classification is organized into two parts. A part deals with functioning and disabilities while the other part deals with contextual factors. The former is further organized into the components body functions and body structures that are the ones considered in the design of the ontological model. The body is an integral part of human functioning and the bio-psychosocial model considers it in interaction with other components. Body functions are thus the physiological aspects of body systems, while structures are the anatomical support (e.g., sight is a function while the eye is a structure). Several ICF concepts describe the functioning of mental faculties and have been used to define user profiles. The concept OrientationFunctioning characterizes the functioning of general mental functions of known and ascertaining one’s relation to time, to place, to self, objects, and space. The concept AttentionFunctioning characterizes specific mental functions focusing on external stimulus or internal experience for the required period of time. The concept MemoryFunctioning characterizes specific mental functions of encoding, storing information, and retrieving it as needed.

Other ICF concepts have been instead used to characterize the interaction capabilities of a person and thus identify interaction preferences determining the way a robot should interact with a person while administrating exercises. The concept SeeingFunctioning models specific functions related to seeing the presence of light and sensing the form, the size, shape, and color of visual stimuli. The concept HearingFunctioning models sensory functions related to sensing the presence of sounds and discriminating the location, pitch, loudness, and quality of sounds.

Following ICF classification, the ontological model defines a number of concepts that represent different FunctioningQuality of a person. As mentioned in Section 2.2, we rely on DOLCE as foundational ontology. Then, the ICF qualities are modeled as subclasses of DOLCE:Quality and are associated to entities of type DOLCE:Person. The concept Profile defines a descriptive context of the overall functioning qualities of a particular person. It represents the outcome of a profiling phase and consists of a number of Measurements. Each measurement associates the evaluation of a functioning quality to a value representing the assigned ICF score (i.e., the outcome of the evaluation). Knowledge-reasoning processes analyze such measurements (i.e., a user profile) to autonomously infer the physical or cognitive impairments characterizing the functioning state of a person. Eq. 1 in the following section shows a general inference rule used to detect such impairments. ∀ x, y, w. ∃ z. ( Measurement ( x ) ∧ measures ( x, y ) ∧ FunctioningQuality ( y ) ∧ hasOutcome ( y, w ) ∧ FunctioningRegion ( w ) ∧ greaterThan ( hasICFscore ( w ) , 0 ) ∧ lowerThan ( hasICFscore ( w ) , 5 ) → Impairment ( z ) ∧ concerns ( z, y ) ∧ satisfies ( z, x ) ) . (1)

In addition to this knowledge, the ontology characterizes properties of intervention plans. A robot can indeed be endowed with a number of “programs” implementing known tests suitable to evaluate/stimulate different functioning qualities, for example, the Free and Cued Selective Reminding Test for episodic long-term memory assessment or the Trailing Making Test form A for selective attention assessment.

Taking inspiration from some works in manufacturing that define the concept of function (Borgo and Leitão, 2004; Borgo et al., 2009), we characterized intervention actions of a robot in terms of their effects on the functioning qualities of a person. The defined semantics characterizes these “programs” according to the functioning qualities they address. For example, an interactive program implementing the Free and Cued Selective Reminding Test is classified as an intervention action whose effects can “improve” (i.e., has positive effects on) the functioning quality MemoryFunctioning. The obtained ontological model fosters an integrated representation of knowledge about the health state of a person and intervention capabilities of a robot. Knowledge processing mechanisms then use this integrated knowledge to infer a set of actions suited to address the inferred impairments of a patient (Umbrico et al., 2020b). For example, if MemoryFunctioning is inferred as soft impairment (value 1) and AttentionFunctioning as no impairment (value 0) only actions implementing the Free and Cued Selective Reminding Test (or other similar tests) are inferred as suitable to a patient.

A Profile encapsulates a rich set of information that can be analyzed to infer interaction capabilities of a person and define robot interaction preferences accordingly. If the analysis of a profile infers a medium impairment of the quality HearingFunctioning then, the interactions between the patient and the robot should rely mainly on visual and textual messages rather than voice and audio. In case that audio interactions cannot be avoided (e.g., recorded audio instructions and recommendations or video conferences) it would be possible to properly set the sound level of the robot in order to help the assisted person as much as possible.

Knowledge-reasoning mechanisms thus infer also how intervention plans should be carried out by the robot in order to effectively interact with the considered patient. In this regard, we have defined four interaction parameters characterizing the execution of robot actions: 1) sound level, 2) subtitle, 3) font size, and 4) explanation. The sound level is an enumeration parameter with values \{ none, regular, high\} specifying the volume of audio communications and messages from the robot to the patient. Patients with soft or medium hearing impairment represented as HearingFunctioning would need a high sound level, while audio would be completely excluded for persons with serious impairments in order to use different interaction modalities. The subtitle is an enumeration parameter with values \{ none, yes, no\} specifying the need of supporting audio messages through text. Patients with no, soft, or medium impairment of SeeingFunctioning and medium or serious impairment of HearingFunctioning would need subtitles to better understand instructions and messages from the robot.

The font size is a binary parameter with values \{ regular, large\} specifying the size of the font of text messages and subtitles, if used. Patients with medium impairment of SeeingFunctioning would need large fonts in text messages in order to better read their content. Finally, explanation is a binary parameter (i.e., yes or non) specifying the need of explaining an exercise to a patient before its execution. Such instructions would be particularly needed for patients with impaired MemoryFunctioning or OrientationFunctioning. Clearly, the way such explanations are carried out complies with the interaction parameters described earlier.

As shown in Figure 1, user profiling is necessary at the beginning of each intervention cycle to set/update robot knowledge about the health state of an assisted person. The outcome of this step is a profile describing the cognitive state of a person with respect to the developed ontological model. A correct acquisition of this information is crucial for the efficacy of the synthesized intervention plan. The robot indeed relies on the user profile to infer the set of cognitive tests (i.e., stimuli) that are suitable for the considered user and then decide personalized intervention plan (i.e., the further assessment).

The user profile is generated according to the scoring obtained through the administration of the Mini-Mental State Examination (MMSE). The MMSE represents the most used screening test for cognitive status, and it is adopted worldwide by clinicians to briefly assess persons with the suspect of dementia. It encompasses 21 items which cover tests of orientation, recall, registration, naming, comprehension, calculation and attention, writing, repetition, drawing, and reading (Folstein et al., 1983). The cognitive status level is obtained by summing the score of the individual items and normalizing it based on the educational level and the age of the patient. Decreasing scores of repeated tests highlight deterioration in cognition. In particular, the participants were asked to undergo the MMSE administrated by ASTRO. The assessment was performed with a Wizard-of-Oz (WOz) method. A clinician guides the robot through the examination phases using a dedicated web interface. The patient is not aware of the presence of the clinician and he/she directly interacts with the robot. The web interface allows the clinician to select the appropriate MMSE tests to perform. The tests require different interaction modalities between the user and the robot, for example, asking questions to the user or showing images to the user through the front tablet. The clinician can ask the robot to repeat the test if necessary.

The caregiver stores the results of the assessment into the robot knowledge base through the same dedicated technical interface. The overall score of the MMSE is automatically processed by the robot at the end of the session, by parsing the annotated answers. The robot automatically correlates MMSE items with (relevant) ICF functions the robot uses to represent the cognitive state of a patient. According to this correlation, the robot then builds a user profile by mapping received MMSE scores to ICF scores. Table 1 shows how each ICF function can be described by one or multiple MMSE categories. This is an original mapping performed by a clinician between MMSE scores and ICF for generating a user profile representation. For instance, the ICF concept MemoryFunctioning is defined as specific mental functions of registering and sorting information and retrieving it as needed. This ICF function can be described by the MMSE items which cover the recalling, counting, and spelling tests. Based on the same similarity approach, the overall mapping shown in Table 1 is obtained. In order to convert the MMSE scoring of each category into the measured level of impairment of the ICF profiling, a proportional method is used. The current MMSE score (i.e., the number of tasks correctly performed) in one category is compared to the maximum MMSE score achievable in the same category and then converted into the ICF scoring. This mapping is based on an inverted scale of values. For example, if the patient correctly accomplished the requested task of MMSE items (high values of MMSE), he/she gets a lower ICF score (no impairment). If the patient partially accomplished the task, an intermediate value of the ICF score is assigned (mild impairment). If the patient did not accomplish the task, a higher value of ICF score is attributed (hard impairment).

The video and audio of the administration sessions were recorded by the robot’s frontal camera. The recorded videos were off-line analyzed by the clinician to extract the quality of interaction (e.g., the number of robot’s repetition) with the robot and additional evidence of the cognitive decline (e.g., coherent and incoherent interaction) (Sorrentino et al., 2021). Table 1 reports the complete list of extracted parameters. These data are then merged with the individual MMSE score returned by the robot and manually mapped into the ICF scores, following the proposed mapping. Figure 4 reports the final results for each user. Data were then analyzed with the proposed framework and discussed with the clinician to corroborate the analysis. The clinician found an adequate representation of the profiles and validated all of them.

The subsequent considered actions consist in the selection of a number of cognitive tests to administer, regarding the inferred impaired functioning qualities of a patient. The assistive capabilities of such actions depend on the features of the associated cognitive tests and thus on the functioning qualities stimulated by them. According to the inferred impairments and the recommendations generated by the developed knowledge processing mechanisms, a number of these tests are selected for administration.

We have considered a total number of 10 cognitive tests that are typically used to further investigate and evaluate the cognitive state of a person. The Free and Cued Selective Reminding Test, the Rey’s Figure Test, the Forward Digit Test, and the Backward Digit Span Test evaluate and stimulate the functioning quality MemoryFunctioning. The Trailing Making Test form A and the Trailing Making Test form B evaluate and stimulate the functioning quality AttentionFunctioning. The Stroop Test evaluates and stimulates the functioning quality OrientationFunctioning. The Boston Naming Test 40-item, the Animals Test, and the Denomination Test evaluate and stimulate the functioning quality LanguageFunctioning.

The experiments have been performed with the objective to demonstrate the capability of combining this knowledge with the ICF scores of Figure 4 (i.e., user profiles) to determine actions that fit the cognitive status of the profiled users and thus achieve personalization. Figure 5 shows results of the experiments. It specifically shows a heat-map with the ranking values of known cognitive tests (i.e., actions) for each user profile. The value expresses the significance of a specific test/action considering the cognitive impairments inferred for a particular user profile. The higher the computed ranking value the more is the relevance of a particular action for the corresponding user. Figure 6 then shows aggregated numbers pointing out the total level of impairments of the considered users in Figure 6A and the inferred impact of each action in Figure 6B. These figures clearly show the most compromised users (i.e., the users with the highest level of impairment) and the most useful actions (i.e., the actions that address the higher number and most significant impairments of users) according to robot knowledge.

The higher the ranking value the higher the “seriousness” of the associated impairments and consequently the significance of a test with respect to the cognitive state of a user. An example is profile 8 whose cognitive state is characterized by several impairments as can be seen from the ICF scores resulting from the outcome of MMSE in Figure 4. Consequently, as shown in Figure 5, many of the considered tests have been computed as relevant to the cognitive state of this patient. Higher values have been computed for tests addressing impaired qualities, for example, the Forward Digit Test addressing MemoryFunctioning (medium impairment in Figure 4) or the Trailing Making Test form A addressing AttentionFunctioning (serious impairment in Figure 4).

Vice versa low ranking values have been computed for not so serious impairments. An example is the user profile 5 whose cognitive state is characterized by few soft impairments (see again the ICF scores resulting from the outcome of MMSE in Figure 4). In this case, a “minimum” ranking value has been computed only for the Denomination Test, the Stroop Test, and the Boston Naming Test 40-item that address the soft impaired functioning qualities OrientationFunctioning and LanguageFunctioning. The outcome of the knowledge-based reasoning mechanisms has been assessed by an expert clinician. The results of this validation are reported in Section 5.4.

Once a personalized set of interventions has been defined, user profiles are further evaluated to decide how such actions should be performed (i.e., interaction preferences). This reasoning step relies on a number of inference rules that link ICF scores to the interaction preferences introduced in Section 4.2. We have considered ICF scores concerning PerceptualFunction (PER) and MemoryFunctioning (MEM). MEM is linked to the interaction preference explanation as described in Section 4.2. PER is linked to the interaction preferences sound level, font size, and subtitles. As shown in Table 1, a clinician assigns a score to PER by evaluating the number of robot’s repetitions. Given the lack of a precise evaluation of hearing and seeing capabilities of users, we have used PER scores to implicitly measure the functioning qualities HearingFunctioning and SeeingFuncitoning and infer the related interaction preferences as described in Section 4.2.

Table 2 shows the interaction preferences inferred for the considered users. The results show the capability of the developed knowledge-reasoning mechanisms to contextualize the execution of intervention actions (and thus robot behaviors) by defining a number of coherent interaction parameters. Users with soft or no impairment conditions of PerceptualFunctioning would not require particular interaction preferences for the execution of the associated intervention actions.

An example, is profile 5 that has no impairment of PerceptualFunctioning (PER = 0 in Figure 4) and is associated to the interaction parameters ⟨ regular, regular, none, and none ⟩ in Table 2. Vice versa users with medium or serious impairment conditions of PerceptualFunctioning would require a specific configuration of robot behaviors for the execution of the associated intervention actions. Examples are profile 4 and profile 8 that have serious impairments of PerceptualFunctioning (PER = 4 in Figure 4) and are both associated to the interaction parameters ⟨ high, large, yes, and yes ⟩ in Table 2. These experiments show the feasibility of the developed knowledge-reasoning mechanisms in personalizing and adapting robot assistive behaviors to the health needs of different patients.

It is worth noting that the use of domain-dependent rules like the ones defined for PER would not limit the generality of the developed knowledge-reasoning approach. Rather, this situation shows how developed reasoning behaviors can be easily tailored to specific needs and features of different assistive scenarios.

The system identifies the subjects with a higher level of impairment, which needs more attention during a further comprehensive neuropsychological testing. Therefore, based on the stored profiles, the system suggests a broader set of further cognitive tests to the users with a higher average of cognitive issues, to provide more informative support for the clinician. We are referring particularly to subject numbers 1, 2, 4, 6, 7, and 8 that, respectively, report a raw score of 16, 17, 12, 18, 15, and 17 out 30 on MMSE. For example, subjects 1, 6, and 8 were strongly suggested to undergo the similar set of tests. In particular, those subjects, which showed a mild to medium-cognitive impairment, were asked to undergo tests related to executive functions and working memory (forward and backward digit span, and Trailing Making Test form A and B). Such cognitive impairments represent a crucial risk factor for the transition to mild cognitive impairment to a full-blown dementia syndrome. In addition, the subject number 8, which showed a more severe impairment related to languages, was suggested to undergo also a denomination test and the Boston Naming Test, both tests for language domain. Such suggestions were not mentioned in the other two subjects. Moreover, the memory function, tested by the Free and Cued Selective Reminding Test, was suggested to be studied in almost all the subjects except for the not impaired subject numbers 3 and 5.

Interestingly, the system reports less need for further test administration for subject numbers 4 and 7, respectively, the two subjects with worst cognitive performances. This is an expected performance as the implemented knowledge processing mechanisms assign ranks to intervention actions in a “non-linear way.” In particular, lower ranks are assigned to tests addressing too compromised functions as they are supposed to be managed separately. That may represent a counter-intuitive data, but a common clinic routine. Too much compromised subjects’ condition makes the clinical picture already clear and further analysis fruitless. Therefore, clinical practice is a balance between the need to accomplish an explicit vision of the case, and the economy of time and resources. On the other hand, regarding the subjects with a non-impaired neuro-cognitive profile (i.e., numbers 3 and 5. Score of 27 and 29 out 30), the system proposed fewer tests as informative. In conclusion, such results are aligned with standard clinical practice, thus the assistive robot could represent a useful tool for assessment process refinement.