Simultaneous Text and Gesture Generation for Social Robots with Small Language Models

Alessio Galatolo^*Katie Winkle

• Uppsala University, Uppsala, Sweden

Increased robot communication abilities come with increased user expectations for verbal and non-verbal social robot behaviour. Recent works have demonstrated how Large Language Models (LLMs) have the potential to support the autonomous generation of such behaviours, including both dialogue and non-verbal behaviour. Specifically considering non-verbal behaviour generation, current LLM-based approaches often rely on multi-step reasoning and multi-turn generation using large, closed-source models. This results in significant computational overhead and limits their applicability, particularly in low-resource and/or privacy-constrained environments. In this work, we explore the performance of current techniques and, motivated by their ineffectiveness in low-compute environments, introduce a novel method for the simultaneous generation of text and gestures that adds minimal to no additional overhead compared to plain text generation. Importantly, our system does not generate low-level pose sequences or joint trajectories. Rather, it operates at a higher level of abstraction, predicting communicative intentions that are then mapped to platform-specific expressions. Our method uses robot-specific 'gesture heads' that are autonomously derived from the model and do not require specialised or pose-based datasets. This design choice enables generalisability across platforms, without committing to a specific embodiment or gesture modality. These gesture heads operate in parallel with the language modelling component, almost eliminating any additional computation time. We validate our method on two robot platforms-(i) Furhat, capable of various facial expressions, and (ii) Pepper, capable of multiple hand gestures and movements. Experimental results show the effectiveness of our method both in terms of computational time, space and overall behavioural performance.