Everything Robots Need to Know About Cooking Actions: Creating Actionable Knowledge Graphs to Support Robotic Meal Preparation

Michaela Kümpel^1*Manuel Scheibl^2*Jan-Philipp Töberg^3*Vanessa Hassouna¹Philipp Cimiano³Britta Wrede^2,3Michael Beetz¹

• ¹Institute for Artificial Intelligence, Bremen University, Bremen, Germany
• ²Medical Assistance Systems Group, Medical School OWL, Bielefeld University, Bielefeld, Germany
• ³Universitat Bielefeld Center for Cognitive Interaction Technology, Bielefeld, Germany

This paper addresses the challenge of enabling robots to autonomously prepare meals by bridging natural language recipe instructions and robotic action execution. We propose a novel methodology leveraging Actionable Knowledge Graphs (AKGs) to map recipe instructions into six core categories of robotic manipulation tasks, termed Action Cores (ACs): cutting, pouring, mixing, preparing, pick & place, and cook & cool. Each AC is subdivided into Action Groups (AGs), which represent a specific motion parameterization required for task execution. Using the Recipe1M+ dataset (Mar´ın et al., 2021), encompassing over one million recipes, we systematically analysed action verbs and matched them to ACs by using direct matching and cosine similarity, achieving a coverage of 76.5%. For the unmatched verbs, we employ a neuro-symbolic approach, matching verbs to existing AGs or generating new action cores utilizing a Large Language Model (LLM). Our findings highlight the versatility of AKGs in adapting general plans to specific robotic tasks, validated through an experimental application in a meal preparation scenario. This work sets a foundation for adaptive robotic systems capable of performing a wide array of complex culinary tasks with minimal human intervention.