The Effect of Syntactic and Semantic Information on Word Grounding through Visual Perception

Abstract

Word grounding refers to the ability of agents to associate linguistic terms (e.g., "apple") with the perceptual concepts they represent, such as color, shape, and spatial features. This paper presents a quantitative evaluation of how syntactic and semantic information affects word grounding performance within a generative, multimodal Bayesian framework. We evaluate five model configurations that integrate syntactic and semantic features into a baseline that grounds words solely through perceptual cues—specifically, Viewpoint Feature Histograms (VFH), color histograms, and spatial centroids. Each enhanced configuration introduces an additional linguistic component in isolation: (i) word position indices, (ii) part-of-speech (POS) tags, (iii) static word embeddings, and (iv) transformer-based contextual embeddings. This design enables a clear assessment of how each linguistic layer contributes to grounding performance. Experiments conducted on a CLEVR–derived 3D scene dataset show that linguistic information improves grounding accuracy across Color, Geometry, and Spatial Relation modalities. Contextual embeddings yield a 53.8% absolute improvement in object grounding over the baseline. Furthermore, only the models augmented with semantic embeddings generalize effectively to synonym-substituted descriptions without retraining. This synonym generalization reflects lexical transfer from the pre-trained embedding space. These results underscore the role of linguistic structure in perceptual alignment and provide empirical foundations for developing language-grounding systems.