Weaponizing Cognitive Bias in Autonomous Systems: A Framework for Black-box Inference Attacks

Shiyong ChuYuwei Chen^*

• Aviation Industry Development Research Center of China, Beijing, China

Autonomous systems deployed in high-dimensional environments increasingly rely on prioritization heuristics to allocate attention and assess risk. While often perceived as objective, these systems can exhibit cognitive biases, such as salience, spatial framing, and temporal familiarity that alter decision-making without modifying the input or accessing internal states. This study introduces Priority Inversion via Operational Reasoning (PRIOR), a black-box, non-perturbative diagnostic framework that probes reasoning vulnerabilities using structurally biased but semantically neutral scenario cues, without altering pixel-level, statistical, or surface semantic properties. As direct access to embodied vision-based systems remains limited, we evaluate PRIOR using large language models as abstract reasoning proxies. These models simulate cognitive prioritization under constrained textual scenarios, though they do not reflect real-time perception or physical-world interaction. Through controlled experiments on textual surveillance analogs inspired by UAV decision contexts, we find that minimal structural cues can induce consistent priority inversions across multiple models. By jointly analyzing model justifications and confidence estimates, we observe systematic distortions in inferred threat relevance, even under input symmetry. These findings reveal the fragility of the inference level in black-box reasoning systems and motivate new evaluation strategies that go beyond output correctness to interrogate internal prioritization logic. Future work should extend these insights to dynamic, embodied, and visually grounded agents operating in real-world deployments.