GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

Rick Wilming^1*Artur Dox²Hjalmar Schulz²Marta Oliveira¹Benedict Clark¹Stefan Haufe^1,2

• ¹Physikalisch-Technische Bundesanstalt, Berlin, Germany
• ²Technische Universitat Berlin, Berlin, Germany

Large pre-trained language models have become a crucial backbone for many downstream tasks in natural language processing (NLP), and while they are trained on a plethora of data containing a variety of biases, such as gender biases, it has been shown that they can also inherit such biases in their weights, potentially affecting their prediction behavior. However, it is unclear to what extent such biases also impact feature attributions, generated by applying "explainable artificial intelligence" (XAI) techniques, in possibly unfavorable ways. To systematically study this question, we create a gender-controlled text dataset, GECO, in which the alteration of grammatical gender forms induces class-specific words and gives rise to ground truth feature attributions for gender classification tasks, enabling the objective evaluation of the correctness of XAI methods. We apply this dataset to the pre-trained BERT model, which we fine-tune to different degrees, to quantitatively measure how pre-training induces undesirable bias in feature attributions and to what extent fine-tuning can mitigate such explanation bias. To this extent, we provide GECOBench, a rigorous quantitative evaluation framework for benchmarking popular XAI methods. We show a clear dependency between explanation performance and the number of fine-tuned layers, where XAI methods are observed to benefit from fine-tuning or complete retraining of embedding layers, particularly.