A BERT-based rice enhancer identification model combined with sequence-representation differential entropy interpretation

Yajing Pu¹Xintong Hao¹Zhaoqi Zheng¹Huiyan Ma²Zhibin Lv^1*

• ¹School of Biomedical Engineering, Sichuan University, Chengdu, China
• ²College of Life Science, Sichuan University, Chengdu, Sichuan Province, China

Rice is an important food crop, and research into its gene expression regulation is of great significance to molecular breeding and yield improvement. Enhancers are key elements that regulate the spatial and temporal expression of genes, making their precise identification a core challenge in functional genomics. Among existing methods employing deep learning for rice enhancer identification, there are limitations in extracting rice enhancer features and model architecture. Therefore, we propose a novel model architecture: RiceEN-BERT-SVM. The model utilizes a pre-trained nucleotide large language model as the feature extraction tool and SVM for enhancer sequence recognition. Additionally, differential entropy of feature representation is used to explain the improvement in model performance. The results indicate that direct application of the pre-trained language model achieves high accuracy in rice enhancer identification, with cross-validation and independent testing accuracies of 88.05% and 87.52%, respectively. These results surpass state-of-the-art models by 1.47–6.87% on the same dataset. Fine-tuning the language model further improved the performance of RiceEN-BERT-SVM by 6.95%, resulting in a final accuracy of 93.63%. We introduce differential entropy for sequential feature representation to explain the reasons for the improved model performance. The calculations indicate that as the number of fine-tuning iterations increases, the differential entropy distributions representing positive and negative sample characteristics gradually separate from their initial superposition state. This separation corresponds with an improvement in model performance. However, there is an optimal number of fine-tuning. In this study, the best model performance is achieved when the number of fine-tunings reaches 6. Exceeding this number, the differential entropy distributions of positive and negative samples begin to overlap again, leading to a decline in model performance.