Abstract
Introduction:
Breast cancer is one of the leading health problems in the world, and the challenge lies in the fact that its diagnosis at the earliest possible and accurate rate is the main factor to guarantee a successful patient outcome. The traditional deep learning (DL) frameworks usually utilize data from a single modality at a time and, therefore, are not capable of addressing the complexity and heterogeneity of the disease, particularly when data are unavailable or incomplete.
Methods:
To address these constraints, a multimodal breast cancer diagnosis model is presented that consists of attention-based transformers to achieve efficient modality specific feature extraction, the modified mantissa search (MMS) algorithm to remove irrelevant features, and the American zebra optimization (AZO) algorithm to dynamically and efficiently combine features. Final classification is then performed using a lightweight convolutional neural network (LCNN) to avoid compromising diagnostic accuracy.
Results and Discussion:
The proposed model is highly generalizable and resilient to missing modalities, achieving 98.958, 97.37, and 99.438% accuracy on Mammographic Image Analysis Society (MIAS), BreakHis, and combined multimodal datasets, respectively. These findings reveal their usefulness and strength in clinical diagnostic cases with a variety of imaging data.
1 Introduction
Breast cancer is the most prevalent cancer that is regularly diagnosed in women all over the world and the most common cause of cancer-related deaths (). Although timely diagnosis greatly enhances prognosis, the disease's heterogeneous nature and its varied manifestations across individuals pose challenges. The traditional imaging systems like mammography, ultrasound, and magnetic resonance imaging (MRI) have proven to provide useful diagnostic information: mammography is extensively used in initial screening, ultrasound can be used in real time and is radiation-free and MRI has high contrast resolution (–). Although these are the strengths, there exist limitations in each modality. As an example, mammography is less effective in dense breast tissue, whereas ultrasound is operator dependent and MRI is costly and not widely available (, ). Artificial intelligence (AI), especially deep learning (DL), has proven to be an effective means of analyzing medical images to improve detection, reduce human error, and support clinical decision-making (–). Image enhancement, segmentation, and feature extraction are used to enable DL models to recognize patterns associated with malignancy across imaging modalities (, ). Convolutional neural networks (CNNs) have also demonstrated encouraging results in categorizing breast lesions based on hierarchical features learned from raw image data (, ). The combination of machine learning (ML) and DL with medical imaging has enabled profound diagnostic sensitivity and specificity (, ). Nonetheless, the majority of the current solutions are single-modal and, therefore, do not reflect the complexity of breast cancer. Recent multimodal models that seek to combine information across various imaging sources continue to have problems: they fail to perform well at global and modality-specific feature extraction, semantic inconsistencies, unable to deal with modalities missing effectively, and do not have dynamic weighting mechanisms that can be used to explain the varying diagnostic value of each modality (–). The findings indicate excellent treatment results on benchmark datasets, which indicate the practical potential and applicability of the model in the development of breast cancer screening and personalized healthcare methods (, ). However, no studies have been carried out so far that investigate the opportunities of different modalities to enrich breast cancer screening (–). This research introduces a powerful diagnostic model that combines histology and mammography images, extracts modality-specific features, and achieves efficiency and accuracy through optimized feature selection and lightweight classification. The attention models, which are transformer based, are applied to identify rich, hierarchical features that represent local and global information in histology and mammography images. The modified mantissa search (MMS) algorithm is a dimensionality-reduction feature search algorithm that eliminates redundant and irrelevant data to minimize computational complexity. To achieve successful multimodal feature fusion, particularly when features are missing, the American zebra optimization (AZO) algorithm is used to provide robust and dependable feature integration. Lightweight convolutional neural network (LCNN) is a fast and accurate classification model, and is ideal for real-time applications in resource-limited clinical environments. It is a powerful solution to issues in diagnosing breast cancer through a combination of smart feature extraction, smart selection, and efficient classification. Depending on the contributions, develop the research questions (RQs).
RQ1: Does the accuracy of detecting breast cancer increase when histology and mammography are combined?
RQ2: What effects do low- and high-level traits unique to a modality have on diagnostic performance?
RQ3: What effects does feature selection have on prediction accuracy and efficiency?
RQ4: Is it possible for a lightweight model to remain accurate when multimodal data is incomplete?
To address this gap, a multimodal breast cancer diagnosis model is proposed that uses both histopathology and mammography images to improve classification performance through optimal feature fusion and lightweight deep learning. The key contributions of the proposed model are given below:
To effectively capture detailed and comprehensive information from each modality, attention-based transformer models are employed. The models are adept at learning both low-level features (such as edges and textures) and high-level semantic features (such as cell or lesion patterns), which are specific to histology and mammography images.
The proposed model addresses the curse of dimensionality and redundancy in high-dimensional medical imaging data using the MMS algorithm. For effective multimodal fusion, especially when dealing with incomplete or missing modality data, the AZO algorithm is used.
The diagnosis and classification of breast cancer is performed using LCNN, which maintains high accuracy while minimizing computational cost, making it suitable for a real-time clinical environment. LCNN benefits from optimized, fused features, leading to improved predictive performance without the overhead of complex architectures.
The effectiveness of the proposed model is validated using two widely used benchmark datasets. The MIAS dataset contains mammographic images with various breast tissue densities and abnormalities, while the BreakHis dataset includes Histopathological breast cancer biopsy images at multiple magnification levels.
The rest of this article is organized as follows. Section 2 presents the review of literature on breast cancer diagnosis using artificial intelligence. Section 3 illustrates the proposed methodology, including feature extraction, feature reduction, optimal feature fusion, and classification. The results and comparative analysis of proposed and existing models are discussed in Section 4. The article concludes in Section 5.
2 Related studies
Youssef et al. () have proposed a hybrid feature extraction method that uses a DL model to improve breast cancer prediction. Support vector machine (SVM) and extreme gradient boosting (XGB) techniques achieve up to 96.22% accuracy, 97.19% sensitivity, and 95.23% specificity, showing potential for early breast cancer detection through thermal imaging. Maurya et al. () have proposed a modified Vision Transformer (Breast Cancer Classification using Lightweight Customized Vision Transformer [BMEA-ViT]), which replaces the original, notoriously computationally intensive multi-headed self-attention (MSA) with a multi-headed external attention (MEA) unit that achieves linear complexity and improves generalizability. Ruan et al. () have developed a diagnostic BI-RADS classification using B-mode images, Nakagami parametric images, and semantic features. Huang et al. () combined deep features with edge-focused information to improve segmentation accuracy using Edge-Aware Multi-Scale Group-Mix Attention Network (EMGANet), which efficiently integrates global and local features and enhances cancer border detection. Gezimati and Singh () have proposed DL-based multimodal breast cancer diagnosis and characterization system using terahertz (THz) and infrared (IR) imaging data. Rahman et al. () have proposed a bimodal sensing system that uses tactile and multispectral sensors to distinguish between tumors and non-tumor breast conditions. Cho et al. () have proposed a DL-based ultrasound image segmentation and classification approach for accurate breast cancer diagnosis. Kansal and Kansal () highlighted the critical need for disease prediction. Deng et al. () have presented the first distributed learning method [Federated Block Coordinate Descent/Federated Stochastic Block Coordinate Descent (FedBCD)] for collaborative breast ultrasound images, addressing data privacy and modality integration. Chikkala et al. () have introduced a bidirectional recurrent neural network (BRNN) for histopathology image analysis.
2.1 Problem description
Breast cancer is still a major worldwide source of anxiety, and in order to lower mortality and enhance patient outcomes, primary and correct recognition is critical. The problems of existing breast cancer diagnosis models are summarized in Table 1. Conventional DL-based diagnostic systems have largely relied on unimodal imaging data, such as histopathology or mammography (–). The clinical application is further hampered by the challenges of high data dimensionality (), inefficient integration of complementary information, fixed modality weighting, and limited robustness to missing modalities (–). The model uses attention-based transformer networks to represent modality-specific high- and low-level features, making it modality-aware. The objectives below used to fill the research gaps.
Multimodal diagnostic model combines both the mammography images and the histology, which appeals to the depth and the accuracy of the breast cancer detection.
Each modality is captured at low- and high-level features, which ensures that its model makes the most out of the strengths of various data.
The data is optimized in terms of dimensionality, and only the most relevant features are considered, which enhances the processing speed and the overall results of prediction.
There is also a lightweight classification framework that can work with incomplete data, but the diagnostic outcomes are high.
Table 1
| References | Methodology | Technique used | Dataset | Findings | Research gaps |
|---|---|---|---|---|---|
| Youssef et al. () | Early breast cancer prediction | Gabor filters, Canny edges, ResNet-50 and MobileNet | Thermal imaging | Accuracy 96.22%, sensitivity 97.19% | Limited to morphologic information about the lesion |
| Maurya et al. () | Breast cancer classification | External attention (MEA) for disease detection | BreakHis | Accuracy 95.74%−97.25% | Malignant lesions exhibit stiffer characteristics |
| Ruan et al. () | Ultrasound breast tumors diagnosis | Two-branch Transformer-CNN, feature fusion module | Multicenter BI-RADS 4 database | AUC 0.856 and accuracy 97.56% | The network had a limitation in not utilizing spatial information |
| Huang et al. () | Breast cancer image segmentation | EMGANet for edge feature and multi-scale attention | Dataset-B, BUSI, BUSI-WHU | Accuracy 98.56%, Mean IoU 90.32% | This model does not learn the unique features of each mode well. |
| Gezimati et al. () | Multimodal breast cancer characterization | THz and IR imaging, decision-level fusion | IR thermography dataset | Accuracy 96.6%, sensitivity 96.3% | An expensive multi-head self-attention model limits performance |
| Rahman et al. () | Bimodal profile for breast cancer detection | CNNs for feature extraction | Tactile and spectral datasets | Accuracy 83% for malignant tumors | The insufficient feature extraction ability creates high false positives |
| Cho et al. () | Breast cancer segmentation and classification | U-Net on B-mode and SE-mode Ultrasound | Clinical ultrasound data | Accuracy 98.46%, Dice score 78.23% | Data accessibility and the challenge of data imbalance. |
| Kansal et al. () | Multi-class breast cancer classification | DenseNet for feature fusion | BUSI ultrasound dataset | Accuracy 98.28% | Complexity and frequently unclear interpretations of decisions |
| Deng et al. () | Joint learning for breast cancer diagnosis | JUVIL model with temporal-spatial adapters | Distributed image and video data | Detection accuracy 95.56% | Limited by the data dimensionality issue, which causes high false alarm |
| Chikkala et al. () | Breast cancer diagnosis | BRNN with ResNet50, GRU, residual branch | BreaKHis | Accuracy 97.25% and precision 87% | Fine details were missed during the segmentation process |
Review summary of ML/DL models on breast cancer diagnosis.
3 Materials and methods
Figure 1 presents the idea of optimal feature fusion and lightweight DL for multimodal breast cancer diagnosis using the MIAS and BreakHis datasets. The suggested model begins with the intensive image preprocessing phase to improve data quality and consistency across modalities. This involves image resizing (MIAS images resized to 299 pixels by 299 pixels and BreakHis histology images resized to 224 pixels by 224 pixels), image normalization to bring out uniformity in the intensity of the pixels, noise elimination, image contrasts to improve the visibility of tumor areas, artifact elimination to remove irrelevant distortions, and patch extraction to bring out important areas of interest.
Figure 1
3.1 Data preparation
The proposed research use of an extensive multimodal dataset comprising histology and mammography images to improve the accuracy of breast cancer diagnosis. Histological data () were obtained by two popular open repositories, BreakHis and BACH, and mammography images (). The data augmentation was used to overcome the imbalance between classes and enrich the training samples (Figure 2). The histology dataset was increased to 95,581 images (51,511 original and 44,070 augmented), and the mammography dataset was increased to 107,346 images (75,658 original and 31,688 augmented). Table 2 reports on the datasets and their sample information. Preprocessing of the image was conducted to align the dimensions of the input: 224 × 224 pixels in histology and 299 × 299 pixels in mammography to fit the DL models.
Figure 2
Table 2
| S. No. | Class type | Number of samples | |||
|---|---|---|---|---|---|
| Original | Augmented | Training | Testing | ||
| BreakHis dataset | |||||
| 1 | Adenosis (A) | 456 | 544 | 700 | 300 |
| 2 | Benign (B) | 100 | 400 | 350 | 150 |
| 3 | Malignant carcinoma (DC) | 2,749 | 2,251 | 3,500 | 1,500 |
| 4 | Fibroadenoma (F) | 1,127 | 1,373 | 1,750 | 750 |
| 5 | In situ carcinoma (IS) | 100 | 900 | 700 | 300 |
| 6 | Invasive carcinoma (IV) | 100 | 1,400 | 1,050 | 450 |
| 7 | Lobular carcinoma (LC) | 426 | 574 | 700 | 300 |
| 8 | Mucinous carcinoma (MC) | 495 | 505 | 700 | 300 |
| 9 | Normal (N) | 96 | 904 | 700 | 300 |
| 10 | Papillary carcinoma (PC) | 348 | 652 | 700 | 300 |
| 11 | Phyllodes tumor (PT) | 469 | 531 | 700 | 300 |
| 12 | Tubular adenoma (TA) | 630 | 870 | 1,050 | 450 |
| Total | 6,096 | 10,904 | 12,600 | 5,400 | |
| MIAS dataset | |||||
| 1 | Normal (N) | 800 | 700 | 1,050 | 450 |
| 2 | Benign with calcification (BC) | 750 | 750 | 1,050 | 450 |
| 3 | Benign with mass (BM) | 780 | 720 | 1,050 | 450 |
| 4 | Calcification (CALC) | 880 | 620 | 1,050 | 450 |
| 5 | Mass (M) | 894 | 606 | 1,050 | 450 |
| Total | 4,104 | 3,396 | 5,250 | 2,250 | |
Summary of class-wise distribution of original and augmented samples for histology and mammography datasets, along with training and testing splits.
3.2 Image preprocessing in histology and mammography images
The image preprocessing pipeline for histology and mammography images is meant to maximize the quality and consistency of the input data, which highly contributes to the model's performance.
Resizing: to fit the input of deep learning models, both the histology and mammography images were resized to 224 × 224 and 299 × 299 pixels, respectively. The image size is also standardized, and the model processes all images in an orderly fashion ().
Pixel normalization: the intensities were normalized to the range of 0–1, which is used to normalize the image data and make the model training process more stable and quicker since the values are on an equal scale ().
Removal of noise, artifact: Gaussian filtering is used to remove noise and undesirable background data. It is essential to enhance classification accuracy, and avoid situations in which the model memorizes irrelevant features ().
Contrast enhancement: it is necessary to enhance the capacity of the model to evaluate minute information, such as tumors or lesions ().
Patch extraction: the patch extraction isolates a region of diagnostic interest (i.e., tumor regions, or regions with abnormal cell structure). It ensures that the model focuses on small but meaningful parts of the image, reduces the computational cost of processing the full image, and increases detection accuracy in critical regions ().
Data augmentation (flips, rotations): augmentation, for example, horizontal and vertical flips, and rotations can be used to combat class imbalance by artificially swelling the size of the dataset. In medical imaging, augmentation is essential, and the acquisition of large datasets across a variety of modalities may be challenging ().
3.3 Feature extraction using attention-based transformer models
The transformer model that was adapted to the medical imaging analysis of multimodal breast cancer diagnosis was an attention-based transformer model, namely BERT (), RoBERTa (), DistilBERT (), and ALBERT () in this research. Although these models are initially designed to be used with natural language processing (NLP) tasks, to learned in vision-based tasks with an intermediate representation process instead of direct raw image input or the application of standard Vision Transformers such as ViT or Swin Transformer. The image data (histopathology and mammography) were converted into structured and token-like sequences as part of the adaptation. First, the grid-based method was used to divide every image into fixed-size patches. Descriptive features were extracted for each patch using the following techniques: intensity histogram encoding, Gray-Level Co-occurrence Matrix (GLCM)-based texture descriptors, and local binary patterns. These feature vectors were subsequently encoded as one-dimensional (1D) sequences, which largely mimicked the token-embedding format used in NLP tasks. The bidirectional attention of BERT enabled it to better learn inter-patch dependencies, whereas RoBERTa achieved better generalization through training optimizations, such as dynamic masking. The simplified models DistilBERT and ALBERT were created to reduce the representational power and computational cost of the original model while retaining much of its representational power, making them especially applicable to scalable medical image analysis. Such cross-domain adaptation leverages the transfer learning capabilities of trained language models and is consistent with cases where annotated medical imaging data are scarce. These NLP-derived transformers can be trained on sizeable image datasets, unlike vision-specific transformers, which require large image datasets. Moreover, combined features of various imaging modalities were obtained by using a hybrid attention fusion mechanism (–). This combination improved the maintenance of modality-specific data and the model's sensitivity to minor pathological features, such as tumor margins and cell morphology, which are essential for early and precise diagnosis.
3.4 Feature reduction using the MMS algorithm
Deep learning-based medical image analysis is a process reduction, particularly when operating on high-dimensional data (extracted from multimodal data). This not only makes the computation process and the memory consumption much simpler, but also contributes to the enhancement of model generalization, decreases overfitting, and accelerates the training process. In that regard, feature reduction is achieved using the modified mantissa search (MMS) algorithm to optimize the massive feature vectors produced by transformer-based attention networks. MMS: it is a bio-inspired metaheuristic optimization algorithm that simulates the predatory behavior of mantissa, especially their adaptive selection of optimal routes to prey (). The adapted version is more useful in its exploitation and exploration abilities by adding adaptive movement plans and opposition-based learning to balance local and global search. The MMS algorithm is used to determine the significance of each feature via a fitness function, typically associated with classification accuracy or information gain. It recursively chooses a set of features that maximize the separation between classes, unique between caring and malignant lesions in histology or abnormal vs. normal tissue in mammography. The algorithm produces a matrix of B solutions with dimensions (P, C), where P is the search space and C is the number of dimensions per solution. B solutions are generated to ensure that only the most relevant information is relayed to the classifier. In the s function, there is also a vector indicating the position of Mantissa h. On the edge of the optimization problem denotes in Equation 1, a random initialization is employed on a vecto ().
Where Rv1, Rv2, Rv6, Rv7, Rv9, Rv10, Rv12, Rv13, and Rv17 random values gotten by optimal solutions. is the lower and is the upper bound of the j-dimension. The predator in search of prey can be evaluated scientifically as follows () in Equation 2:
where , , , , and random vectors drawn from a uniform distribution between zero and one. Also, the number vector generated by the Lévy-Flight represents |τ2| arbitrary numeral result of normal dissemination with a kind of zero and regular deviation of one in Equation 3.
Randomly selected solutions from the archive are sorted to determine the hth mantissa location () in Equation 4.
The bunkering behavior of mantissas and their prey is described as follows () in Equation 5:
Where are hth represents the current position of the gth dimension of the mantissa in Equation 6.
X represents the number of cycles used to generate the transmission, which involves detecting the speed and range of the victim's strike. When the mantissa reaches the number 0, the mantissa realizes that it is not the right time to attack its prey () in Equation 7.
It indicates the location of the prey and is used to reduce the distance between them and speed up the attack process. A mantissa changes its orientation in response to the exchange of a pair of mantissas randomly selected from the entire population () in Equation 8.
When the function s evaluates to +1, this term describes the latest position of the gth dimension of the mantissa of h, and the two random mantissas, are selected from the current population () (mentioned in Equation 9).
A lower value of variable (m) results in higher exploitation and lower search cost in Equation 10.
where represents the female praying mantissa, represents the fruit chosen randomly from the population that represents the male used by the female for reproduction and consumption. The male-female partners (mentioned in Equation 11) create a new child using the standard intersection operator of genetic operators ():
where denotes the male, μ is the fraction of the male consumed, and the fraction [cos(2πl)] denotes the freedom of the female to replace the male in the objective process. The employed method of feature reduction by MMS is described in Algorithm 1. Using the MMS algorithm for feature reduction enhances the efficiency and accuracy of downstream classification by refining the feature space, improving training convergence, and enabling lightweight models such as LCNN, to make more accurate diagnostic predictions. Not only does this enhance model performance by removing noisy or redundant features, but it also accelerates training and reduces computational expenses, enabling the classifier to function efficiently without sacrificing accuracy.
Algorithm 1
3.5 Feature fusion
Multimodal medical image analysis using feature fusion that seeks to combine supplementary data of various types of data (histology and mammography images) to produce a more informative feature representation. By combining the advantages of each modality, feature fusion enhances diagnostic power and robustness in the most daunting tasks, such as breast cancer classification. Nonetheless, one of the issues that most multimodal fusion systems face is the missing modality problem, where a single modality of data could be missing or be incomplete. To address this, the American zebra optimization (AZO) algorithm is adopted as a successful feature fusion strategy. AZO is a metaheuristic algorithm based on nature that replicates the migration and defense behavior of American zebras when environmental changes occur (). Here, once individual features are identified in both the histology and mammography images and reduced with the MMS algorithm, there is a search for an optimal fusion strategy with the AZO algorithm. This entails selecting the most suitable subset of features for each modality and intelligently combining them to create a strong fused feature vector. It optimizes the various combinations of fusions based on a fitness function, usually classification, to determine which combination of multimodal features is most synergistic (–). They have a life cycle that has separate phases, such as group formation, mate separation, migration, dominance establishment, and resource allocation. American Zebra Optimization Algorithm (AZOA) models such behaviors to solve global optimization problems that mimic the coordination, assignment of roles, and migration among the Meta resources of the zebras (e.g., water bodies) in the same way that feature fusion searches for the best solution. Dominant zebras help the other people to orient toward superior solutions, maintain diversity and exploration, and social boundaries discourage inbreeding and improve convergence. Youthful zebras struggle to get fresh grass and green leaves, and therefore, they rely on the pinnacle of the family. We are told about the feeding of American zebras as follows mentioned in Equations 12 and 13 ().
In AZOA algorithm, the mare is represented by and symbolizes the hth zebra of the gth set, separately, Bg symbolizes the participants in the gth set, r1 directs even casual worth among [−2, 2] that brings the nourishing of zebra at several slants of 360 marks about the lead of the set, r2 indicates the flexible variable that is assessed, r3 indicates an integer between zero and one, the sin and cos activities aid the endeavor of other hth associates from multiple perspectives within the clan's head, is the health score of the hth zebra, and finally, indicates a fresh clinging location when eating mentioned in Equation 14 ().
The extreme repetition and the existing repetition are indicated here by S and s. The reproductive function of zebra fish is given as follows in Equations 15 and 16 ().
where depicts the predicament of the hth collection's small zebra, means point of zebra b from gth set, embodies the point hth objective function, and are the situations of zebra y in the gth set and Kth collection, individually. A set of zebras, tougher than the rest, maintains the optimal solution by fixing the thresholds of maximum rate of change in Equations 17 and 18 ().
where r4 characterizes random numbers in [−2, 2], r5 indicates the flexible limitation which is resolute, r6 symbolize sun varying Radom numbers in [0, 1], Zr designates the aquatic capitals, is the gth collection lead horse existing location, is the gth set lead mare following location, and is its capability worth of pony in gth set () in Equation 19.
In any case, if the leader in the collection develops weak, it is necessary to vary the lead. The leadership transition phase is used to elect a new leader () in Equation 20.
where characterizes jth set leads' horse existing location and is the capability worth of the spearhead stallion. The AZO algorithm simulates zebra behaviors, such as migration toward optimal resources and social structures, to ensure diversity and cooperation (Algorithm 2). In feature fusion, these behaviors help select and combine the best features from histology and mammography images. The living arrangement of the zebra, whereby the strongest ones lead, is similar to the mechanism of determining the synergistic properties so as to achieve better classification. All zebra solutions are attracted to superior feature subsets to enable a comprehensive search over the fusion space. The social boundaries do not allow convergence prematurely; diversity is preserved in the search for features.
Algorithm 2
3.6 Detection and classification of breast cancer
Detection aims to identify possible tumor areas, whereas classification identifies the type of lesion. It is important to note that this process is essential to make a timely diagnosis that greatly enhances treatment outcomes and survival rates (). In this regard, a lightweight convolutional neural network (LCNN) is employed because it is efficient and incurs less computational power. LCNN is a smaller implementation of regular CNN models (), with fewer parameters and layers but still achieving high accuracy. Here, the LCNN accepts the optimized fused features, which were previously obtained in the preprocessing phase, feature extraction phase, feature reduction phase, and fusion, and subjects them to the convolutional layers, which are used to extract spatial hierarchies of patterns. It uses a depth-wise separable convolution layer as the first stage, which takes preprocessed mammography and histology images, followed by convolution layers. The training is made more stable by batch normalization, whereas non-linearity is added by Rectified Linear Unit (ReLU) activation. Maximum pooling reduces spatial dimensionality, and dropout mitigates overfitting (). The last layers are used to classify them into certain groups that include benign, malignant, and particular types of breast cancer. Once the pictures have been shrunk by the convergence layer, less computational power is utilized in the subsequent steps in Equation 21 ().
The variable b characterizes the number of input matrices, P = [P1, P2, ...Pb], is the kth participation matrix, ωd is the intricacy kernel of filter d, , is the medium of the kth sub-convergence seed of the convergence grain, and * designates the convergence procedure mentioned in Equation 22 ().
where ⊗ denotes the convolution process, μd = e[V]D, , γ, and β are the learned rule constants and bias relations, both of which are trainable limits, and e[V] and Var[V] are the error and adjustment of the essentials in direction v, separately. The mesh has the following activation function (Equation 23) ().
Batch Normalization (BN) may fuse homogeneously to a convolution with extra bias because of its homogeneity. It is evident that may be constructed as an extra bias term for for every branch, if a similar productivity is needed for the neural network core ωd. The convolutional layer is where the two BN branches combine (). Set and to be the grains of the conforming riddles in the 3 × 3 intricacy and K × K complication layers, separately, and to be the deep, large neural network kernel following fusion, to be the gained bias term for each filter d in Equations 24, 25 and 26 ().
where ⊕ is the initial sum of kernel limits in the same conditions, that is, the smaller kernel merges with the larger kernel. and ôd are the outputs of the branches of the 3 × 3 complication and the creative K × K complication, separately. The cth element in the output of the compression function z is defined as follows in Equation 27 ().
where , V = [V1, V2, ...Vd]. The output of transform V can be understood as a set of local descriptors whose statistical properties can be generalized to cover the entire image. The final output is obtained by rescaling v with the beginnings in Equation 28 ().
where the channel-wise development among the feature mapping vd and the scalar td is indicated by and fScale. The network can use global information to intelligently highlight relevant information features while suppressing irrelevance (). The information gathered from the collection and the initial tags that accompanied it are here, and the jth classifying neuron's result indicates that the example belongs to the gth classification in Equation 29 ().
is the structure removed from all covers, θ* is the restrictions of all features deposits excluding the classifier, and is all potential network settings that can be trained (). The calculation method of the trial pb is to exploit the later possibility, that is mentioned in Equation 30,
Throughout the exercise stage, the cross-entropy (CE) loss purpose inside the system, cross-wise the whole of the objective function (Equation 31) ().
where B; H{·} is an indicator function, H{true} = 1, and H{false} = 0. Algorithm 3 outlines how LCNN works for cancer diagnosis. Algorithm 3 describes the working process of LCNN for cancer detection. The residual connections form a deep architecture of ResNet, which results in a high number of parameters and computational cost, and the dense connections of DenseNet, which results in high resource demands. Compared to LCNN, the fewer number of layers and separable convolutions in LCNN are employed to reduce both the number of parameters and computational complexity, leading to a shorter inference time and lowering memory consumption, thus making LCNN best suited to a real-time application with resource constraints. It has a shorter training period, which is more applicable to mobile or embedded medical devices, and still achieves high-classification accuracy.
Algorithm 3
4 Results analysis
The results and comparison between the proposed model's effectiveness and the existing models in different simulation scenarios are provided. The benchmark MIAS and BreakHis datasets have been used to validate the effectiveness of the method. The outline and algorithmic processes were implemented in Python using libraries including TensorFlow, Keras, NumPy, and Matplotlib. It was first run on the Google Colaboratory (Google Colab) platform, which was supported by a Google Compute Engine with a backend that is compatible with a graphics card. Such augmentations were random rotations, horizontal and vertical flips, scaling, and brightness. Only the training set could be augmented to avoid data leakage into either validation or testing phases, so that performance measures could indicate the model's actual ability on unseen data.
4.1 Results analysis of feature extraction models
The feature extraction model results that comprise BERTAtt, RoBERTaAtt, DistilBERTAtt, and ALBERTAtt are examined with regard to k-fold cross-validation. The result of the presentation comparison of four transformer-based attention replicates, the MIAS dataset with the Dice Similarity Index, and Jaccard Index, as indicated in Figure 3, demonstrates informative trends. When comparing the BreakHis dataset with 12 histology subtypes using the Dice Similarity Index and the Jaccard Index, it is clear that RoBERTaAtt achieves the highest, or nearly the highest, scores in the majority of the classes. In other classes, ALBERTAtt has shown comparatively lower Dice scores, losing to RoBERTaAtt by 1.232%, BERTAtt by 0.502%, and ALBERTAtt by 2.15%, indicating that it is considerably less capable of localizing features at the pixel level in histology images. In the majority of the categories, DistilBERTAtt has shown a superiority of 0.502% over BERTAtt, which is 0.502% less than RoBERTaAtt; this observation also applies to the other classes. The normal class had an increase of 0.927% over BERTAtt and an even higher increase of 2.101% over DistilBERTAtt. DistilBERTAtt achieved the largest Jaccard score of 93.765% in the PC class, an increase of 0.782% over RoBERTaAtt and 1.616% BERTAtt. ALBERTAtt had always had lower Jaccard values, lagging behind RoBERTaAtt by 2.5–3.2% in a few classes, including LC, PT, and TA, once again confirms its lower segmentation overlap precision (Figure 4). RoBERTaAtt had the best and most balanced results in the BreakHis data, both in Dice and Jaccard indices.
Figure 3
Figure 4
4.2 Impact of feature reduction
Table 3 shows the effect of the proposed model on feature reduction across three large datasets, namely, MIAS, BreakHis, and a custom multimodal dataset. The findings reveal that incorporation of the MMS based feature reduction is a major contributor to the discriminative ability of the transformer-based models like BERTAtt, RoBERTaAtt, DistilBERTAtt, and ALBERTAtt. In the MIAS dataset, the classification accuracy had significant improvement in all classes, with the majority of the models achieving improvement of up to 3%. As an example, the precision of the malignant category with RoBERTaAtt increased from 94.827 to 97.872%, indicating the well-generated feature space achieved with the help of MMS. The same level of performance increase can be observed in the BreakHis dataset, especially in the hard classes such as MC and TA, where the MMS-enhanced models achieved by up to 3.5% higher accuracy.
Table 3
| Class | Without MMS algorithm | With MMS algorithm | ||||||
|---|---|---|---|---|---|---|---|---|
| BERTAtt | RoBERTaAtt | DistilBERTAtt | ALBERTAtt | BERTAtt | RoBERTaAtt | DistilBERTAtt | ALBERTAtt | |
| MIAS dataset | ||||||||
| N | 94.312 | 94.918 | 94.021 | 94.087 | 97.218 | 97.945 | 97.004 | 96.781 |
| BC | 94.415 | 94.702 | 94.335 | 94.103 | 97.304 | 97.836 | 97.123 | 96.911 |
| BM | 94.689 | 94.991 | 94.578 | 94.266 | 97.438 | 97.984 | 97.305 | 97.029 |
| CALC | 94.081 | 94.383 | 94.012 | 94.096 | 97.104 | 97.726 | 96.963 | 96.738 |
| M | 94.598 | 94.827 | 94.452 | 94.209 | 97.351 | 97.872 | 97.143 | 96.981 |
| BreakHis dataset | ||||||||
| A | 94.219 | 94.618 | 94.088 | 94.022 | 97.178 | 97.744 | 96.849 | 96.512 |
| B | 94.302 | 94.727 | 94.163 | 94.054 | 97.309 | 97.803 | 97.098 | 96.784 |
| DC | 94.445 | 94.916 | 94.307 | 94.101 | 97.426 | 97.926 | 97.241 | 96.882 |
| F | 94.618 | 94.995 | 94.382 | 94.278 | 97.504 | 98.061 | 97.374 | 97.142 |
| IS | 94.036 | 94.504 | 94.001 | 94.025 | 97.016 | 97.583 | 96.724 | 96.391 |
| IV | 94.285 | 94.762 | 94.212 | 94.038 | 97.278 | 97.819 | 97.083 | 96.789 |
| LC | 94.489 | 94.847 | 94.308 | 94.149 | 97.401 | 97.921 | 97.215 | 96.976 |
| MC | 94.663 | 94.991 | 94.487 | 94.333 | 97.523 | 98.044 | 97.397 | 97.163 |
| N | 94.712 | 95.003 | 94.532 | 94.397 | 97.598 | 98.072 | 97.502 | 97.194 |
| PC | 94.386 | 94.742 | 94.145 | 94.032 | 97.328 | 97.896 | 97.059 | 96.701 |
| PT | 94.113 | 94.589 | 94.008 | 94.061 | 97.105 | 97.643 | 96.771 | 96.428 |
| TA | 94.652 | 94.993 | 94.414 | 94.255 | 97.509 | 98.067 | 97.358 | 97.126 |
| Multimodal dataset | ||||||||
| Normal | 95.029 | 95.412 | 94.814 | 94.391 | 98.006 | 98.633 | 97.835 | 97.481 |
| Benign | 94.917 | 95.238 | 94.705 | 94.221 | 97.902 | 98.514 | 97.702 | 97.358 |
| Malignant | 94.778 | 95.073 | 94.592 | 94.063 | 97.781 | 98.382 | 97.604 | 97.122 |
Results of with and without feature reduction on MIAS, BreakHis, and multimodality datasets.
4.3 Results of feature fusion models
Table 4 presents a detailed analysis of the various approaches to feature fusion used on the multimodal breast cancer dataset with transformer-based models and lightweight CNNs. The findings clearly indicate that there is a significant benefit in applying feature fusion techniques such as MMS and AZO to the classification system. The models performed at baseline with no feature fusion had the highest accuracy of RoBERTaAtt + LCNN 94.85%. When using MMS, all models became more accurate. The DistilBERTAtt and ALBERTAtt versions also enjoyed the same advantages with the combined MMS + AZO fusion, with improvements of 5.1 and 5.1% in accuracy, respectively.
Table 4
| Feature fusion | Model | Values in % | |||
|---|---|---|---|---|---|
| Accuracy | Precision | Recall | F-measure | ||
| Without feature fusion | BERTAtt + LCNN | 94.210 | 93.800 | 94.500 | 94.150 |
| RoBERTaAtt + LCNN | 94.850 | 94.200 | 94.900 | 94.550 | |
| DistilBERTAtt + LCNN | 93.600 | 93.000 | 93.800 | 93.400 | |
| ALBERTAtt + LCNN | 93.900 | 93.400 | 94.100 | 93.750 | |
| BERTAtt + MMS + LCNN | 95.800 | 95.300 | 96.000 | 95.650 | |
| RoBERTaAtt + MMS + LCNN | 96.300 | 95.700 | 96.400 | 96.050 | |
| DistilBERTAtt + MMS + LCNN | 95.100 | 94.600 | 95.400 | 95.000 | |
| ALBERTAtt + MMS + LCNN | 95.400 | 94.800 | 95.600 | 95.200 | |
| With feature fusion | BERTAtt + AZO + LCNN | 97.500 | 97.100 | 97.600 | 97.350 |
| RoBERTaAtt + AZO + LCNN | 98.100 | 97.700 | 98.200 | 97.950 | |
| DistilBERTAtt + AZO + LCNN | 97.000 | 96.600 | 97.200 | 96.900 | |
| ALBERTAtt + AZO + LCNN | 97.300 | 96.900 | 97.400 | 97.150 | |
| BERTAtt + MMS + AZO + LCNN | 99.100 | 98.800 | 99.200 | 99.000 | |
| RoBERTaAtt + MMS + AZO + LCNN | 99.860 | 99.700 | 99.900 | 99.800 | |
| DistilBERTAtt + MMS + AZO + LCNN | 98.700 | 98.300 | 98.900 | 98.600 | |
| ALBERTAtt + MMS + AZO + LCNN | 99.000 | 98.600 | 99.100 | 98.850 | |
Results comparison of feature fusion models on the multimodality dataset.
4.4 Performance of detection and classification models
The changes in the trend of increasing accuracy across the MIAS, BreakHis, and multimodal datasets, which are presented in Figures 5, 6 clearly show that the trend of accurate change of the model above 500 epochs is significant and uniform, and, therefore, the model has a strong power of learning breast cancer. The training accuracy on the MIAS-dataset has improved significantly at the 500th epoch, reaching 99.85%, a very impressive 12.43% increase. The BreakHis dataset experienced an improvement of 94.45%, as suggested by an improvement of 98.25%. The multimodal data that integrates the data of more than one source showed an even better performance as the training accuracy increased to 99.75% against 50%, and the improvement was 99.5%. For the MIAS dataset, the testing accuracy is 92.81%, compared to 43.69%, a significant gain of 12.35%. The BreakHis dataset was close behind it with a testing accuracy of 95.42%. The multimodal data again proved to be the best at the start at 46 and 91.1%, and the actual increase in data was 98.04. The trends in training and testing loss of the LCNN model with breast cancer diagnosis on the MIAS, BreakHis, and multimodal data (Figure 6) are a good indicator of the efficiency of the model in terms of learning and throughout the 500 training epochs. The training loss on the MIAS dataset started at 0.991 and gradually dropped to 0 during the 500th epoch, showing a 10% drop in loss.
Figure 5
Figure 6
As stated in Figure 7, the accuracy results of the LCNN model on a class-wise basis from 10-fold cross-validation on the MIAS, BreakHis, and multimodal datasets show the reliable performance of the breast cancer diagnosis model. As shown in Figure 8, the MIAS, BreakHis, and multimodal datasets were used to evaluate the performance of the proposed LCNN model in breast cancer detection (Figure 9). As can be seen in Table 5, the proposed LCNN model is better than both the ResNet and the TwinCNN baseline models in terms of training time and inference time on all modalities. The comparison in Table 6 highlights the suitability of the suggested multimodal model, especially the ALBERTAtt + MMS + AZO + LCNN, which is more effective in a variety of data/sets than the previous approaches. Although the previous literature like Joo et al. (), Jiang et al. (), and Misra et al. () conducted experiments and demonstrated a maximum accuracy of up to 94.76% with singular modalities such as MRI, ultrasound, and so on, the model has reached a maximum accuracy of 12% and significant performance with a p-value of 0.021, which indicate the considerable strength of multimodality over single-modality approaches in the process of early detection of breast cancer (Table 7).
Figure 7
Figure 8
Figure 9
Table 5
| Modality | Model | Training time (h) | Inference time (s) | Model size (MB) | Memory usage (MB) |
|---|---|---|---|---|---|
| Histology | Vision transformer | 15.26 | 3.265 | 250 | 450 |
| EfficientNet | 17.859 | 5.632 | 312 | 700 | |
| Multimodal transformer | 12.965 | 2.365 | 157 | 398 | |
| LCNN | 2.556 | 0.085 | 25 | 200 | |
| ResNet | 12.525 | 0.254 | 230 | 800 | |
| TwinCNN | 7.876 | 0.185 | 150 | 500 | |
| Mammography | Vision transformer | 15.478 | 0.858 | 300 | 900 |
| EfficientNet | 21.385 | 0.745 | 350 | 800 | |
| Multimodal transformer | 12.78 | 0.328 | 250 | 500 | |
| LCNN | 2.345 | 0.078 | 25 | 200 | |
| ResNet | 13.256 | 0.345 | 230 | 800 | |
| TwinCNN | 8.423 | 0.247 | 150 | 500 | |
| Multimodality | Vision transformer | 18.347 | 0.964 | 350 | 900 |
| EfficientNet | 13.525 | 0.754 | 258 | 950 | |
| Multimodal transformer | 10.636 | 0.569 | 190 | 750 | |
| LCNN | 3.152 | 0.125 | 30 | 250 | |
| ResNet | 15.068 | 0.357 | 230 | 800 | |
| TwinCNN | 9.559 | 0.229 | 160 | 600 |
Performance comparison between the proposed LCNN and baseline models for various modalities.
Table 6
| Reference | Model | Modalities | Accuracy (mean ±SD) | 95% confidence interval |
|---|---|---|---|---|
| Joo et al. () | Two 3D ResNet-50 networks | MRI + clinical | AUC 82.7% | NA |
| Jiang et al. () | Residual + inception CNN | B-mode, elastic ultrasound | 94.76% | NA |
| Misra et al. () | Single CNN | B-mode + SE ultrasound | Specificity 94.28% | NA |
| Kayikci and Khoshgofaar () | Sigmoid-gated attention + dense | Text + gene expression + CNA | Accuracy 86.35% | NA |
| Muduli et al. () | Single CNN | Gray-scale (MIAS, DDSM, INbreast) | 96.55%, 90.68%, 91.28% | NA |
| Oyelade et al. () | TwinCNN + binary opt | Histology + grayscale | 97.7%, 91.3%, 68.4% | NA |
| This study | BERTAtt + MMS + AZO + LCNN | Histology | 98.50 ± 0.17% | [98.36%, 98.64%] |
| Mammography | 96.67 ± 0.18% | [96.51%, 96.83%] | ||
| Multimodal | 99.12 ± 0.13% | [99.00%, 99.24%] | ||
| This study | RoBERTaAtt + MMS + AZO + LCNN | Histology | 98.65 ± 0.17% | [98.51%, 98.79%] |
| Mammography | 96.85 ± 0.19% | [96.68%, 97.02%] | ||
| Multimodal | 99.25 ± 0.13% | [99.13%, 99.37%] | ||
| This study | DistilBERTAtt + MMS + AZO + LCNN | Histology | 98.74 ± 0.14% | [98.63%, 98.85%] |
| Mammography | 97.12 ± 0.16% | [96.98%, 97.26%] | ||
| Multimodal | 99.32 ± 0.10% | [99.23%, 99.41%] | ||
| This study | ALBERTAtt + MMS + AZO + LCNN | Histology | 98.96 ± 0.13% | [98.85%, 99.07%] |
| Mammography | 97.37 ± 0.16% | [97.23%, 97.51%] | ||
| Multimodal | 99.44 ± 0.09% | [99.35%, 99.53%] |
Comparison of SOTA multimodal breast cancer diagnosis studies, highlighting the model architecture, imaging or data modalities used, and reported performance metrics.
Table 7
| Model 1 | Model 2 | Modality | Accuracy (%) | |||||
|---|---|---|---|---|---|---|---|---|
| Model 1 | Model 2 | t-statistic | p-value | Effect size (Cohen's d) | Wilcoxon p-value | |||
| BERTAtt | RoBERTaAtt | Histology | 98.500 | 98.650 | −0.640 | 0.534 | 0.120 | 0.672 |
| BERTAtt | DistilBERTAtt | Histology | 98.500 | 98.740 | −1.210 | 0.276 | 0.170 | 0.594 |
| BERTAtt | ALBERTAtt | Histology | 98.500 | 98.958 | −2.730 | 0.029 | 0.450 | 0.028 |
| RoBERTaAtt | DistilBERTAtt | Histology | 98.650 | 98.740 | −0.670 | 0.513 | 0.090 | 0.647 |
| RoBERTaAtt | ALBERTAtt | Histology | 98.650 | 98.958 | −2.410 | 0.039 | 0.440 | 0.043 |
| DistilBERTAtt | ALBERTAtt | Histology | 98.740 | 98.958 | −2.110 | 0.048 | 0.340 | 0.050 |
| BERTAtt | RoBERTaAtt | Mammography | 96.670 | 96.850 | −1.430 | 0.181 | 0.150 | 0.380 |
| BERTAtt | DistilBERTAtt | Mammography | 96.670 | 97.120 | −2.090 | 0.045 | 0.230 | 0.039 |
| BERTAtt | ALBERTAtt | Mammography | 96.670 | 97.370 | –−2.530 | 0.024 | 0.300 | 0.023 |
| RoBERTaAtt | DistilBERTAtt | Mammography | 96.850 | 97.120 | −1.150 | 0.258 | 0.140 | 0.438 |
| RoBERTaAtt | ALBERTAtt | Mammography | 96.850 | 97.370 | −1.770 | 0.091 | 0.220 | 0.180 |
| DistilBERTAtt | ALBERTAtt | Mammography | 97.120 | 97.370 | −1.460 | 0.181 | 0.180 | 0.352 |
| BERTAtt | RoBERTaAtt | Multimodality | 99.120 | 99.250 | −1.220 | 0.264 | 0.150 | 0.440 |
| BERTAtt | DistilBERTAtt | Multimodality | 99.120 | 99.320 | −2.050 | 0.045 | 0.230 | 0.048 |
| BERTAtt | ALBERTAtt | Multimodality | 99.120 | 99.438 | −2.860 | 0.021 | 0.400 | 0.015 |
| RoBERTaAtt | DistilBERTAtt | Multimodality | 99.250 | 99.320 | −1.120 | 0.278 | 0.100 | 0.564 |
| RoBERTaAtt | ALBERTAtt | Multimodality | 99.250 | 99.438 | −1.950 | 0.056 | 0.270 | 0.070 |
| DistilBERTAtt | ALBERTAtt | Multimodality | 99.320 | 99.438 | −1.560 | 0.137 | 0.230 | 0.245 |
Statistical analysis of different feature extraction models with MMS + AZO + LCNN for breast cancer prediction.
A comparison of computational metrics for the proposed lightweight and baseline models, such as MobileNetV3 and ShuffleNet, is shown in Table 8 across the histology, mammography, and multimodal datasets. Its findings indicate that all the suggested models make considerable reductions in the parameter count, FLOPs, memory footprint, and inference speed, which is due to their effectiveness and applicability to resource-limited clinical settings. Table 9 shows an ablation study that measures the effect of various modalities, fusion strategies, feature selection, and transformers to predict breast cancer using the MMS + AZO + LCNN framework. Relative to the traditional techniques of concatenation or weighted fusion in the absence of MMS, it can be seen that the AZO + MMS + LCNN model in question outperforms the baseline by 0.22% in accuracy across the modalities, which is a clear consequence of the synergy between the optimal features selective step, adaptive fusion process, and lightweight transformer-based features extraction.
Table 8
| Model | Modalities | Parameter count (M) | FLOPs (B) | Memory footprint (MB) | Inference time (s) |
|---|---|---|---|---|---|
| MobileNetV3 | Histology | 2.912 | 0.219 | 16.423 | 6.213 |
| Mammography | 2.912 | 0.219 | 16.423 | 6.215 | |
| Multimodal | 2.912 | 0.219 | 16.423 | 6.218 | |
| ShuffleNet | Histology | 1.034 | 0.074 | 10.237 | 4.127 |
| Mammography | 1.034 | 0.074 | 10.237 | 4.130 | |
| Multimodal | 1.034 | 0.074 | 10.237 | 4.133 | |
| BERTAtt + MMS + AZO + LCNN | Histology | 0.512 | 0.027 | 4.531 | 1.218 |
| Mammography | 0.512 | 0.027 | 4.531 | 1.220 | |
| Multimodal | 0.512 | 0.027 | 4.531 | 1.223 | |
| RoBERTaAtt + MMS + AZO + LCNN | Histology | 0.518 | 0.028 | 4.542 | 1.226 |
| Mammography | 0.518 | 0.028 | 4.542 | 1.228 | |
| Multimodal | 0.518 | 0.028 | 4.542 | 1.231 | |
| DistilBERTAtt + MMS + AZO + LCNN | Histology | 0.507 | 0.027 | 4.526 | 1.214 |
| Mammography | 0.507 | 0.027 | 4.526 | 1.217 | |
| Multimodal | 0.507 | 0.027 | 4.526 | 1.220 | |
| ALBERTAtt + MMS + AZO + LCNN | Histology | 0.505 | 0.027 | 4.519 | 1.212 |
| Mammography | 0.505 | 0.027 | 4.519 | 1.215 | |
| Multimodal | 0.505 | 0.027 | 4.519 | 1.218 |
Computational metrics of lightweight and baseline models across histology, mammography, and multimodal datasets.
Table 9
| Modality | Feature fusion and selection | Transformer | Accuracy (%) | Precision (%) | Recall (%) | F-measure (%) |
|---|---|---|---|---|---|---|
| Histology | LCNN only | BERTAtt | 98.500 | 98.455 | 98.500 | 98.478 |
| Mammography | LCNN only | BERTAtt | 96.670 | 96.705 | 96.670 | 96.688 |
| Histology + Mammography | Concatenation + LCNN | BERTAtt | 98.900 | 98.953 | 98.900 | 98.927 |
| Histology + Mammography | Weighted + LCNN | BERTAtt | 99.000 | 99.050 | 99.000 | 99.025 |
| Histology + Mammography | AZO + LCNN | BERTAtt | 99.120 | 99.153 | 99.120 | 99.137 |
| Histology + Mammography | AZO + MMS + LCNN | BERTAtt | 99.320 | 99.353 | 99.320 | 99.337 |
| Multimodality | AZO + MMS + LCNN | BERTAtt | 99.320 | 99.333 | 99.320 | 99.327 |
| Multimodality | AZO + MMS + LCNN | RoBERTaAtt | 99.250 | 99.273 | 99.250 | 99.262 |
| Multimodality | AZO + MMS + LCNN | DistilBERTAtt | 99.320 | 99.333 | 99.320 | 99.327 |
| Multimodality | AZO + MMS + LCNN | ALBERTAtt | 99.438 | 99.453 | 99.438 | 99.445 |
| Multimodality | AZO + MMS + LCNN (high λ) | BERTAtt | 99.353 | 99.364 | 99.353 | 99.358 |
| Multimodality | AZO + MMS + LCNN (low λ) | BERTAtt | 99.120 | 99.153 | 99.120 | 99.137 |
| Histology + Mammography | AZO + LCNN (no MMS) | BERTAtt | 99.100 | 99.123 | 99.100 | 99.112 |
| Multimodality | Concatenation + MMS + LCNN | BERTAtt | 99.200 | 99.222 | 99.200 | 99.211 |
| Multimodality | Weighted + MMS + LCNN | BERTAtt | 99.280 | 99.303 | 99.280 | 99.291 |
Ablation study of proposed models showing modality, fusion strategy, and transformer impact on breast cancer prediction metrics.
4.5 Failure case analysis
Although the performance of the proposed multimodal framework comprising BERT-based attention, Modified Mantissa Search (MMS), American Zebra Optimization (AZO), and lightweight CNN (LCNN) was strong, several significant failures were observed during the process. These false classifications highlight the existing difficulties in the diagnosis of breast cancer. Low-contrast histology images: there were cases in which poorly differentiated cellular structures or low staining revealed subtle differences between normal and cancerous tissues. The tissue structures around the lesions were also very similar, which increased the difficulty of extracting the features, especially when the lesions were small or embedded. Ambiguity in multimodal fusion: although the AZO-based fusion tended to improve the classification performance, inconsistent results between the histology and mammography tended to generate misleading results for the fusion strategy. Effects of class imbalance: even after extensive rebalancing, the over-augmented training data still had rare malignant subtypes (e.g., mucinous carcinoma, papillary carcinoma in BreakHis) underrepresented. Figure 10 shows the confusion matrix and sample misclassification images of the MIAS data and visualizes these cases of failure.
Figure 10
5 Conclusion
The study presents a multimodal breast cancer diagnosis model, featuring attention-based transformers (BERT, RoBERTa, DistilBERT, and ALBERT), which are learned to extract all the features of both histology and mammography images. The model uses the modified mantissa search (MMS) algorithm for dimensionality reduction and the American zebra optimization (AZO) algorithm for robust feature fusion and addressing missing modality data. The lightweight convolutional neural network (LCNN) provides accurate classification, making the model useful in real-world, resource-limited clinical settings. The assessment of the model with the MIAS and BreakHis datasets identifies the model as more efficient than state-of-the-art classifications. The proposed BERTAtt + MMS + AZO + LCNN model (99.12% accuracy) is more accurate than the previous multimodal benchmark models (TwinCNN, 68.4% accuracy) by more than 30 points, and RoBERTaAtt and DistilBERTAtt, as well as ALBERTAtt, are even more successful (99.25, 99.32, and 99.438, respectively).
5.1 Limitations and future study
The use of publicly available datasets may not be fully capture the variability and heterogeneity of a real clinical environment. Although the model is effective in managing missing modality data, the performance loss in the presence of several modalities absent or highly corrupted requires further research. The explainability mechanisms are not explicitly included in the current study, which is essential in clinical adoption and trust. To overcome these constraints in future study, there are several pictorial modalities (MRI and ultrasound) to consider; it is better to create explainable parts of AI, and optimize the system to run it on edge and mobile computing. The idea behind such efforts is to bring this research into a practical application as clinical tool that would underpin disease prediction.
Statements
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.
Author contributions
VM: Conceptualization, Writing – original draft, Writing – review & editing, Data curation, Formal analysis, Resources. TA: Formal analysis, Investigation, Methodology, Resources, Writing – original draft, Writing – review & editing. MK: Conceptualization, Project administration, Supervision, Visualization, Writing – original draft, Writing – review & editing. UL: Methodology, Project administration, Resources, Software, Writing – original draft, Writing – review & editing. AM: Data curation, Investigation, Resources, Validation, Writing – original draft, Writing – review & editing. SM: Conceptualization, Formal analysis, Investigation, Visualization, Writing – original draft, Writing – review & editing. LM: Data curation, Investigation, Project administration, Resources, Writing – original draft, Writing – review & editing. MA: Formal analysis, Investigation, Supervision, Visualization, Writing – original draft, Writing – review & editing. LJ: Funding acquisition, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This research has been supported by Princess Nourah bint Abdulrahman University and King Khalid University.
Acknowledgments
This research is supported by Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2026R897), Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia. The authors extend their appreciation to the Deanship of Research and Graduate Studies at King Khalid University for funding this work through Large Research Project under grant number RGP2/3/47.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that Generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1.
ZhangYJiYLiuSLiJWuJJinQet al. Global burden of female breast cancer: new estimates in 2022, temporal trend and future projections up to 2050 based on the latest release from GLOBOCAN. J Natl Cancer Cent. (2025) 5:287–96. doi: 10.1016/j.jncc.2025.02.002
2.
ZhangHPengY. Unique molecular alteration of lobular breast cancer: association with pathological classification, tumor biology and behavior, and clinical management. Cancers. (2025) 17:417. doi: 10.3390/cancers17030417
3.
PolyakovMVSirotinDS. A realistic breast phantom for investigating the features of the microwave radiometry method using mathematical and physical modelling. Technologies. (2025) 13:106. doi: 10.3390/technologies13030106
4.
ParekhRRThakkerM. Machine Vision Based Biomedical System Controlled Using Artificial Intelligence (PhD thesis). Ahmedabad: Gujarat Technological University (2025).
5.
HaSMLeeJMJangMJKimHKChangJM. Breast cancer detection with standalone AI versus radiologist interpretation of unilateral surveillance mammography after mastectomy. Radiology. (2025) 315:e242955. doi: 10.1148/radiol.242955
6.
MarianoLNicosiaLLatronicoABozziniACDominelliVPupoDet al. The role and potential of digital breast tomosynthesis in neoadjuvant systemic therapy evaluation for optimising breast cancer management: a pictorial essay. Br J Radiol. (2025) 98:485–95. doi: 10.1093/bjr/tqae252
7.
SatoMTateishiRZoharYSatoJWatadaniTMoriyamaMet al. Retrospective evaluation of a novel ultrasound-based imaging analysis software for predicting radiofrequency ablation areas. PLoS ONE. (2025) 20:e0317469. doi: 10.1371/journal.pone.0317469
8.
AftabMMehmoodFZhangCNadeemADongZJiangYet al. AI in oncology: transforming cancer detection through machine learning and deep learning applications. arXiv [Preprint]. arXiv:2501.15489 (2025).
9.
ShahidMSImranA. Breast cancer detection using deep learning techniques: challenges and future directions. Multimed Tools Appl. (2025) 84:3257–304. doi: 10.1007/s11042-025-20606-7
10.
AlamTShiaWCHsuFRLiaoLDHussainTKuoYH. A comprehensive review: efficacy of segmentation and machine learning, deep learning techniques in breast tumor detection. Authorea [Preprints] (2025). doi: 10.36227/techrxiv.173609860.03508397/v1
11.
MaheshTRKhanSBMishraKKAlzahraniSAlojailM. Enhancing diagnostic precision in breast cancer classification through EfficientNetB7 using advanced image augmentation and interpretation techniques. Int J Imaging Syst Technol. (2025) 35:e70000. doi: 10.1002/ima.70000
12.
Taghipour-GorjikolaieMGhavamiNPapiniLBadiaMFracassiniABigottiAet al. AI-based hierarchical approach for optimizing breast cancer detection using MammoWave device. Biomed Signal Process Control. (2025) 100:107143. doi: 10.1016/j.bspc.2024.107143
13.
WahedMAAlqaralehMAlzboonMSAl-BatahMS. Evaluating AI and machine learning models in breast cancer detection: a review of convolutional neural networks (CNN) and global research trends. LatIA. (2025) 3:117–117. doi: 10.62486/latia2025117
14.
Al-RaeeiM. The role of artificial intelligence in enhancing breast disease management: early detection and prognostic innovations. Arch Med Health Sci. (2025) 13:475–9. doi: 10.4103/amhs.amhs_351_24
15.
YanLLiQFuKZhouXZhangK. Progress in the application of artificial intelligence in ultrasound-assisted medical diagnosis. Bioengineering. (2025) 12:288. doi: 10.3390/bioengineering12030288
16.
XiongXZhengLWDingYChenYFCaiYWWangLet al. Breast cancer: pathogenesis and treatments. Signal Transduct Target Ther. (2025) 10:49. doi: 10.1038/s41392-024-02108-4
17.
Kumar SahAAgarwalSAbbasAMShalabiMGPrabhakarPKElshaikhRHet al. (2025). Advances in image processing and pattern recognition in cancer detection, prediction, diagnosis, and prognosis. Preprints doi: 10.20944/preprints202503.0055.v1
18.
HamadYAShakirS. Rashid AA, Safonova A, Jagdale R. Optimized machine learning model for breast tumor classification on histopathological images. Int J Comput. (2025) 17:1–10. doi: 10.12785/ijcds/1571107240
19.
DaveDAkhunzadaAIvkovićNGyawaliSCengizKAhmedAet al. Diagnostic test accuracy of AI-assisted mammography for breast imaging: a narrative review. PeerJ Comput Sci. (2025) 11:e2476. doi: 10.7717/peerj-cs.2476
20.
FengKYiZXuB. Artificial intelligence and breast cancer management: from data to the clinic. Cancer Innov. (2025) 4:e159. doi: 10.1002/cai2.159
21.
KhanSUR. Multi-level feature fusion network for kidney disease detection. Comput Biol Med. (2025) 191:110214. doi: 10.1016/j.compbiomed.2025.110214
22.
HekmatAZupingZBilalOKhanSUR. Differential evolution-driven optimized ensemble network for brain tumor detection. Int J Mach Learn Cybernet. (2025) 6447–72. doi: 10.1007/s13042-025-02629-6
23.
KhanSURAsifSBilalO. Ensemble architecture of vision transformer and CNNs for breast cancer tumor detection from mammograms. Int J Imaging Syst Technol. (2025) 35:e70090. doi: 10.1002/ima.70090
24.
KhanSURAsimMNVollmerSDengelA. AI-driven diabetic retinopathy diagnosis enhancement through image processing and salp swarm algorithm-optimized ensemble network. arXiv [Preprint]. arXiv:2503.14209 (2025).
25.
KhanSURAsimMNVollmerSDengelA. Robust and precise knowledge distillation-based novel context-aware predictor for disease detection in brain and gastrointestinal. arXiv [Preprint]. arXiv:2505.06381 (2025).
26.
HekmatAZhangZKhanSURBilalO. Brain tumor diagnosis redefined: Leveraging image fusion for MRI enhancement classification. Biomed Signal Process Control. (2025) 109:108040. doi: 10.1016/j.bspc.2025.108040
27.
YoussefDAtefHGamalSEl-AzabJIsmailT. Early breast cancer prediction using thermal images and hybrid feature extraction-based System. IEEE Access. (2025) 13:29327–39. doi: 10.1109/ACCESS.2025.3541051
28.
MauryaRPandeyNNMahapatraS. BMEA-ViT: breast cancer classification using lightweight customized vision transformer architecture with multi-head external attention. IEEE Access. (2025) 13:44317–29. doi: 10.1109/ACCESS.2025.3547862
29.
RuanZSongCXuPWangCZhaoJChenMet al. Multiparametric ultrasound breast tumors diagnosis within BI-RADS category 4 via feature disentanglement and cross-fusion. IEEE Trans Med Imaging. (2025) 44:3064–75. doi: 10.1109/TMI.2025.3558786
30.
HuangJMaoYDengJYeZZhangYZhangJet al. EMGANet: Edge-aware multi-scale group-mix attention network for breast cancer ultrasound image segmentation. IEEE J Biomed Health Inf. (2025) 29:5631–41. doi: 10.1109/JBHI.2025.3546345
31.
GezimatiMSinghG. Deep learning for multimodal breast cancer characterization with emergence of terahertz and infrared imaging. IEEE Trans Instrum Meas. (2025) 74: 1–14. doi: 10.1109/TIM.2025.3547084
32.
RahmanNOleksyukVWonCH. Bimodal profile diagrams for breast cancer classification using convolution neural network. IEEE Sens J. (2025) 25:10476–85. doi: 10.1109/JSEN.2025.3533385
33.
ChoYMisraSManaguliRBarrRGLeeJKimC. Attention-based fusion network for breast cancer segmentation and classification using multi-modal ultrasound images. Ultrasound Med Biol. (2025) 51:568–77. doi: 10.1016/j.ultrasmedbio.2024.11.020
34.
KansalKKansalK. A Fine-tuned densenet-based diagnosis approach for multi-class breast cancer classification. In: 2025 2nd international conference on computational intelligence, communication technology and networking (CICTN)Ghaziabad: IEEE. p. 974–8 (2025). doi: 10.1109/CICTN64563.2025.10932617
35.
DengTHuangCCaiMLiuYLiuMLinJet al. FedBCD: federated ultrasound video and image joint learning for breast cancer diagnosis. IEEE Trans Med Imaging. (2025) 44:2395–407. doi: 10.1109/TMI.2025.3532474
36.
ChikkalaRBAnuradhaCMurtyPSCRajeswariSRajeswaranNMurugappanMet al. Enhancing breast cancer diagnosis with bidirectional recurrent neural networks: a novel approach for histopathological image multi-classification. IEEE Access. (2025) 13:41682–707. doi: 10.1109/ACCESS.2025.3542989
37.
ShahDAsmatMAbrarMAminF. Bader Fahad Alkhamees, AlSalman H. Enhancing the quality and authenticity of synthetic mammogram images for improved breast cancer detection. IEEE Access. (2024) 12:12189–98. doi: 10.1109/ACCESS.2024.3354826
38.
ShahDAsmatMAbrarM. Reliable breast cancer diagnosis with deep learning: DCGAN-driven mammogram synthesis and validity assessment. Appl Computat Intell Soft Comput. (2024) 2024:1–13. doi: 10.1155/2024/1122109
39.
ShahDUllah KhanMAAbrarMTahirM. (2025). Dual-view deep learning model for accurate breast cancer detection in mammograms. Int J Intell Syst. (2025) 2025:14. doi: 10.1155/int/7638868
40.
ShahDKhanMAUAbrarMTahirM. Optimizing breast cancer detection with an ensemble deep learning approach. Int J Intell Syst. (2024) 2024:5564649. doi: 10.1155/2024/5564649
41.
OyeladeONIrunokhaiEAWangH.A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification. Sci Rep. (2024) 14:692. doi: 10.1038/s41598-024-51329-8
42.
MajidpourJAhmedHAAhmedMHJalalSIArabiH. Applications of GAN models in breast cancer detection: a comprehensive review. Arch Comput Methods Eng. (2025) 33:859–915. doi: 10.1007/s11831-025-10323-7
43.
MajidpourJBeitollahiH. A comprehensive examination of machine learning and deep learning approaches for breast cancer detection, classification, segmentation, augmentation, and feature selection. Arch Comput Methods Eng. (2025) 33:1913–44. doi: 10.1007/s11831-025-10359-9
44.
RashidTAMajidpourJThinakaranRBatumalayMDewiDAHassanBAet al. NSGA-II-DL: metaheuristic optimal feature selection with deep learning framework for HER2 classification in breast cancer. IEEE Access. (2024) 12:38885–98. doi: 10.1109/ACCESS.2024.3374890
45.
MirimoghaddamMMMajidpourJPashaeiFArabalibeikHSamizadehERoshanNMet al. HER2GAN: Overcome the Scarcity of HER2 Breast Cancer Dataset Based on Transfer Learning and GAN Model. Clin Breast Cancer. (2024) 24:53–64. doi: 10.1016/j.clbc.2023.09.014
46.
TomaTABiswasSMiahMSAlibakhshikenariMVirdeeBSFernandoSet al. Breast cancer detection based on simplified deep learning technique with histopathological image using BreaKHis database. Radio Sci. (2023) 58:1–18. doi: 10.1029/2023RS007761
47.
KimY.J., Yoo, E.Y. and Kim, K.G. (2021). Deep learning based pectoral muscle segmentation on mammographic image analysis society (MIAS) mammograms. Precis Future Med.5:77–82. doi: 10.23838/pfm.2020.00170
48.
SinghARAthisayamaniSKarimFKIbrahimAZAlshetewiSMostafaSM.. An enhanced denoising system for mammogram images using deep transformer model with fusion of local and global features. Sci Rep. (2025) 15:6562. doi: 10.1038/s41598-025-89451-w
49.
LiHXuK. Innovative adaptive edge detection for noisy images using wavelet and Gaussian method. Sci Rep. (2025) 15:5838. doi: 10.1038/s41598-025-86860-9
50.
VijayalakshmiDElangovanPSandhyaKumariTKumar NathM. Optimized multi-scale framework for image enhancement using spatial information-based histogram equalization. Imaging Sci J. (2025) 73:176–203. doi: 10.1080/13682199.2024.2343979
51.
LinWHuangQ. Automated surface patch extraction for 3D printing qualification. IEEE Trans Autom Sci Eng. (2025) 22:11419–30. doi: 10.1109/TASE.2025.3535900
52.
GhavidelAPazosP. Machine learning (ML) techniques to predict breast cancer in imbalanced datasets: a systematic review. J Cancer Surviv. (2025) 19:270–94. doi: 10.1007/s11764-023-01465-3
53.
HeBZhaoRTangD. CABiLSTM-BERT: aspect-based sentiment analysis model based on deep implicit feature extraction. Knowl Based Syst. (2025) 309:112782. doi: 10.1016/j.knosys.2024.112782
54.
PookduangPKlangbunrueangRChansanamWLunrasriT. Advancing sentiment analysis: evaluating Roberta against traditional and deep learning models. Eng Technol Appl Sci Res. (2025) 15:20167–74. doi: 10.48084/etasr.9703
55.
ZaimiRSafi EljilKHafidiMLamiaMNait-AbdesselamF. An enhanced mechanism for malicious URL detection using deep learning and DistilBERT-based feature extraction. J Supercomput. (2025) 81:438. doi: 10.1007/s11227-024-06908-x
56.
HuangKLiWGaoF. Barabási-albert model-enhanced genetic algorithm for optimizing LGBM in ship power grid fault diagnosis. Measurement. (2025) 249:116954. doi: 10.1016/j.measurement.2025.116954
57.
XingHWeiWZhangLZhangY. Multi-scale feature extraction and fusion with attention interaction for RGB-T tracking. Pattern Recognit. (2025) 157:110917. doi: 10.1016/j.patcog.2024.110917
58.
LiWChenQGuGSuiX. Object matching of visible–infrared image based on attention mechanism and feature fusion. Pattern Recognit. (2025) 158:110972. doi: 10.1016/j.patcog.2024.110972
59.
ZouYWuLZuoCChenLZhouBZhangH. White blood cell classification network using MobileNetv2 with multiscale feature extraction module and attention mechanism. Biomed Signal Process Control. (2025) 99:106820. doi: 10.1016/j.bspc.2024.106820
60.
GowrisankarLMuraliJGRavichandiranYD. Characterization of integrated nanomaterials using deep learning method-based Mantissa search algorithm. J Comput Electron. (2025) 24:46. doi: 10.1007/s10825-025-02284-8
61.
WongQMChewWTOngSLSeaYWZaidiAFAAhmadNSet al. Three-phase multilevel inverter with minimized low-order harmonics using mantissa search algorithm. in 2025 21st IEEE international colloquium on signal processing and its applications (CSPA). Penang: IEEE. (2025). p. 155–60 doi: 10.1109/CSPA64953.2025.10933374
62.
DivyaMV. Multi-channel mantis search spatial attention convolutional neural network based hyperspectral image change detection. Multimed Tools Appl. (2025) 84, 41193–214. doi: 10.1007/s11042-025-20810-5
63.
Abdel-BassetMMohamedRAbouhawwashM. Fungal growth optimizer: a novel nature-inspired metaheuristic algorithm for stochastic optimization. Comput Methods Appl Mech Eng. (2025) 437:117825. doi: 10.1016/j.cma.2025.117825
64.
Al-MatarnehFM. (2025). Integrating hybrid bald eagle crow search algorithm and deep learning for enhanced malicious node detection in secure distributed systems. Sci Rep. 15:12647. doi: 10.1038/s41598-025-93549-6
65.
IzciDEkinciSJabariMBajajMBlazekVProkopLet al. A new intelligent control strategy for CSTH temperature regulation based on the starfish optimization algorithm. Sci Rep (2025) 15:12327. doi: 10.1038/s41598-025-96621-3
66.
FeofanovVWenSAlonsoMIlbertRGuoHTiomokoMet al. Mantissa: Lightweight calibrated foundation model for user-friendly time series classification. arXiv [Preprint]. arXiv:2502.15637 (2025).
67.
MandourSGamalASleemA. Mantissa search algorithm integrated with opposition-based learning and simulated annealing for feature selection. Sustain Mach Intell J. (2024) 8:5–56. doi: 10.61356/SMIJ.2024.8300
68.
JameelMAbouhawwashM. Multi-objective mantissa search algorithm (MOMSA): a novel approach for engineering design problems and validation. Comput Methods Appl Mech Eng. (2024) 422:116840. doi: 10.1016/j.cma.2024.116840
69.
BinduNVMNassaVKVasukiPManikandanGJeenaRMahaveerakannanR. IoT botnet detection from software defined network using American zebra optimization algorithm with SSRNN-ELM. Int J Inf Technol. (2025) 17:959–967:. doi: 10.1007/s41870-024-02348-1
70.
AliMMMaqsoodF. Enhancing breast cancer diagnosis with channel-wise attention mechanisms in deep learning. Comput Mater Contin. (2023) 77:2699–714. doi: 10.32604/cmc.2023.045310
71.
ZhenfeiWAliMMSahibzadaKIMaqsoodFRehmanNUAftabMet al. Hybrid feature extraction for breast cancer classification using the ensemble residual VGG16 deep learning model. Curr Bioinform. (2025) 20:149–63. doi: 10.2174/0115748936333380240816053223
72.
AkbarAHanSRehmanNUIrshadRAhmedKAliMMAl MazroaA. Reinforcement tokenization and graph convolution for high-precision breast tumor segmentation in DCE-MRI. Biomed Signal Process Control. (2025) 100:106947. doi: 10.1016/j.bspc.2024.106947
73.
ÖzbayFA. An enhanced zebra optimization algorithm with multiple strategies for global optimization and feature selection problems: a hepatocellular carcinoma case study. IEEE Access. (2025) 13:30036–57. doi: 10.1109/ACCESS.2025.3541975
74.
BasheriM. Automated gesture recognition using zebra optimization algorithm with deep learning model for visually challenged people. Fusion Pract Appl. (2024) 16:23–36. doi: 10.54216/FPA.160102
75.
ChapleMDDeotaleNTDeotaleNT. Torque ripple minimization in an 8/6 switched reluctance motor drive: optimization using the AZOA method. Aust J Electr Electron Eng. (2025) 22:27–44. doi: 10.1080/1448837X.2024.2344876
76.
HexSBSW. On the Multimodal Communication and Social Cognition of the Plains Zebra (Equusquagga) (Doctoral dissertation). Princeton University (2025).
77.
FangNXuCGongXWuZ. A new human-based offensive defensive optimization algorithm for solving optimization problems. Sci Rep. (2025) 15:12119. doi: 10.1038/s41598-025-96559-6
78.
VardhiniDANakkaJ. Efficient energy management in microgrid using zebra optimization algorithm. Asian J Water Environ Pollut. (2025) 22:122–33. doi: 10.36922/AJWEP025050030
79.
GuMZhangYWenYAiGZhangHWangPWangG. A lightweight convolutional neural network hardware implementation for wearable heart rate anomaly detection. Comput Biol Med. (2023) 155:106623. doi: 10.1016/j.compbiomed.2023.106623
80.
ShenHWangZZhangJZhangM. L-net: a lightweight convolutional neural network for devices with low computing power. Inf Sci. (2024) 660:120131. doi: 10.1016/j.ins.2024.120131
81.
WieczorekMSiłkaJWozniakMGargSHassanMM. Lightweight convolutional neural network model for human face detection in risk situations. IEEE Trans Industr Inform. (2021) 18:4820–9. doi: 10.1109/TII.2021.3129629
82.
ZhangDHaoXWangDQinCZhaoBLiangLLiuW. An efficient lightweight convolutional neural network for industrial surface defect detection. Artif Intell Rev. (2023) 56:10651–77. doi: 10.1007/s10462-023-10438-y
83.
YunJJiangDLiuYSunYTaoBKongJet al. Real-time target detection method based on lightweight convolutional neural network. Front Bioeng Biotechnol. (2022) 10:861286. doi: 10.3389/fbioe.2022.861286
84.
LiangYLiMJiangCLiuG. CEModule: a computation efficient module for lightweight convolutional neural networks. IEEE Trans Neural Netw Learn Syst. (2021) 34:6069–80. doi: 10.1109/TNNLS.2021.3133127
85.
BaoWYangXLiangDHuGYangX. Lightweight convolutional neural network model for field wheat ear disease identification. Comput Electron Agric. (2021) 189:106367. doi: 10.1016/j.compag.2021.106367
86.
ChenLYaoHFuJNgCT. The classification and localization of crack using lightweight convolutional neural network with CBAM. Eng Struct. (2023) 275:115291. doi: 10.1016/j.engstruct.2022.115291
87.
LinJChenXPanRCaoTCaiJChenYet al. GrapeNet: a lightweight convolutional neural network model for identification of grape leaf diseases. Agriculture (2022) 12:887. doi: 10.3390/agriculture12060887
88.
KhanMAParkHChaeJ. A lightweight convolutional neural network (CNN) architecture for traffic sign recognition in urban road networks. Electronics. (2023) 12:1802. doi: 10.3390/electronics12081802
89.
AhmadMQadriSFQadriSSaeedIAZareenSSIqbalZet al. A lightweight convolutional neural network model for liver segmentation in medical diagnosis. Comput Intell Neurosc. (2022) 2022:7954333. doi: 10.1155/2022/7954333
90.
ChenYChenXLinJPanRCaoTCaiJet al. and Zhang, X. DFCANet: A novel lightweight convolutional neural network model for corn disease identification. Agriculture. (2022) 12:2047. doi: 10.3390/agriculture12122047
91.
HariPSinghMP. A lightweight convolutional neural network for disease detection of fruit leaves. Neural Comput Appl. (2023) 35:14855–66. doi: 10.1007/s00521-023-08496-y
92.
LiuWZhouLChenJ. Face recognition based on lightweight convolutional neural networks. Information. (2021) 12:191. doi: 10.3390/info12050191
93.
JooSKoESKwonSJeonEJungHKimJ-Yet al. Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer. Sci Rep. (2021) 11:1. doi: 10.1038/s41598-021-98408-8
94.
JiangMLeiSZhangJHouLZhangMLuoY. Multimodal imaging of target detection algorithm under artificial intelligence in the diagnosis of early breast cancer. J Healthc Eng. (2022) 2022:9322937. doi: 10.1155/2022/9322937
95.
MisraSYoonCKimKJManaguliRBarrRGBaekJet al. Deep learning-based multimodal fusion network for segmentation and classification of breast cancers using B-mode and elastography ultrasound images. Bioeng. Transl. Med. (2022) 8:6. doi: 10.1002/btm2.10480
96.
KayikciSKhoshgofaarTM. Breast cancer prediction using gated attentive multimodal deep learning. J Big Data. (2023) 10:62. doi: 10.1186/s40537-023-00749-w
97.
MuduliDDashRMajhiB. Automated diagnosis of breast cancer using multi-modal datasets: a deep convolution neuralnetwork based approach. Biomed. Signal Process. Control. (2022) 71:108235. doi: 10.1016/j.bspc.2021.102825
Summary
Keywords
cancer detection, clinical image analytics, cross-modal learning, image-based disease prediction, medical decision support
Citation
Malik V, Alsubait T, Khan M, Lakhina U, Menshawi A, Mehla S, Mukhija L, Altaf M and Jamel L (2026) Multimodal breast cancer diagnosis using feature fusion and deep learning. Front. Med. 13:1877662. doi: 10.3389/fmed.2026.1877662
Received
10 May 2026
Revised
06 June 2026
Accepted
17 June 2026
Published
21 July 2026
Volume
13 - 2026
Edited by
Sunyoung Jang, SUNY Upstate Medical University, United States
Reviewed by
Mustafa Güler, Istanbul University-Cerrahpasa, Türkiye
Marina Artiyasa, Nusa Putra University, Indonesia
Updates
Copyright
© 2026 Malik, Alsubait, Khan, Lakhina, Menshawi, Mehla, Mukhija, Altaf and Jamel.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Mudassir Khan, mudassirkhan12@gmail.com; Meteb Altaf, maltaf@kacst.gov.sa
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.