Identifying key features for phishing website detection through feature selection techniques

Abstract

Over the past few years, phishing has evolved into an increasingly prevalent form of cybercrime, as more people use the Internet and its applications. Phishing is a type of social engineering that targets users' sensitive or personal information. This paper seeks to achieve two main objectives: first, to identify the most effective classifier for detecting phishing among 40 classifiers representing six learning strategies. Secondly, it aims to determine which feature selection method performs best on websites with phishing datasets. By analyzing three unique datasets on phishing and evaluating eight metrics, this study found that Random Forest and Random Tree were superior at identifying phishing websites compared with other approaches. Similarly, GainRatioAttributeEval, along with InfoGainAttributeEval, performed better than the five alternative feature selection methods considered in this study.

1 Introduction

Due to the widespread use of online services like e-commerce and social media and the increased access afforded by the Internet, users are increasingly susceptible to cyberattacks targeting sensitive information, such as usernames or credit card details. One popular method used by attackers is called phishing, which uses fraudulent websites that appear authentic and trick individuals into divulging their private data (Athulya and Praveen, 2020). This can be accomplished using email or text messages designed solely for this purpose; even communication between clients and companies may contain such deceptive links. Typically motivated by financial gain, malware infections on user machines, or identity theft, most phishing attempts involve these motives.

Recent findings indicate a dramatic increase in unique reported instances, exceeding 199 thousand detections in December 2020 alone—an alarming statistic compared with the Anti-Phishing Working Group's results from previous years (APWG, 2021). Moreover, since the early days of the pandemic in March last year, when global COVID-19 fears were high, scammers have frequently issued phony certificates containing the words “COVID” or “corona.” These scammers have increasingly relied on digital certification policies and HTTPS protocols rather than on traditional tactics (Warburton, 2020).

Broadly, there are two ways to identify phishing: through user knowledge or anti-phishing software. Due to the realism of phishing emails and websites, many users find it challenging to detect them. Consequently, accurate software solutions for detecting these threats have become increasingly necessary. Software-based detection strategies include blocklisting, heuristics, and machine learning (Athulya and Praveen, 2020). Previous studies using machine learning often relied on numerous features to achieve high accuracy; however, extracting these features is not always possible in real-time scenarios, requiring more resilient solutions.

The purpose of this paper is to support the worldwide effort to combat phishing scams by leveraging advanced machine learning techniques to predict fraudulent websites accurately.

Numerous classification models have been proposed and employed to identify phishing websites, claiming superiority over other approaches (Alazaidah et al., 2018). Moreover, this study aims to determine the most suitable classification method (classifier) for phishing datasets. To obtain a comprehensive overview of the findings, more than 40 classifiers across six learning strategies are evaluated using several metrics, including accuracy, precision, recall, and F1-measure.

Feature selection is one of several necessary preprocessing steps when creating any machine learning (ML)-based learning model. Its purpose is to identify relevant features that aid in constructing intended models by selecting non-redundant consistent attributes (Alluwaici M. et al., 2020). The feature selection procedure always prioritizes characteristics that closely align with the objective qualities of the dataset's attributes (Alluwaici M. et al., 2020).

To achieve the goal, 40 classifiers from six well-known learning strategies were selected for assessment. The evaluation phase encompasses eight diverse, commonly used metrics, including accuracy, precision, recall, and AUC. Besides, it aims to implicitly identify the best learning strategy among those considered using four distinct evaluation indicators: accuracy, precision, recall, F-Measure, MCC, PRC area, and ROC-Area (receiver operating characteristics).

The second objective of this study is to determine the optimal feature selection technique for predicting phishing websites. To achieve this objective, five commonly used feature selection methods were assessed and compared with identical classifiers used in the first goal across three evaluation metrics: accuracy, precision, and recall.

The remaining sections of the paper are structured as follows: Section 2 reviews the current literature on implementing ML techniques for phishing. In Section 3, we present our methodology, results, and discussion. Finally, concluding remarks and future directions are proposed in Section 4.

2 Related research

In this section, we examine prior research that has used machine learning techniques to detect phishing. In their study on fuzzy rough set feature selection, Zabihimayvan and Doran (2019) used multiple features to construct a model intended to detect fraudulent activity attempts by criminals intentionally sidestepping existing anti-phishing measures on Iranian banking websites. They trained and tested their system using fuzzy experts, achieving an accuracy of around 88%. Still, they acknowledged that there is scope for optimizing feature selection during the training/testing phases, which could increase predictive power while reducing prediction time.

A different approach was taken by Cui (2019), leveraging data analytics across multiple search engines as its source material identifying idle URLs previously exposed through popular searches or internal links shared between identified related sites along with additional input from frequently visited pages from URL structural similarity evaluation utilizing twelve (12) distinct characteristics depicting intra-relatedness/popularity degrees among entered site structures and components; altogether building classifiers resulting overall classification rates exceeding nearly ninety-five percent success rate coupled at about one-and-a-half false positives per classifying session—however may overlook obfuscated content when analyzing linked materials such as domain name variations generated algorithmically/hosted solely off malicious web domains themselves/limited character string-denser link shortening platforms commonly employed against undetected trapping activities.

Gandotra and Gupta (2021) compared various ML techniques using a 30-feature set comprising approximately 5,000 phishing websites and over 6,000 authentic webpages. This study found that incorporating feature selection enables faster creation of effective phishing detection models while maintaining accuracy. Notably, their results highlight that random forest classification (RF) achieves superior accuracy regardless of whether feature selection is used.

Detecting phishing attempts using ML often involves analyzing lexical features of URLs. This method, pioneered by Abutaha et al. (2021), was intended for use as a browser plug-in that scrutinizes a webpage's URL to alert users before they visit it. To test the efficacy of this technique, over one million legitimate and fraudulent URLs were used in experiments that extracted 22 variables, which were reduced to 10 key ones.

Findings revealed an accuracy rate of 99.89% when combined with SVM classification, surpassing the RF classifier, gradient boosting classifier (GBC), and neural network approaches trialed alongside it.

Chapla et al. (2019) proposed a fuzzy-logic-based framework for detecting phishing websites, using a dataset containing both legitimate and fraudulent URLs. The model achieved 91.4% accuracy but was limited by a small sample size of 1,000 features focused solely on URL-related attributes; as a result, it is less effective at identifying other bypass techniques.

The author in Tan (2018) improved the performance of their phishing URL detection system by using lexical features. A model proposed in Chiew et al. (2019) achieved high accuracy while being independent of third-party services and source code analysis, thereby requiring less processing time. Meanwhile, authors in Abdelhamid et al. (2014) sought to enhance the accuracy of phishing detection systems through feature selection and an ensemble learning approach, achieving 95% accuracy in their experiments.

In yet another effort detailed in article (Su et al., 2023), an innovative approach used seven distinct machine learning algorithms for detecting potential risks posed by various unwanted attacks, including those utilizing zero-day exploits, with selected implemented security features overcoming issues such as language dependency or reliance on external parties during real-time monitoring operations without issue!

Rahman et al.'s research also explored machine learning classifiers' ability concerning various datasets related to phishing practices (Gandotra and Gupta, 2021). This initiative likewise demonstrated equivalent results, with gradient boosting trees (GBT) outperforming all metrics and achieving higher success rates than other methods, such as random forest (RF).

OFS-NN was proposed by Sahingoz et al. (2019) and combines optimal feature selection with a neural network to mitigate overfitting by using a new metric, the feature validity value (FVV). Experimental results on two datasets demonstrated that FVV outperformed information gain and optimal feature selection across various categories, including specific features such as abnormal, domain, HTML/JavaScript, and even address-bar features. The OFS-NN model achieved an overall accuracy of 0.945; however, among the feature types used for detection, the highest accuracy, 0.903, was observed with “address bar,” while the lowest, around half accurate at 0.562, was observed with HTML/JavaScript.

Another phishing detection system was introduced by Sahingoz et al. (2019), which comprises 40 NLP-based traits, along with additional hybrid characteristics derived from word vectorization, totaling about 1,700 more relevant aspects.

In their study, the authors compared seven distinct algorithms offering diverse options but ultimately determined random forest's implementation made using solely natural language processing delivered the most superior performance, scoring almost perfect precision statistics, peaking up to staggering score amounts nearing practically zenith level, i.e., tracing fraudulent websites based upon this criterion managed to reach correct outcomes nearly 98 percent times—rendering maximum efficacy amongst all tested methodologies researched herein.

In Alazaidah et al. (2024), the authors conduct a comparative analysis of 24 classifiers across two datasets using several evaluation metrics. The results revealed the superiority of the random forest, filtered classifier, and J48 classifiers. The author suggests considering additional classification models with different learning strategies, as well as more datasets and evaluation metrics.

The research in Aljofey et al. (2025) proposed a hybrid methodology that combines URL character embeddings with several handcrafted features. Three datasets were used in this work: two are benchmarks, and the third was collected and preprocessed by the authors. The results showed excellent performance across accuracy and other evaluation metrics.

Several deep learning optimization techniques were used in Barik et al. (2025) to improve phishing prediction on websites. The authors used standardization and variational autoencoder techniques in the preprocessing step, and an enhanced grid search optimizer to improve accuracy. The results showed superior performance across accuracy, precision, and F1-score metrics. Unfortunately, utilizing one dataset only does not help in generalizing the finding of the conducted research. Several other related research works could be find in Ganjei and Boostani (2022), Gareth et al. (2023), (Ni et al. (2022), Nti et al. (2022), Rashid et al. (2020), Srivastava (2014), Ubing et al. (2019).

Throughout this literature review, random forests perform comparatively better than their counterparts in detecting phishing using machine learning. However, gradient boosting machines (GBM) were frequently not a subject of comparison, affecting project linearity and requiring deeper exploration, while lackluster attempts, such as minimal input/no-noise coefficient data filtering, were still in early phases, indicating that extensive future research remains vital.

3 Research methodology

The methodology employed in this paper is depicted in Figure 1. The first phase in Figure 1 involves collecting the datasets. Afterward, the datasets are cleaned and preprocessed. Then, several feature selection techniques are trained on the pre-processed datasets and evaluated. Next, 40 classification models are trained on the datasets using the selected features from the previous step. These classifiers are compared using several well-known evaluation metrics.

Figure 1

The description of three website phishing datasets used in this research is provided in Section (A), while Sections (B, C, and D) evaluate the performance of feature selection and machine learning algorithms on these datasets.

Moreover, Section 4 considers which classification model is most appropriate for phishing website datasets. Therefore, three datasets are considered in this section.

In addition to that, Section 5 evaluates and identifies the best among five renowned feature selection methods, as well as identifying the most efficient classifiers, which are outlined in Section 6 before finally discussing primary results obtained from these sections' analyses at length.

In addition, 40 classifiers from six learning strategies are evaluated and contrasted in terms of their predictive efficacy across the three datasets under consideration. These examined classifiers encompass:

Random tree, random forest, REPTree, DecisionStump, HoeffdingTree, LMT, J4B, and REPTree from the Trees learning strategy; BayesNet, NaiveBayesUpdateable, and NaiveBayes from the Bayes learning strategy. Logistic, MultilayerPerceptron, SimpleLogistic, VotedPerceptron, and SMO from the Functions strategy. IBK, KStar, and LWL from the lazy learning strategy; AdaBoostM1, AttributeSelectedClassifier, Bagging, ClassificationViaRegression, FilteredClassifier, IterativeClassifierOptimizer, LogitBoost, MultiClassClassifier, MultiClassClassifierUpdateable, RandomCommittee, RandomizableFilteredClassifier, RandomSubSpace, Stacking, WeightedInstancesHandlerWrapper, vot, and CVParameterSelectionr from the Meta learning strategy; DecisionTable, JRip, OneR, PART, and ZeroR learning strategy. Finally, InputMappedClassifier from the misc learning strategy.

The WEKA software's default settings are utilized for all classification models. This renowned data analysis tool, also known as (Waikato Environment for Knowledge Analysis), is frequently used (Rao et al., 2020). The outcome validation process uses 10-fold cross-validation to ensure the results.

To compare the considered classification models, six performance metrics were analyzed: Accuracy, precision, recall, F-measure, MCC (Matthews correlation coefficient), ROC Area, and PRC Area. Next up are the equations needed to calculate these metrics.

Accuracy is a metric that indicates how frequently a machine learning model predicts the correct outcome. The number of right guesses divided by the total number of forecasts yields accuracy (Alzyoud et al., 2024; Alazaidah et al., 2023a,b).

Precision is a metric that indicates how often a machine learning model correctly predicts the positive class. Precision can be calculated as the number of correct positive predictions (true positives) divided by the total number of positive predictions made by the model (including true and false positives).

Recall is a metric that indicates how often a machine learning model accurately detects positive examples (true positives) from all actual positive samples in the dataset. Divide the number of true positives by the number of positive cases to determine recall. The latter includes true positives (correctly identified cases) and false negatives (missed cases) (Al-Batah et al., 2023; Pei et al., 2022).

MCC is the best single-value classification metric for summarizing a confusion or error matrix. A confusion matrix has four entities:

• True positives (TP)

• True negatives (TN)

• False positives (FP)

• False negatives (FN)

And is calculated by the formula:

F-measure is an alternative machine learning evaluation metric that assesses the predictive skill of a model by elaborating on its class-wise performance rather than its overall performance, as done by accuracy. The F1 score combines two competing metrics—precision and recall—of a model, making it widely used in recent literature.

ROCArea: a metric that graphically assesses classifier performance across varying thresholds by plotting the false positive rate on the x-axis and the true positive rate on the y-axis.

True Positives (TPs): instances in which the model correctly identifies examples.

True Negatives (TNs): represent cases where the model correctly recognizes and labels negative examples.

False Positives (FPs): occur once the model mistakenly identifies examples as positive. In words, these are instances where negative examples are mistakenly labeled as “positive.”

False Negatives (FNs): arise when positive examples are incorrectly classified as negative. These are cases in which positive examples are incorrectly labeled as “negative.”

3.1 Description of datasets

In the study, three datasets are available for download from the UCI repository. The first dataset, a binary classification set, contains 11,055 instances with 30 integer features. Most of these features are binary. On the other hand, the second dataset comprises three class labels, supports multiclassification, and provides nine integer-type features and 10,000 examples; the third dataset comprises two class labels, consists of 13 integer-type features, and provides 2,670 instances. Table 1 presents the distinguishing qualities of both sets for quick reference. This research focuses on the first two datasets, which are the largest and have 3 class labels, while the third dataset is relatively small with only two classes: selection and understanding.

This step focused on collecting datasets and understanding the attributes. Three datasets, denoted DS1, DS2, and DS3 (Su et al., 2023; Alluwaici M. A. et al., 2020), and DS3 (Mohammad et al., 2015), were selected, as they have different numbers of features and only some are common. Table 2 summarizes the feature categories across the three datasets. DS1, DS2, and DS3 contain both internal features (i.e., derived from webpage URLs and HTML/JavaScript source code available on the webpage itself) and external features (i.e., obtained from querying third-party services such as DNS, search engines, and WHOIS records). DS2 only contains internal features (Mohammad et al., 2015).

3.2 Data preparation

Data preprocessing involves operations such as handling missing values, removing outliers, and eliminating redundant information. As stated in reference (Alazaidah et al., 2023a), the DS1, DS2, and DS3 datasets were free of missing data but required cleaning before use. For instance, the HttpsInHostname attribute in DS3 had all values set to 0, making it unnecessary for analysis.

To identify common attributes across these datasets (DS1-DS2-DS3), the authors checked their descriptions available in references (Mohammad et al., 2015) and (Alzyoud et al., 2024). The authors' citations for each dataset feature significantly simplified this preprocessing step.

It was noted that some feature pairs captured similar information expressed in different formats, such as UrlLength, which is numeric, and its counterpart, “UrlLengthRT,” which is categorical. In cases where those occurred only once, they would be mapped to the same variable, URL_Length, found solely in dataset DP1; otherwise, they would remain separate. Ultimately, after scrutinizing these intricate details across variables, we discovered a match between 18 key attributes among the three aforementioned sources (as shown in Table 3).

3.3 Feature selection

The significance of independent features was assessed using P-values, with a threshold of 0.05 to identify statistically significant features.

To begin with, the Spearman rank-order correlation method assessed collinearity between feature pairs. In Figure 2, we show the correlation matrix for the DS1-2-3 matching feature, with the pop-up window and on-mouse-over having the highest observed value at 0.73, followed by the pop-up window and favicon pair, which had a corresponding score of 0.66. Most pairs showed small or negligible correlations.

Figure 2

To identify multicollinearity—where three or more variables converge even when no two have high individual similarities—the Variance inflation factor (VIF) scores were used (Ubing et al., 2019).

Each trait received its VIF rating calculated as follows:

Unadjusted coefficient of determination for regressing the ith independent variable on the remaining ones.

Based on VIF analysis, in addition to p-values, the combined DS1-2-3 data identified 15 features as noteworthy and independent.

This process used various Python packages, including statsmodels to calculate VIF scores and p-values, scikit-learn to build logistic regression models, and Matplotlib and Seaborn to generate visualizations.

For the feature selection and ranking step, four techniques have been considered and evaluated. The first technique is called Correlation Attribute Evaluator (CAE). CAE measures the linear correlations between the input features and the output feature (class) and is usually implemented using Pearson's correlation coefficient. The second technique is the Gain Ratio Attribute Evaluator (GRAE). This technique assesses feature significance by measuring each feature's gain ratio relative to the class label. The third technique is dubbed the Information Gain Attribute Evaluator (IGAE). IAGE measures how a feature is worth based on the value of information gain for this feature with respect to the class label. The last technique is the Principal Components Analysis (PCA). This technique aims to reduce data dimensionality by transforming a large dataset into a smaller one with low-correlated features.

4 Comparative analysis amongst the classification models in the domain of website phishing

This section describes the process of determining the ideal classification model for phishing datasets. To attain this objective, three distinct sets of data cognate to phishing have been analyzed in detail. Table 4 outlines the highlighted attributes associated with these datasets, all of which can be obtained from the UCI repository with ease.

The results of using 40 classifiers on the phishing website dataset 1 (DS1) are presented in Table 4 and analyzed with respect to accuracy and pre-session metrics. The data reveal that IBK achieves the highest accuracy, whereas RandomCommittee achieves outstanding accuracy and precision.

Evaluating learning strategies indicates that Lazy achieves optimal accuracy, while RandomCommittee yields superior precision.

The Recall and MCC metric results for the phishing website dataset after applying 40 classifiers are outlined in Table 5. The table shows that random forest classification models have produced superior results when evaluated against these criteria.

Additionally, Tree outperforms other learning strategies on both precision and MCC metrics in this dataset (DS1).

A comparative analysis of 40 classifiers on the phishing dataset, in terms of accuracy and precision, is presented in Table 5.

Random forest outperforms the other considered classifiers in accuracy and precision on the phishing dataset (DS1), as shown in the table.

Moreover, among the eight learning strategies assessed using these two measures, the Functions Tree strategy yields better outcomes than its counterparts.

The precision metrics obtained from applying 40 classifiers to the phishing dataset are shown in Table 6. According to the table, among all classification models, the RandomCommittee learning strategy achieves the highest precision. Similarly, for the Random Forest metric, based on Table 6 and the Trees learning strategy, we can see that the Random Forest classification model delivers superior outcomes.

In conclusion, regarding optimizing the precision metrics shown in Table 6, function learning is our preferred approach, yielding the best results compared to other available strategies.

In Table 7, the random forest classification models achieve the best recall and MCC results on the phishing dataset (DS1). The random forest classifier belongs to the Tree learning strategy.

Moreover, regarding the best learning strategy, Table 7 shows that the tree learning strategy achieves the best results for the recall and MCC metrics.

According to Table 8, the classifier in the tree learning strategy, random forest, has the highest precision metric. Additionally, when it comes to the accuracy metric and other compared classifiers, this same classifier performs best again. Furthermore, among the seven considered learning strategies, Tree stands out as achieving superior results across comparisons.

The outcomes of the 40 classifiers applied to the phishing website dataset, with respect to recall and MCC, are shown in Table 9.

Analysis of Table 9 indicates that, among all classification models, the random tree classifier achieved the highest accuracy and precision on the given dataset (DS2). Additionally, compared with other learning strategies exhibited by the remaining classifying algorithms in Table 9, the tree strategy was found to outperform others in terms of efficient data processing.

The results from implementing 40 classifiers on the phishing website dataset (DS2) are shown in the table, including accuracy and precision metrics.

The Random Forest model, a tree-based learning strategy, achieves higher accuracy and precision than other classification models, as shown in Table 10.

Besides, when focusing solely on optimizing the precision metric through a strategic approach perspective, adopting the tree learning strategy can be highly effective.

Table 11 presents the results of applying 40 classifiers to the phishing website dataset (DS2), focusing on recall and MCC.

According to Table 11, the Random Tree classifier performs exceptionally well on the Recall metric. At the same time, the Random Forest model achieves the best MCC among all considered classification models.

Furthermore, Trees prove themselves to be an exceptional learning strategy, producing superior output compared to seven alternative strategies from both recall and MCC perspectives.

The results obtained from the 40 classifiers applied to the phishing website dataset (DS2) for the recall and MCC metrics are presented in Table 12. Random tree classifier demonstrates superior recall, while the random forest and the random tree stand out with exceptional performance on MCC among the classification models considered. Also, compared to the seven learning strategies under review, Trees shows better results for both the Recall and MCC measures.

Additionally, these two classifiers have been most effective on this dataset, as indicated by their respective evaluation scores in Table 12.

The accuracy and precision metrics for the phishing dataset (DS2) were evaluated using 40 classifiers, and the results are presented in Table 13.

From the table, it is evident that the IBK model under the lazy learning strategy, along with the random tree model under the tree learning approach, achieved the highest accuracy and precision values.

Furthermore, based on the findings in Table 13 regarding optimizing the precision metric for the Learning Strategy factor, Tree Learning should be selected for its superior performance.

Table 14 displays the results of forty classifiers applied to a dataset (DS3) containing phishing websites. The evaluation metrics were F-measure and ROC area. Among these, the random forest classifier showed exceptional performance in both F-measure and ROC, compared with all seven learning strategies under scrutiny. Additionally, Trees displayed better outcomes than others on both measures.

From Table 14, the scores for each evaluation method indicate that, among the classifiers tested, they were most efficient on this dataset when compared with the other methods employed herein.

The results of running 40 classifiers on the phishing website dataset (DS3) are shown in Table 15, including F-measure and ROC metrics. According to the table, the random forest classifier outperforms other classification models on both F-measure and ROC for this dataset.

Additionally, Trees is the most effective learning strategy for achieving high marks on both evaluation measures among the seven strategies considered here.

When 40 classifiers were applied to the phishing website dataset (DS3), Table 16 shows the results for both the F-measure and ROC metrics. According to this table, among the considered classification models, the random forest classifier achieves superior results in terms of F-measure and ROC on the same dataset. Besides, Trees, as a learning strategy, demonstrates top-notch performance across both evaluation criteria when juxtaposed with seven other strategies.

The results of applying 40 classifiers to the phishing website dataset, with respect to F-measure and ROC metrics, are shown in Table 17. The random forest classifier outperforms the other considered classification models on both measures for this dataset, as shown in Table 17.

Notably, Trees proves superior as a learning strategy, based on its performance across all evaluation criteria among the seven strategies compared here, particularly on F-measure and ROC metrics.

The results of using 40 classifiers on the phishing website detection dataset (DS3) are depicted in Table 18 and analyzed using accuracy and pre-session metrics. The data reveal that Random Forest achieves the best F-MEASURE and ROC scores, while other top-performing methods, Random Committee and J4B, also achieve outstanding F-MEASURE and ROC scores.

Evaluating the learning strategy indicates that Tree attains optimal results for F-MEASURE and ROC METRICS.

The summary of the comparative analysis of 40 classifiers across three datasets, as presented in Tables 4–18, is shown in Table 19. In this table, “RC” refers to a random committee, “RF” denotes random forest, and “RT” stands for random tree.