Self-decisive algorithm for unconstrained optimization problems as in biomedical image analysis

Abstract

This study describes the construction of a new algorithm where image processing along with the two-step quasi-Newton methods is used in biomedical image analysis. It is a well-known fact that medical informatics is an essential component in the perspective of health care. Image processing and imaging technology are the recent advances in medical informatics, which include image content representation, image interpretation, and image acquisition, and focus on image information in the medical field. For this purpose, an algorithm was developed based on the image processing method that uses principle component analysis to find the image value of a particular test function and then direct the function toward its best method for evaluation. To validate the proposed algorithm, two functions, namely, the modified trigonometric and rosenbrock functions, are tested on variable space.

1. Introduction

Imaging informatics plays a significant role in medical and engineering fields. In the diagnostic application software, during the segmentation procedure, different tools are used to interact with a visualized image and a graphical user interface (GUI) is used to parameterize the algorithms and for the visualization of multi-modal images and segmentation results in 2D and 3D. Hence, different toolkits, such as Medical Interaction Toolkit (Wolf et al., 2005) or the MevisLab (Ritter et al., 2011), are used to build appropriate GUIs, yielding an interface to integrate new algorithms from science to application. To produce better results, we used different sensors to generate a perfect image for denoising the tasks, thus focusing on massive denoising, as sometimes it is difficult for humans and computers to recognize the image. Furthermore, different filters (Wang, 2018) are used to get better denoising. The commonly used mathematical method for this purpose is the quasi-Newton method, which is preferred due to its better performance than other classical methods. Schröter and Sauer (2010) investigated quasi-Newton algorithms for medical image registration. Mannel and Rund (2021) implemented a hybrid semi-smooth quasi-Newton method for the non-smooth optimal control problems and proved its efficiency. Recently, Moghrabi et al. (2022) derived self-scaled quasi-Newton methods and proved their efficiency over the non-scaled version. Among the quasi-Newton methods, Broyden-Fletcher-Goldfarb-Shanno (BFGS) is a widely used method due to its better performance. Hence, this motivated the researchers to further develop these methods, one such method is the two-step quasi-Newton method.

The two-step quasi-Newton methods are considered to minimize unconstrained optimization problems.

The multi-step quasi-Newton methods, which were introduced by Ford and Moghrabi (1994, 1993), obtained the best results over the single-step quasi-Newton method. In the single-step quasi-Newton method, updation of Hessian approximation (B_i+1) is required to satisfy the secant equation

where s_i is the step size in the variable space x_i

and y_i is the step size in gradient space g(x_i), such as

The quasi-Newton equation must satisfy the true Hessian M_i+1 of Newton equation which is defined as follows:

In the case of the two-step quasi-Newton methods, the secant equation (2) is replaced by

or

which is derived by interpolating the quadratic curve x(τ) in variable space and g(τ) in gradient space

for the purpose of interpolating the Lagrange polynomial that is found suitable in Jaffar and Aamir (2020). Therefore,

the derivatives of Equations (10) and (11) are defined as

the above relations obtained are substituted in Equation (14), which is a two-step form of Equation (5)

The new secant condition for the two-step quasi-Newton method is obtained in the form of Equation (6/7), which should be satisfied by the updated Hessian approximation B_i+1. The value of α_i in Equation (6) is given by

where

Hence, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) formula for the two-step method is defined as

The standard Lagrange polynomial depends on the values of and is defined as

The parametric values τ_k, for k= 0, 1, 2....,m, used in the computation of vectors, r_i and w_i, are found by the metric of the form

The matrix N is a positive definite matrix, and three choices are given for N as I, B_i, and B_i+1 on variable space and z₁, z₂ ϵ Rⁿ.

This metric between different iterates in the current interpolation is measured by fixed-point and accumulative approaches (Ford and Moghrabi, 1994). In this study, an accumulative approach is used for finding the parametric values.

• Accumulative Methods

These methods accumulate the distance between the consecutive iterates in their natural sequence. The latest iterate x_i+1, corresponding to the value τ_m of τ, is considered as the origin or base point, and the other values of τ are calculated by accumulating the distance between the consecutive pairs. Therefore, we have

In the two-step method, the accumulative type is determined as A1, A2, and A3, and the parametric values are found with the help of Equation (21) for k=0,1, where the base point will be τ₂ = 0 for m=2 from Equation (20).

• Algorithm A1

The identity matrix I is taken as matrix N in this algorithm.

• Algorithm A2

In this algorithm, matrix N is taken as the current Hessian approximation B_i.

The above equation involves a matrix vector product which is computationally expensive. For instance, with the help of search direction, we can easily compute the parameters of the same situation as

since

By substituting Equation (26) in Equation (24), we get

The above expression is easy to calculate but the expression in τ₀ is very difficult to compute in every iteration. Therefore, to lessen the computational cost, Ford and Moghrabi (1994) claimed that, in multi-step methods, when the quasi-Newton equation in Equation (2) is not satisfied, then we can consider that it satisfies approximately by replacing i+1 with i in Equation (2), where B_i is an approximation of matrix N. Therefore, we obtain

using Equation (28), we have

• Algorithm A3

In this algorithm, the choice of matrix N is B_i+1, which is the Hessian approximation at x_i+1.

Since τ₁ and τ₀ are expensive to compute, Equation (2) is used.

Aamir and Ford (2021) investigated the multi-step skipping technique in which one-step and two-step skipping strategies were experimented and produced better results than those without skipping strategy. The authors also modified the search direction, which was implemented with/without the skipping technique to achieve good performance in minimum time duration. We experimented two test functions of different dimensions by the two-step quasi-Newton methods with different techniques, i.e., one-step skipping with no modified search direction and one-step skipping with modified search direction on variable space with a high rate of computational effort. Therefore, to lessen the computational burden and increase efficiency, an algorithm was developed to execute a particular function by using the best method only. Section 2 discusses the two-step quasi-Newton method with different techniques in detail. Section 3 proposes a self-decisive algorithm which is developed based on image processing method to find the image values of different test functions. Section 4 discusses the experimental setup of the proposed strategy. Section 5 analyzes the numerical results of one function, which can help the algorithm to execute a particular function by the best method only. In the last section, the conclusion is drawn based on different numerical simulations.

2. Two-step quasi-Newton methods with different techniques

Different techniques in two-step methods, such as the one-step skipping technique with no modified search direction and the one-step skipping technique with modified search direction, are implemented on the selected test functions for the purpose of minimization. These functions are examined by function evaluation, the number of iterations, and time in seconds. The notation of different methods on different techniques is given in Table 1.

2.1. Skipping technique

In quasi-Newton methods, updation of Inverse Hessian approximation H_i to H_i+1 is a very expensive procedure under certain circumstances. Tamara et al. (1998) introduced the idea of skipping updates for certain steps to lessen the burden of computational cost. They investigated the question of “how much and which information can be dropped in BFGS and other quasi-Newton methods without destroying the property of quadratic termination” and called this procedure “backing up.” They used this idea in the algorithm if the step length is 1.0 or the current iteration is odd.

Aamir and Ford (2021) investigated the skipping technique in single step and multi-step methods. The experimental results of comparison between the skipping and non-skipping methods revealed that skipping algorithms outperformed non-skipping algorithms.

• Algorithm of the Multi-step Skipping Method

The general algorithm of the skipping technique is as follows:

• Select an initial approximation x₀ and H₀ and set i=1

• For j=1: m (where m is the number of steps to be skipped)

  • Calculate a search direction p_i+j−2= −H_i−1g_i+j−2.

  • Find t by giving t_i+j−2 for executing line search x_i+j−2+tp_i+j−2.

  • Calculate new approximation x_j=x_j+t_jp_j.

End for

• 3. By the use of different methods, update H_i+j−2 to give H_i+m−1.

• 4. If ||g_i|| ≤ ϵ, then stop, else i:=i+m and go to step 2. End if.

• Application of the skipping technique on the two-step method

Now that we are at x_i+1, the matrix is updated by B_i−1,s_i,s_i−1,y_i, and y_i−1, using the following steps:

1. Using the above terms, compute τ_k and then find δ.

2. By the use of all the above values, through which we find r_i and w_i, we have

3. The Hessian approximation is updated by using all the above values.

Now we compute τ_k and/or δ under different methods.

2.2. The two-step method with skipping and modified search directions

Here, we explained the derivation of the modified search direction. The following notations are used during the derivation of modified search direction in the skipping technique.

Ĥ_i represents that the matrix is never computed.

represents modified search direction.

Now, let us consider that the single-step BFGS updated the matrix H_i−1. The search direction p_i−1 is defined as

By using the skipping technique, the next search direction is

With the help of Ĥ_i, we can find the modified search direction . We define

where

Now, the modified search direction is

and by Equation (35), we get

From the above equation, it can be observed that H_i−1y_i−1 and λ_i−1 cannot be easily computable due to the matrix vector product than other terms. However, with the help of Equation (34) and Equation (35), the expression H_i−1y_i−1 can be defined as

Therefore, using the above equation in Equation (37), modified search direction can be calculated efficiently without explicitly computing .

• Algorithm of the Multi-step Skipping Quasi-Newton method with Modified Search Direction

The general algorithm is given as follows:

• Select x₀ and H₀ as an initial approximation; set i=1

• For j=1:m, where m is the number of steps,

• Calculate p_i+k−2= −H_i−1g_i+k−2.

• Calculate modified search direction .

• Do the line search along x_i+j−2+tp_i+j−2 and also providing a value of t_i+j−2 for t.

• Calculate new approximation .End

• Update H_i−1 to produce H_i+m−1 by using different methods discussed in previous sections.

• Check for convergence, if it is not converged, then i=i+1 and go to step no: 2.

3. Image processing

In the viewpoint of image processing “an image is an array or matrix of numeric values called pixels (Picture Element) arranged in columns and rows”. In mathematics an image is defined as “a graph of a spatial function” or “it is a two-dimensional function f(x,y), where x and y are the spatial (plane) coordinates, and the amplitude at any pair of coordinates (x,y) is called the intensity of the image at that level.” If x,y and the amplitude values of f are finite and discrete quantities, we call the image a digital image. A digital image is composed of a finite number of pixels, each of which has a particular location and value. Image processing is a process in which different mathematical operations are performed subject to application on the image to get improved or to extract significant information from the image for subsequent processing. When this process is applied to digital images is called digital image processing.

Digital image processing has a wide scope for researchers to work on various areas of science (such as, a agriculture, biomedical, and engineering). Previous studies showed that researchers applied and investigated different techniques of image processing for analysis and problem solving, such as detection and measurement of paddy leaf disease symptoms using image processing (Narmadha and Arulvadivu, 2017), breast cancer detection using image processing techniques (Christian et al., 2000), a novel outlier detection method for monitoring data in dam engineering (Shao et al., 2022), and counterfeit electronics detection using image processing and machine learning (Navid et al., 2017).

3.1. Proposed strategy

In the proposed strategy, the algorithm is developed by which the image values of different images I(x, y) of test functions are obtained by statistical technique, and the desired objective is achieved. In the first step, the images of different test functions are obtained with a resolution of 600 × 600 pixels. In the second step, the window of a size W×W is generated around each pixel of the image I(x, y), where the suggested size of the generated window is 3 × 3 and this window is treated as matrix S. The rows of the matrix S are considered as observations, and columns are considered as variables. Figure 1 shows the schematic diagram of the matrix S generation.

Figure 1

In the third step, covariance matrix C^{(x, y)} of matrix S is computed with the help of the following equation:

In the fourth step, eigenvalues of the covariance matrix are calculated. The sum of the eigenvalues is directly proportional to edge strength, which is calculated as follows:

The third and fourth steps are done twice, the first time for the horizontal edge strength and the second time for the vertical edge strength calculation. Therefore, Equations (39) and (40) are used for calculating horizontal and vertical edge strength generation as follows:

The sum of horizontal and vertical edge strength gives the value of a pixel of I(x, y). Hence, the value of each pixel of an image is calculated as

the sum of all pixel values gives the value of an image I(x, y) defined as

4. Experimental setup and results

Two test functions were selected from literature and were executed by using different techniques of the two-step quasi-Newton methods on variable space. The execution of test functions by every technique was computationally expensive. Therefore, an algorithm is required to enable the researchers to execute a particular function by the best method only to reduce computational cost.

Hence, our objective is to develop such an algorithm that can compute the image value of every input image of the test function and forward each function to the method by which it outperformed. The algorithm works in the following steps:

• Obtain the images of each test function.

• Compute the image value of each image.

• Classify test functions by their image values.

• Forward the particular function toward the best method.

4.1. Test functions

To check the performance of different techniques used in two-step methods, we considered two test functions of different dimensions with four different starting points and epsilon value from the literature (Hillstrom et al., 1981), which are reported in Tables 2, 3. These test functions are categorized into three classes, namely soft, medium, and hard.

• Soft: (2 ≤ n ≤ 20);

• Medium: (20 ≤ n ≤ 60);

• Hard: (61 ≤ n ≤ 150);

• Combined: (2 ≤ n ≤ 150).

The equations of both test functions are mentioned below by which 600 × 600 resolution images (displayed in Figure 2) are generated to calculate the image value of each function (as reported in Table 4) and which are programmed successfully in the self-decisive algorithm.

• Extended Rosenbrock function:

• Modified Trigonometric function:

Figure 2

4.2. Self decisive algorithm

An outline of the self decisive algorithm can be defined as follows:

Step 0: Obtain image I(x, y) to generate window/matrix S.

Step 1: Compute covariance matrix .

Step 2: Find eigenvalues of C^{(x, y)}.

Step 3: Calculate the value of each pixel V(x, y)= (strength h)+ (strength v).

Step 4: Compute the value of each image .

Step 5: Set threshold on V(I).

Step 6: Function execution by indicated/best method type.

5. Numerical analysis of test functions

Two test functions, namely, Rosenbrock and modified trigonometric functions, are selected from the literature (Hillstrom et al., 1981) to evaluate their performance by using two different two-step techniques, i.e., one-step skipping with no modified search direction and one-step skipping with modified search direction. These functions are of different dimensions ranging from 2 to 150.

5.1. Discussion on rosenbrock function

• It is evident from Table 5 that the function outperformed by the method ) during function evaluation and number of iterations, and time reduction is noted by ) in soft dimension. In medium dimension, the ) and ) methods outperformed in function evaluation, the number of iterations, respectively, while the ) method exhibited reduction in time. In hard dimension, the ) method showed better results to minimize the function evaluation and the number of iterations and the ) method reduced the computational effort.

• The results of Table 6 show that the ) method outperformed in terms of function evaluation, the number of iterations, and computational time in soft dimension problems. In medium dimension, the ) method exhibited best results in function evaluation, the number of iterations, and computational time. The ) method showed a reduction in function evaluation and the number of iterations, and the ) method is executed in less time.

• Comparative analysis of both techniques

• The behavior of both techniques was compared, and our analysis concluded that one-step skipping with no modified search direction outperformed in function evaluation and computational time, while the second technique, i.e., one-step skipping with modified search direction, showed a reduction in the number of iterations in all dimension problems.

5.2. Discussion on modified trigonometric function

• Table 7 shows the function outperformed by the method in function evaluation, the number of iterations, and in computational time in soft dimension. In medium dimension, the method outperformed in function evaluation and the number of iteration while less computational time is noted by the method. The method showed better results in function evaluation, while the and methods outperformed in the number of iterations and time, respectively, in hard dimension.

• In Table 8, the method showed the best results in function evaluation and the number of iterations in soft and medium dimensions, and the time reduction is observed by the method in soft dimension and by the method in medium dimension. In hard dimension, the method exhibited minimum results in function evaluation and computational time, and the method showed a reduction in the number of iterations.

• Comparative analysis of both techniques

• Both techniques were compared and analyzed based on experimental results. From the analysis, it can be concluded that one-step skipping with no modified search direction outperformed in function evaluation, the number of iterations, and computational time, except the one case of medium dimension, in which the second technique, i.e., one-step skipping with modified search direction, showed a reduction in function evaluation.

6. Conclusion

An algorithm was developed to compute the image value of a particular test function and direct it to its best method for execution. The two-step quasi-Newton methods with two techniques (one-step skipping with no modified search direction and one-step skipping with modified search direction) were chosen and experimented on two test functions, namely, Rosenbrock and modified trigonometric function. The best method was determined using the experimental results obtained in terms of function evaluation, the number of iterations, and computational time. This study concluded that the one-step skipping without modification in search direction technique showed superiority over the one-step skipping with modified search direction technique under both test functions. Hence, this algorithm directed all the functions having the same image value as Rosenbrock and modified trigonometric functions to the one-step skipping technique with no modified search direction.

7. Future work

To further strengthen the algorithm reported in this study, we propose to investigate the image recognition in terms of picture or graph instead of image value and then direct the reported function (or medical image) to the best method available for the obtaining solution. Based on the literary research, in the future, we are planning to collaborate with some biomedical labs to validate the practicality of the proposed algorithm.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1

  AamirN.FordJ. (2021). Two-step skipping techniques for solution of nonlinear unconstrained optimization problems. Int. J. Eng. Works8, 170–174. 10.34259/ijew.21.806170174

  • CrossRef
  • Google Scholar

• 2

  ChristianC. T.SuttonM. A.BezdekJ. C. (2000). “Breast cancer detection using image processing techniques,” in Ninth IEEE International Conference on Fuzzy Systems. FUZZ-IEEE 2000 (Cat. No. 00CH37063), Vol. 2 (Vellore: IEEE), 973–976.

  • Google Scholar

• 3

  FordJ.MoghrabiI. (1993). Alternative parameter choices for multi-step quasi-Newton methods. Optim. Meth. Softw. 2, 357–370. 10.1080/10556789308805550

  • CrossRef
  • Google Scholar

• 4

  FordJ.MoghrabiI. (1994). Multi-step quasi-Newton methods for optimization. J. Comput. Appl. Math. 50, 305–323. 10.1016/0377-0427(94)90309-3

  • CrossRef
  • Google Scholar

• 5

  HillstromK.MoreJ.GarbowB. (1981). Testing unconstrained optimization software. ACM Trans. Math. Softw. 7, 136–140. 10.1145/355934.355943

  • CrossRef
  • Google Scholar

• 6

  JaffarF.AamirN. (2020). Comparative analysis of single/multi-step quasi-newton methods at different delta values. Punjab Univ. J. Math. 52, 65–77.

  • Google Scholar

• 7

  MannelF.RundA. (2021). A hybrid semismooth quasi-newton method for nonsmooth optimal control with pdes. Optim. Eng.22, 2087U2125. 10.1007/s11081-020-09523-w

  • CrossRef
  • Google Scholar

• 8

  MoghrabiI. A.HassanB. A.AskarA. (2022). New self-scaling quasi-newton methods for unconstrained optimization. Int. J. Math. Comput. Sci.17, 1061U1077. 10.11591/ijeecs.v21.i3.pp1830-1836

  • CrossRef
  • Google Scholar

• 9

  NarmadhaR. P.ArulvadivuG. (2017). “Detection and measurement of paddy leaf disease symptoms using image processing,” in 2017 International Conference on Computer Communication and Informatics (ICCCI) (Coimbatore: IEEE), 1–4.

  • Google Scholar

• 10

  NavidA.TehranipoorM.ForteD. (2017). Counterfeit electronics detection using image processing and machine learning. J. Phys. Conf. Ser. 787, 012023. 10.1088/1742-6596/787/1/012023

  • CrossRef
  • Google Scholar

• 11

  RitterF.LaueT. B. A. H. H.SchwierM.LinkF.PeitgenH.-O. (2011). Medical image analysis. IEEE Pulse2, 60–70. 10.1109/MPUL.2011.942929

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  SchröterM.SauerO. A. (2010). Quasi-newton algorithms for medical image registration. IFMBE Proc.25, 433–436. 10.1007/978-3-642-03882-2_115

  • CrossRef
  • Google Scholar

• 13

  ShaoC.ZhengS.GuC.HuY.QinX. (2022). A novel outlier detection method for monitoring data in dam engineering. Expert Syst. Appl. 193, 116476. 10.1016/j.eswa.2021.116476

  • CrossRef
  • Google Scholar

• 14

  TamaraD. P. O.KoldaG.NazarethL. (1998). BFGS with update skipping and varying memory. J. Optim8, 1060–1083. 10.1137/S1052623496306450

  • CrossRef
  • Google Scholar

• 15

  WangJ. (2018). Application of improved quasi-newton method to the massive image denoising. Multimed Tools Appl. 77, 12157–12170. 10.1007/s11042-017-4863-y

  • CrossRef
  • Google Scholar

• 16

  WolfI.VetterM.WegnerI.BottgerT.NoldenM.SchobingerM.et al. (2005). The medical imaging interaction toolkit. Med. Image Anal. 9, 594–604. 10.1016/j.media.2005.04.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar