As vastly known, NPDEs are commonly applied to describe a lot of relevant dynamic processes and phenomena in mechanics, biology, physics, chemistry, etc. [1]. The solutions of NPDEs may provide a significant information for scientists and engineers. The ISM, proposed in Galaktionov [2] and modified in Ma [3], is one of strongest techniques to derive the solutions of NPDEs. The technique involve several invariant subspaces which are defined as subspaces of solutions to linear ODEs have been utilized to solve special NPDEs [3]. In Shen et al. [4], Zhu and Qu [5], and Song et al. [6], the maximal dimensions of invariant subspaces for studying n system of NPDEs has been reported. On the other hand, the ansatz technique is a powerful technique used in deriving the soliton solutions of NPDEs. The approach is based upon substituting an ansatz directly into the equation. The method has been used to obtain the solutions of several NPDEs [7–10]. In the last few decades, several powerful integration approaches have utilized to study many equations [11–19].

In this paper, we aim to study Equation (3) using the ISM [4–6]. Then, we will classify the soliton solutions of the equation by utilizing the the powerful ansatz approach [7, 8].

Fokas and Liu [20] introduced a system of integrable KdV system. The system in it's original form is given by

Gurses and Karasu [12] further simplified Equation (1) by considering a linear transformation of the form

where m₁, m₂, n₁ and n₂ are arbitrary constants, s and r new dynamical variables, qⁱ = (s, r). On properly choosing the constants, the coupled nonlinear Fokas-Liu system Equation (1) is reduced to a simpler form represented by:

with transformation parameters given by Baskonus et al. [15]:

In Equation (3), u is the elevation of the water wave, υ is the surface velocity of water along x-direction [15]. The parameters a, b, c, e, f, d are constants. The only condition on the parameters a, b, c, e, f, d is given by c = fb. This guarantees the integrability of the above system.

Let us give a brief account of the ISM [6]

The operator F1≡F1[u¯1,u¯2] and F2≡F1[u¯1,u¯2] are smooth functions with orders k₁ and k₂, namely

In the above and subsequent sections, we will apply the following notation

Let W be a new linear subspace Wn11×Wn22, where

and f1(x)q,…,fnq(x)q are linearly independent. If the vector operator F = (F¹, F²) satisfies the condition

i.e.,

satisfies

Then the vector operator 𝔽 admit an invariant subspace given by W. If the subspace W is being admitted by the operator 𝔽, then Equation (5) has a solution given by

with λjq(t) being functions of t satisfying the following ODEs

Suppose Wnqq=L{f1q(x),…,fnqq(x)} is generated by the solutions of the linear n_qth-order ODEs

Thus, the invariant conditions represented by

one can denote by [H_q] the equation Lq[u¯q]=0 and its differentials w.r.t x. Once one determined the maximal dimension, then the complete classification and exact solutions of the equation can be investigated. From Equation (15) representing the invariant condition, the estimation has been determined in Shen et al. [4].

Theorem 3.1. Let 𝔽 = (F¹, F²) be a nonlinear vector and be coupled. We can assume without loss of generality (k₁ ≥ k₂). If the operator 𝔽 admits the invariant subspace Wn11×Wn22(n1≥n2>0), then there holds n₁ − n₂ ≤ k₂, n₁ ≤ 2(k₁ + k₂) + 1.

In theorem 2.1, the operator 𝔽 is couple meaning

𝔽 represents a nonlinear vector, i.e., for certain i₀, j₀, l₀ ∈ {1, 2}, p₀, q₀ ∈ {0, 1, …, k_i0}, there holds

In the case of k₁ = k₂, the estimation of maximal dimension is given in Zhu and Qu [5]. Next, we consider the case 0 < n₁ < n₂. We give the following results from Song et al. [6] in a more general form which we shall apply in the next section.

In this section, we will construct the invariant subspace and solutions of Equation (3). Let us take an invariant subspace W2,2=W21×W22 defined by

where a₀, a₁, b₀, and b₁ are constants to be determined. The corresponding invariance conditions are given by

where

Substitute the expressions for F and G into the above equations, we obtain an overdetermined system of algebraic expressions which can be solved in general to obtain the invariant conditions given by

Therefore, Equation (14) reduces to

Thus, we get W21=span{1,x} and W22=span{1,x}. This invariant subspace takes the exact solution of Equation (3) as

where λ_i(t), i = 1, 2, 3 are unknown function to be determined. Putting Equation (23) into Equation (3), we acquire the following system of ODEs:

Solving Equation (24), we acquire

Subsequently, we obtain the following algebraic function solution

where c_i(i = 1, …, 3) are arbitrary constants.

In this section, we will utilize the ansatz approach to derive the topological, non-topological and singular soliton solutions of Equation (3).