RL is an important branch of machine learning (Carapuço et al., 2018), but unlike supervised learning and unsupervised learning, it is an active learning process, which does not require specific training data, and agents need to obtain samples in the process of continuous interaction with the environment. As shown in Figure 2, by taking the goal of maximizing the cumulative reward, RL continuously optimizes the strategy based on the state, action, reward and other information, and finally finds the optimal state-action sequence during the training process. The process is very similar to that of human learning, in which strategies are continually improved through interaction and trial and error with the environment.

The interactive process can be expressed by Markov decision processes (Bi et al., 2019). Suppose the environment is completely observable, the state space of the environment is represented by S , and the action space is represented by A ; the behavior of the agent is defined by policy π, which defines a probability distribution p( A ) to represent the relationship between state and action. At time t, let s t   s t ∈ S be the state of the environment, and a t ( a t ∈ A ) be the action taken by the agent according to the state s t and then at time t+1, the state transitions to s t + 1 . In this process, the instant reward received by the agent can be expressed as r s t , a t . For the entire process t = 1 , 2 , 3 , … , T , the historical state information can be represented by state-action pairs s t = s 1 , a 1 , … , s t − 1 , a t − 1 . The sum of discounted future reward returned R t by the agent after performing the action a t is defined as R t = ∑ i = t T γ i − t r s i , a i (1)

Where γ is the discounting factor, and γ ∈ 0 , 1 . It can be found that R t has a great relationship with the action selected by the policy, and RL is to learn the optimal policy to maximize the expected return from the start distribution J = E r i , s i ∼ E , a i ∼ π R 1 . We represent the discounted state access distribution of policy π as ρ π .

In order to describe the expected return of the model under the state s t , the action a t taken by the agent, and under the guidance of the policy π, an action-value function Q π s t , a t is used to express it (Sutton et al., 2000), which is defined as Q π s t , a t = E r i ≥ t , s i ≥ t ∼ E , a i ≥ t ∼ π R t | s t , a t (2)

The above formula can be converted into a recursive form through the Bellman equation as: Q π s t , a t = E r t , s t + 1 ∼ E r s t , a t + γ E a t + 1 ∼ π Q π s t + 1 , a t + 1 (3)

DDPG is a model-free DRL method based on the critic-actor framework and deterministic policy gradient algorithm (Lillic et al., 2016; Thomas and Brunskill, 2017). In the processing of high-dimensional state space and action space, DDPG uses deep neural networks (Sen Peng et al., 2018) as the approximator of action function and action-value function, which also brings a problem. The training process of the neural network needs to assume that the samples follow an independent distribution, but the samples obtained in chronological order obviously do not meet this requirement. To solve this problem, DDPG draws on the experience replay mechanism in deep Q-network (Mnih et al., 2015) and the minibatch training method in deep neural networks to ensure the stability of the training process of large-scale nonlinear networks.

To avoid the inner expectation of the deterministic policy, the deterministic policy function μ : S ← A is used to describe the action-value function: Q μ s t , a t = E r t , s t + 1 ∼ E r s t , a t + γ Q μ s t + 1 , μ s t + 1 (4)

An adept UTSG model is crucial for the design and testing of control algorithms. Typically, thermal-hydraulic models based on conservation principles of mass, energy, and momentum are employed to precisely simulate the operational characteristics of steam generators. However, such models often exhibit intricate non-linear features, posing challenges in controller design. In practice, a UTSG model that is relatively straightforward yet accurate, faithfully capturing dynamic traits, is preferred. The linear model proposed by Irving (Irving et al., 1980), derived through a fusion of experimental and theoretical approaches, has undergone rigorous validation across multiple power levels, affirming its precision in replicating operational characteristics. Consequently, it has found extensive application in the realm of control algorithm research. This model establishes a transfer function model related to feed water flow Q e , steam flow Q v and narrow-range water level Y : Y s = G 1 s Q e s − Q v s − G 2 1 + τ 2 s Q e s − Q v s + G 3 s s 2 + 2 τ 1 − 1 + τ 1 − 2 + 4 π 2 T − 2 Q e s (12)

Where s is Laplace variable, G 1 , G 2 and G 3 are constant, τ 2 is the delay time of the shrink and swell phenomenon, τ 1 is the delay time of the mechanical oscillation, and T is the period of the mechanical oscillation. The first term X 1 s = G 1 s Q e s − Q v s calculates the change in water level by summing the flow in and out, which represents the wide-ranging effect of UTSG. The second term X 2 s = G 2 1 + τ 2 s Q e s − Q v s is used to describe the inverse kinetic phenomena caused by shrink and swell effects. The third term X 3 s = G 3 s s 2 + 2 τ 1 − 1 + τ 1 − 2 + 4 π 2 T − 2 Q e s represents the effect of water level oscillations generated by the feed water in the annular descending channel. The values of the power-related parameters of this model at 5 typical power levels are given in Table 2.

In order to understand the dynamic characteristics of the model, this section will briefly analyze the response characteristics of the model when the feed water flow and steam flow step +1 kg/s respectively in conjunction with the shrink and swell phenomenon.

When the feedwater flow rate experiences a step increase of +1 kg/s, the corresponding dynamic response of the UTSG water level is illustrated in Figure 4A. It becomes evident that the initial surge in feedwater flow prompts a surge in water level, irrespective of the power levels. Subsequently, as the feedwater temperature falls below the saturation temperature, leading to an augmentation in subcooling within the bundle and consequent steam condensation, the water level descends. Given that the feedwater flow surpasses the steam flow, the water level sustains an upward trajectory, a phenomenon colloquially referred to as the ‘shrink effect’.

When a step increase of +1 kg/s in steam flow is applied, the associated dynamic response of the UTSG water level is depicted in Figure 4B. It becomes evident that, across varying power levels, as steam flow escalates, the pressure within the steam dome diminishes, leading to a reduction in the saturation temperature of the water and an augmentation in boiling within the bundle area. Consequently, the water level initially experiences an ascent. As the steam flow surpasses that of the feedwater, a sustained decline in the water level ensues, a phenomenon commonly referred to as the ‘swell effect’.

Simultaneously, it is noteworthy that, for distinct power levels, identical disturbances in feedwater flow or steam flow yield varying degrees of both the shrink and swell effects. Additionally, it is observed that the transition time during low-power operations exceeds that observed during high-power conditions. This phenomenon underscores the inherent challenges in regulating water levels within the low-power range.

Compared with single control loop, the cascaded control is more controllable and safer, and has better robustness (Jia et al., 2020). Therefore, CPI controller is adopted as the basic controller in this paper. The working process of CPI controller is shown in Figure 8. In the outer loop control, the difference between the expected water level and the model output water level is used as the input of the controller. Its function is mainly used to control the water level to track the change of the expected value. In the inner loop control, the sum of the output of the outer loop controller and the steam flow rate minus the feed water flow is used as the input of the controller, which is mainly used to suppress the steam flow disturbance.

The working principle of PI controller is expressed as: u t = K p e t + K i ∫ 0 t e t d t (13)

Where e t is the error, K p is the proportional coefficient, and K i is the integral coefficient; in this paper, the proportional and integral coefficients of the outer loop controller are defined as K p 1 , K i 1 , and the corresponding parameters of the inner loop controller are defined as K p 2 , K i 2 .

The IHA controller proposed in this paper uses a double level controller structure, shown in Figure 5. The CPI controller is used as primary controller, which is responsible for directly controlling the water level of the UTSG model; the advanced controller uses an agent-based controller with intelligent characteristics, which is responsible for online adjustment of the parameters of the CPI controller. In control process, the primary controller and the advanced controller work together to adjust the control policy in real time according to the state of the system and realize intelligent autonomous control.

The accurate observer information should be provided to represent the dynamic characteristics of the controlled object. In the controller system, the error and the reciprocal of error are often used to indicate the state of the system, which is more suitable for single-target control. However, the UTSG needs to change between different water levels, and various system states need to be considered using the above state expression. To improve this phenomenon, the relative error and reciprocal of relative error are used to represent the state of system. In this way, different target control can be achieved by designing only one state representation, which greatly simplifies the complexity. At the same time, this paper draws on the ideas of (Mnih et al., 2015). In continuous control tasks, the continuous-time environmental state is related, and the observed variable for a period is embraced as the environmental state representation, which can more accurately reflect the state of the system.

In this paper, the values of relative error r e t and reciprocal ∂ r e t ∂ t in consecutive 3s are used as observer information to obtain the observer vector s 1 t of the error term and the observation vector s 2 t of the reciprocal term, which are defined as follows: s 1 t = w 1 0 , 0 , r e t T , t = 0 w 1 0 , r e t − 1 , r e t T , t = 1 w 1 r e t − 2 , r e t − 1 , r e t T , t ≥ 2 (14a) s 2 t = w 2 0 , 0 , ∂ r e t ∂ t T , t = 0 w 2 0 , ∂ r e t − 1 ∂ t , ∂ r e t ∂ t T , t = 1 w 2 ∂ r e t − 2 ∂ t , ∂ r e t − 1 ∂ t , ∂ r e t ∂ t T , t ≥ 2 (14b) r e t = t a − y t t a (14c) where t a is the target value, w 1 and w 2 are normalization coefficients, which is used to transform the value to the interval [0,1], to promote the training efficiency of the neural network. At time t = 0 , e t = ∂ r e t ∂ t = 0 . Finally, the observation vectors s 1 t and s 2 t are combined to obtain a comprehensive observation matrix s t with a dimension of 3 × 2: s t = s 1 t , s 2 t (15)

Therefore, the dimension of observer information is determined to be 3 × 2, and the dimension of action information is determined to be 4 × 1. The network structure of action network and critic network is further confirmed, as shown in Table 3.

The reward function can also be called the evaluation function. A good reward function not only speeds up the learning process, but also makes it easier to find the global optimal solution. The commonly used evaluation functions are ITAE, ITSE and integral of squared time weighted errors. However, these functions are suitable for evaluating the entire control process. In the RL process, the control effect of each step needs to be evaluated, and it has a strong guiding effect on the learning process. Therefore, a new evaluation function is needed to evaluate the learning process.

In fact, in the control process, when the water level error is large, the PI controller needs a large gain to obtain a large response speed, and when the error is small, the value of the gain needs to be reduced to avoid overshoot. Therefore, this paper constructs a segmented evaluation function r t , which can guide the learning process by adjusting some parameters. Experiments show that this function can quickly and effectively guide the agent’s learning, which will be introduced below.

According to the difference in absolute value of relative error r e t , we specifies that r e t > 200 % is the abnormal area, 200 % ≥ r e t > 15 % is the large error area, and r e t ≤ 15 % is the low error area.

Within the abnormal region, where the system strays significantly from the target value, a proactive approach is adopted. The ongoing task is promptly terminated, and a fresh training process is initiated to conserve valuable training time. Simultaneously, a correspondingly modest reward value is prescribed to guide the agent away from this undesirable state. In contrast, the expansive error zone warrants a heightened emphasis on speed of response, with the overarching goal being swifter rectification without excessive deliberation. Consequently, when the relative error resides within this territory, the reward value is uniformly designated as −2. Within the realm of low error, where the system’s output closely approximates the desired value but is susceptible to overshooting and necessitates prolonged adjustment, the formulation of the reward function assumes paramount significance.

Considering the relative error r e t and the reciprocal ∂ r e t ∂ t can accurately represent the state of the system; Therefore, we set the reward function as a function related to them. At the same time, the power function is introduced to further optimize the reward function. As shown in Figure 6, we plotted the power function y = x α in the interval 0 , 0.15 , from which we can see that when α > 1 , y is not sensitive to the change of x . When α < 1 , y is more sensitive to the change of x , and the smaller α is, the more sensitive it is, so it can play a magnifying effect on the local small features. Below we introduce in detail the low error area reward function.

The evaluation term r 1 t of relative error, defined in formula 16, is used to evaluate the degree of deviation of the system state from the expected value. When the steady-state error is 0, r 1 t takes the maximum value of 0. r 1 t = − w 1 r e t α 1 (16) where, α 1 is the exponential adjustment factor, which can adjust the local feature. When α 1 > 1 , it means to reduce the local micro feature; when α 1 < 1 , it means to enlarge the local micro feature.

The evaluation item r 2 t of the reciprocal of the relative error, defined in formula 17, is used to evaluate the degree of fluctuation of the system state. When the system state is stable, r 2 t takes the maximum value of 0. r 2 t = − w 2 ∂ r e t ∂ t α 2 (17) where α 2 is the exponential adjustment factor, its effect is consistent with that of α 1 .

In order to prevent the influence of parameter mutation in the control process, especially in the case of sudden step of reference value, the reciprocal of error is very large. Therefore, we built a c l i p function, as shown in formula 18, which can limit the value to a certain range. c l i p a , b , x = a , x > a x , x ≤ a b , x < b (18)

The c l i p function is used to process r 2 t , and the following results are obtained: r 2 ′ t = c l i p 1 , − 1 , r 2 t (19)

By adding r 1 t and r 2 ′ t , the reward function of the low error area is obtained: r 3 t = r 1 t + r 2 ′ t (20)

In summary, the final reward function is obtained: r t = − 100 , r e t > 200 % − 2 , 15 % < r e t ≤ 200 % r 3 t , r e t < 15 % (21)

In order to reduce the complexity, this paper defines α 1 = α 2 and to determine the values of α 1 and α 2 , a water level tracking simulation experiment was carried out. The power level of the model used is 5%. At 10s, the water level reference value is adjusted from 0 mm to 100 mm, the simulation time is set to 600s, and the number of trainings is set to 1,200. At the end of the training, we tested the best performance of the agent obtained under different parameters as shown in Table 4, from which we can see that when α 1 , α 2 = 0.8 , the shortest setting time can be obtained, indicating that good control effect can be obtained at this time.

In this paper, the water level adjustment performance is trained to obtain the best control performance. The detailed training content is consistent with Section 3.5. The main parameter Settings of the program are given in Table 5, which are determined by suggestions given in paper (Mnih et al., 2015) and several experimental tests.

Considering that similar results can be achieved across different power levels, we present training results for only the 5% power level and 50% power level. These training results are depicted in Figure 7. From the figures, it becomes evident that in the initial stages of the training process, when the agent has not yet collected sufficient experience and undergone an insufficient number of training steps, the episode reward remains low, indicating an exploratory phase. As the number of episodes increases, the agent progressively discerns patterns, and the control performance improves. During this phase, the episode reward exhibits an upward trend. After a substantial number of training episodes, the agent starts to converge, with convergence values around −220 for the 5% power level and around −315 for the 50% power level. During this period, there is no distinct trend in episode rewards, signifying that the optimal control policy has been achieved.

Subsequently, an assessment of the trained controller’s performance is scheduled, encompassing three distinctive tests: a water level tracking test, an anti-interference test, and a comparative analysis against findings within publicly available literature. Concurrently, two meticulously optimized controllers, distinguished by their commendable performance, serve as benchmarking mechanisms for each power level. The first of these controllers, christened ‘FCPI,’ benefits from parameter optimization via a fuzzy logic algorithm, incorporating modules such as fuzzification, fuzzy rules, fuzzy inference, and defuzzification. The FCPI controller parameters can adapt with both power levels and water level errors. Due to space constraints, readers are encouraged to refer to the paper (Liu et al., 2010; Aulia et al., 2021) for details on the configuration strategy. The second controller, known as ‘ACPI,’ attains its optimized parameters through the gain scheduling algorithm and the relationship between the parameters of the CPI controller and power p is expressed by formula 22. k p 1 = − 0.5991 * p 3 + 0.6281 * p 2 + 0.7725 * p − 0.0018 k p 2 = 100 − 90 1 + p 0.3117 − 32.68 k i 1 = 8.97 * 10 − 6 * ⁡ log ⁡ 2 p + 4.876 * 10 − 5 k i 2 = 1 (22)

In order to gauge the efficacy of control, we use the evaluation indices ITSE and ITAE. These indices have been thoughtfully introduced, as they offer a practical framework for assessing the performance of the control system. They effectively encapsulate the system’s precision and responsiveness, with smaller values indicating superior performance. I T S E = ∫ 0 ∞ t e 2 t d t (23) I T A E = ∫ 0 ∞ t e t d t (24) where e t is the error.

This section mainly tests the system’s output response under the action of step function, so as to show the dynamic performance of the system. The initial value of the reference water level is set to 0mm, and then jumps to 100 mm at 10s. The control effects of the three methods are compared at low power level (5%), medium power level (50%) and high power level (100%), respectively.

Figures 8, 9, 10 show the comparison results of the three methods at different power levels. It can be seen from these figures that the three methods can track the change of water level and have good control effect. At the low power level, the proposed method achieves a rise time of 73.9 s, which is 23.5% faster than the FCPI method and 59.4% faster than the ACPI method. At the medium power level, the proposed method achieves a rise time of 13.6 s, which is 29.6% faster than the FCPI method and 56.5% faster than the ACPI method. At the high power level, the proposed method achieves a rise time of 16.4 s, which is 10.4% faster than the FCPI method and 28.6% faster than the ACPI method. The above statements emphasize that the proposed method offers a faster response speed and superior control performance in terms of water level tracking. The proposed method, IHA, can iteratively engage with the steam generator model to acquire the water level control strategy. It employs deep neural networks to comprehend the intricate nonlinear relationship between system states and optimized actions. This adaptation allows the controller parameters to accommodate the dynamic variations of the system without the necessity of manual design for optimization strategies, as required in methods like FCPI and ACPI. Given the prolonged delay in false water level generation and the extended response time in low-power scenarios, more time is required to achieve control.

Table 6 shows the comparison results of ITSE and ITAE of different methods, from which under different power levels, the values of ITSE and ITAE are IHA < FCPI < ACPI. At the low power level, the proposed method exhibits an ITSE that is 10.7% lower than FCPI and 38.4% lower than ACPI. Additionally, the ITAE of the proposed method is 26.2% lower than FCPI and 83.3% lower than ACPI. At the medium power level, the proposed method achieves an ITSE that is 1.3% lower than FCPI and 7.1% lower than ACPI. Furthermore, the ITAE of the proposed method is 6.2% lower than FCPI and 23.7% lower than ACPI. At the high power level, the proposed method demonstrates an ITSE that is 3.2% lower than FCPI and 10.3% lower than ACPI. Likewise, the ITAE of the proposed method is 6.8% lower than FCPI and 20.7% lower than ACPI. The statements above highlight that the IHA method excels in terms of control accuracy and speed, particularly evident at low power levels. In summary, the IHA method exhibits the best control performance, followed by FCPI and ACPI, which aligns with the conclusions drawn from the figures.

Figures 11, 12, 13 depict the variation curves of the IHA controller parameters K p 1 , K i 1 , K p 2 , and K i 2 at different power levels. It is evident that the controller parameters adaptively change with the system’s state during the control process. In theory, the integral coefficient K i of the PI controller primarily works to reduce steady-state error, while the proportional coefficient Kp reflects the system’s response speed, rapidly reducing error. Consequently, K i has a noticeable effect towards the end of the control process, whereas K p ′ s impact is more pronounced in the early stages. In the parameter curve results, when the water level error is significant, the proportional coefficient K p 1 plays a major role. At such times, K p 1 assumes larger values across all three power levels, such as the time range for the 5% power level from 10s to 200s, the 50% power level from 10s to 37s, and the 100% power level from 10s to 25s. However, the integral coefficient shows no distinct pattern because it has little influence when the error is substantial. As the water level error decreases, the value of K p 1 decreases as well, reducing the likelihood of overshoot. For instance, the time range for the 5% power level shifts to 200s–350s, the 50% power level to 37s–50s, and the 100% power level to 25s–40s. When the error approaches zero, K p 1 stabilizes and does not have a fixed value, while K i 1 and K i 2 generally assume larger values. This is because when the error is zero, K p 1 has minimal effect, and increasing the integral coefficient is beneficial for reducing steady-state error. Simultaneously, K p 2 does not exhibit a clear pattern throughout the control cycle. Furthermore, manual parameter adjustments reveal that K p 2 has little influence on the results within a certain range. Hence, under varying power levels, the proposed method can autonomously generate optimized strategies for the CPI controller parameters ( K p 1 , K p 2 , K i 1 , K i 2 ) based on the system’s state. This assurance ensures that the system can efficiently regulate the water level to the specified position in the shortest possible time, optimizing overall performance.

Under the work of the ACPI method, the control law can adapt to changes in power levels but struggles to adapt effectively to variations in water level states. Consequently, the ACPI method falls short of achieving an optimal control effect. The FCPI method, while capable of adjusting the control law adaptively with both power level and water level state, relies heavily on the design of fuzzy membership functions and fuzzy rules, which are inherently influenced by human experience. This design challenge makes it difficult to encompass all possible system states, making it also challenging for the FCPI method to achieve an optimal control effect. However, the proposed method excels in achieving an ideal control effect across all power levels, primarily due to its efficient reinforcement learning mechanism. Throughout the control process, both gain and control law can adaptively evolve in response to changes in power levels and state information. During the learning process, the controller agent accumulates control experience continuously through repeated interactions with the environment. It autonomously learns from this experience and explores new strategies within the control policy space. Over time, the controller agent matures and evolves into a master of control, thus achieving exceptional control performance.

To assess the anti-interference capability of the proposed controller, we conducted a steam flow disturbance benchmark test on models at different power levels. During the test, a step disturbance in steam flow of +35.88 kg/s was introduced at 10 s (Liu et al., 2010; Ansarifar et al., 2012). The test results are depicted in Figures 14, 15, 16. From these figures, it is evident that all three methods exhibit strong anti-interference capabilities and swiftly restore the water level to its normal state. Moreover, the oscillation amplitude and adjustment time of the water level decrease as the power level increases, signifying more efficient water level control at higher power levels compared to lower ones. Notably, at the 5% power level, the proposed method restores the water level to its normal state in approximately 200 s in Figure 14A, outperforming the FCPI and ACPI methods. This rapid recovery demonstrates the exceptional anti-interference performance of the IHA controller across various power levels. In Figure 14B- Figure 16B, we observe the changes in feed water flow and steam flow under the control of the IHA method. It is apparent that the feed water flow quickly tracks the variations in steam flow. However, due to the non-minimum phase characteristics of the system at low power levels, the system’s recovery time is longer in this scenario.

Table 7 provides a comparison of ITSE and ITAE results in the anti-interference tests of different methods. At the 5% power level, the ITSE of the IHA method is 77.8% lower than that of FCPI and 34.2% lower than that of ACPI. Similarly, the ITAE of the IHA method is 84.4% lower than that of FCPI and 70.2% lower than that of ACPI. These results clearly demonstrate that the IHA method outperforms the other two methods significantly in terms of both ITSE and ITAE at the 5% power level. Conversely, the comparison results among the three methods show similarity at the 50% and 100% power levels, which aligns with the observations in Figure 14A- Figure 16A.

In summary, the proposed method exhibits a strong anti-interference effect, particularly evident at certain power levels. However, it does not consistently demonstrate clear advantages across all power levels. This limitation stems from the focus of this paper, which primarily investigated water level tracking tasks during the training process of deep reinforcement learning. The development of a comprehensive anti-interference strategy is a potential area for future optimization and research.

It is well known that the water level of UTSG is the most difficult to control at low power level (Choi et al., 1989). In order to highlight the advantages of the proposed method at low power level, we compare the water level tracking effect of the IHA controller at 5% power level with the research results in the public literature, and the test content is to adjust the water level from 0mm to 100 mm.

The setting time serves as the evaluation index, defined as the minimum time required for the water level to reach and stabilize within ±5% of the set value. The comparison results are shown in Table 8, from which we can see that the proposed method can shorten the adjustment time to 375s, with a considerable advantage over other methods, fully embodies the advantages of reinforcement learning.

To underscore the merits of the proposed method under conditions of low power level, we juxtapose the water level tracking efficacy of the IHA controller at a power level of 5% with the discoveries derived from other public research. The experimental setting entails the modulation of water levels ranging from 0mm to 100 mm. It is worth noting that the referenced investigations introduced methodologies such as SVR, FOPID, IMC, and DSMC, all of which featured test content and equipment models consistent with our current study. Consequently, this paper directly assimilates their resultant data for the purpose of comparative scrutiny.