Soft tissues are complex structures that exhibit non-linear, time-dependent elastic properties. Variations in mechanical properties can provide information pertaining to tissue health, including detecting and diagnosing lesions of the breast [6, 34], liver [26, 47, 56], and thyroid [9, 10, 54], monitoring thermal lesions during ablation therapy [41, 53], and characterizing atherosclerotic plaques [2, 3, 11]. Conventional medical imaging modalities are insensitive to this mechanical contrast. Elasticity imaging methods have been developed to fill this niche and directly evaluate the mechanical properties of soft tissue.

Elasticity imaging can be largely grouped into quasi-static or dynamic methods based on the type of mechanical stimulus applied to induce tissue motion. Comprehensive reviews of the various techniques can be found in several articles [33, 35, 38, 39, 46, 52]. We focus our discussion on quasi-static ultrasound elastography (QUSE), wherein local tissue displacements are estimated from RF echo frames acquired as an ultrasound probe is slowly pressed into the tissue surface. Quasi-static loading allows time for the force stimulus to propagate through the entire tissue section. The resulting force-displacement measurements at points provide an enormous amount of information pertaining to tissue material properties throughout the contiguous volume and at boundaries.

Estimating material properties from force-displacement measurements constitutes the inverse problem in QUSE. The majority of quantitative QUSE approaches are built upon model-based optimization methods. While effective, model-based methods are prone to modeling errors that can lead to artifacts in reconstructed images of pre-selected material property distributions. More importantly, mathematically defined constitutive models are inherently limited in their ability to capture material behavior. This limitation could preclude discovery of clinically relevant soft tissue material properties.

Data-driven inverse methods circumvent modeling errors by removing the constitutive model assumption and building a soft-computational model of material behavior from measurement data. The Autoprogressive Method (AutoP) is one such method. Unlike other data-driven approaches, AutoP combines physical modeling through finite element analysis (FEA) with artificial neural networks (ANNs) to extract the stress-strain relationship embedded within force-displacement measurements. In the context of elasticity imaging, ANNs characterizing the stress-strain behavior of soft tissues can be interrogated to infer material parameters that best summarize the learned mechanical properties.

Many of the research articles that employed AutoP have described the training process, but none have directly compared and contrasted its operation with the more typical model-based approaches. In this review, we aim to more thoroughly examine the operating principles behind AutoP to better highlight the fundamental differences when compared to model-based inverse methods. Of particular importance is the manner in which physical principles are directly exploited in AutoP to generate increasingly accurate estimates of stress and strain distributions from force-displacement measurements under arbitrary geometry and loading. We believe the working principles of AutoP can be applied as a data-driven, physics-guided alternative to other difficult boundary value inverse problems.

Section 2 reviews model-based inverse methods in QUSE to establish a basis for comparison. We focus on iterative methods utilizing FEA as the forward solver to keep the discussion concise and comparable to AutoP. Section 3 reviews AutoP and builds intuition of its operation by drawing parallels to typical optimization methods. We also cover a novel neural network architecture developed for QUSE and a regularization term uniquely suited for AutoP. Challenges with non-linear viscoelasticity imaging and potential approaches with alternative ANN structures are briefly discussed in Section 4. Data acquisition and displacement estimation details are omitted from our discussions, but are available in [24, 25]. Even though such details are important in practice and will affect the efficacy of QUSE methods, we wish to compare the operation of AutoP and model-based methods at a more conceptual level.

Reconstructing a map of material properties from force-displacement measurements is an ill-posed inverse problem. ¹ By combining a prior assumption of the underlying mechanical behavior with a system model incorporating physical principles, model-based inverse methods reduce the solution space and estimate parameter values of a pre-selected constitutive model. The goal of QUSE methods is to find the set of spatially distributed material parameters θ ( x ) that minimize the difference between displacements U ( x ) computed in a forward problem and measured displacements U ˜ ( x ) , θ ^ = argmin θ ∈ ℝ ∫ Ω C ( U , U ˜ ) d Ω , (1) where Ω denotes domain of the scan plane(s). The dependence of variables on position x is assumed and not explicitly stated to simplify notation. The cost function C in Eq. 1 is often defined as the L 2 -norm of the difference between computed and measured displacements, augmented by a regularization term ℛ ( θ ) acting on the set of parameters, C ( U , U ˜ ) = 1 2 ( U − U ˜ ) 2 + α ℛ ( θ ) , (2) where α is a hyperparameter that controls how strongly the regularization term affects the solution.

Contrary to model-based inverse methods, the goal of AutoP is to describe the mechanical behavior of a material non-parametrically. By replacing the classic mathematically defined constitutive model with an artificial neural network, any arbitrary stress-strain relationship can be captured. This formulation replaces parameters in Eq. 1 with the connection weights of one or more ANNs ³ . The resulting, trained ANN describing the stress-strain behavior is referred to as a neural network constitutive model (NNCM). One example of a NNCM architecture is shown in Figure 2A (left). The NNCM displayed is a feed-forward, fully connected ANN that accepts a strain vector as input and returns a stress vector at the output. Alternative network architectures have been proposed by other investigators, but this form is the basis for most AutoP applications thus far. Even considering the various NNCM architectures utilized within AutoP, to the best of our knowledge all have comprised at most two hidden layers with up to a few dozen nodes each. Compared to deep-learning methods, NNCMs are very small ANNs.

Existing efforts to build model-free descriptions of material behavior typically rely on experimental test data acquired from a sample in a well-defined loading scenario, such as strip extensiometry, to obtain stress-strain data from force-displacement measurements. Recent investigations in data-driven mechanics use this type of measurement data directly (i.e., no NNCM) to solve computational mechanics problems [31, 32, 37, 50]. Alternatively, stress-strain measurement data can be used to train NNCMs for later use in computational problems (see reviews [14, 15, 44, 55]). More recently, researchers have developed physics-informed deep neural networks (PINNs) that embed physical laws into the cost function used in backpropagation-based training [40].

Some measure of success has been achieved with each of these methods; however, the resulting soft-computational constitutive model, in whichever applicable form, is fully defined by the measurement data acquired from a homogeneous sample. That is, the models are trained once and deployed. When used for inference against input data not within the domain of the training set, the solution becomes unreliable, analogous to modeling errors in model-based inverse methods. Furthermore, these methods rely on the existence of stress-strain training data. Unfortunately, the complexity of soft tissues precludes excision of geometrically precise, homogeneous samples for measurement in well-defined loading scenarios. Nor do ex vivo load tests provide a good measure of soft tissue properties in vivo. In short, current data-driven methods fail to extract relevant information from the available force-displacement measurements acquired in a normal clinical setting.

The Autoprogressive Method [17] circumvents limitations with model-based and other data-driven methods through a tight coupling of NNCMs with FEA and knowledge of both internal and external object geometry. It has been successfully used to build soft-computational mechanical models of many different materials in a multitude of loading scenarios, including soils [19, 20], concrete [27, 28], steel [30, 57], red blood cells [48, 49], and the cornea [16]. AutoP has even been applied to thermal constitutive models [1]. More recently, we have been developing an adaptation of AutoP for QUSE using both boundary and interior displacement measurements [22]. The caveat that internal geometry must be known precludes most clinical imaging applications and is addressed by introducing a new type of NNCM, discussed in Section 3.1.

AutoP is a method for generating information from measurements. When applied to inverse problems in continuum mechanics, the goal is to recover the stress and strain fields induced by measured forces and displacements in a model-free way. Replacing the mathematical constitutive model with an ANN is paramount to provide the flexibility inherent to said networks, but alone is not a sufficient strategy. The two key components that make AutoP work are 1) the application of force and displacement BCs in separate FEAs—FEA _σ and FEA _ε —to obtain physically consistent estimates of stresses and strains through known physical principles and 2) NNCMs participate in the generation of their own training data by acting as the material model in the assembly of the system of equations in Eq. 6. The former point is analogous to model-based inverse methods wherein the parameter estimates in the current iteration affect the displacements computed in the forward problem. For clarity, the FEA equations can be rewritten as σ N N = NN ( ε ) , (16) D N N = ∂ σ N N ∂ ε , (17) K ( θ ) = ∑ N e ∫ Ω e B e T D e N N ( θ ) B e d Ω e , (18) P = ∑ N e ∫ Ω e B e T σ e N N d Ω e . (19)

We introduce the operator NN ( ⋅ ) to represent inference by the NNCM. The analytical expression for D N N is derived in [21].

Imposing measured forces and displacements in separate FEAs exploits stress equilibrium and compatibility requirements. In FEA _σ , measured forces are applied as force BCs ( σ ⋅ n = p ˜ ) ⁴ . Stress equilibrium, defined in Eq. 3, relates forces to stresses and we therefore claim the stresses σ σ computed in FEA _σ are physically consistent approximations of the true stress field. In a separate FEA _ε , corresponding displacement measurements are applied as displacement BCs ( u = u ˜ ) . Because compatibility requirements relate displacements to strain, we claim the strains ε ε computed in FEA _ε are physically consistent approximations of the true strain field. The AutoP training process is summarized in Figure 1B and can be broken into the five steps listed in Algorithm 2.

Linear-elastic pretraining is used only to provide NNCMs with a physically consistent starting state. Unlike model-based reconstruction methods that can be sensitive to parameter initialization [4], investigations with AutoP have demonstrated that linear-elastic pretraining does not constrain the mechanical properties learned by the NNCMs.

Convergence in AutoP is determined by comparing the values of a cost function to predefined limits. In most AutoP applications thus far, the cost function has assumed the form [17]. Φ ( θ ) = ‖ U − U ˜ ‖ 1 . (20)

Aside from the use of the L 1 -norm to diminish the effects of outliers, it is important to note that the magnitude of the cost function has no direct effect on the backpropagation-based training of the NNCMs. Even so, training the NNCMs with the stresses and strains computed in FEA _σ and FEA _ε , respectively, adjust the connection weights so that the material properties described by the NNCMs more accurately resemble the true material properties. We also note that the because the cost function does not directly influence NNCM training, adjustments to Φ ( θ ) do not significantly affect AutoP whereas altering the cost function in model-based inverse methods can significantly change the resulting parameter estimates.

It is not immediately clear why the combination of the stresses and strains from FEAs solved under separate, but conjugate, boundary conditions should lead the ANNs to the correct material properties. We will demonstrate that it is in fact an implicit displacement error that guides the learning process. Define u σ to be the full set of displacements computed in the solution of FEA _σ , including any imposed BCs. Likewise, u ε represents the displacements computed in the solution of FEA _ε . By definition of a NNCM, σ N N = N N ( ε ) = N N [ f u ( u ) ] , where f u ( ⋅ ) is a function that receives a displacement vector as input and returns the corresponding strain vector. Let Δ u = u σ − u ε . Thus, u σ = u ε + Δ u and therefore σ σ = N N [ f u ( u ε + Δ u ) ] . What does this mean? Inference by the NNCM using physically consistent strains ε ε as input results in physically consistent stresses σ σ only when the input is augmented by the displacement error.

To better understand this interaction, assume the NNCM describes a linear-elastic material so that σ N N ≈ D N N ⋅ ε and deformation is infinitesimal, leading to f u ( u + Δ u ) = f u ( u ) + f u ( Δ u ) . We also denote the strains computed in FEA _σ as ε σ while σ ε = N N ( ε ε ) are the stresses output by the NNCM when strains computed in FEA _ε are supplied as input to the network. Training a NNCM with a backpropagation-based method consists of minimizing a cost function, in this case R ^ θ m = argmin R θ m ∈   ℝ 1 2 ‖ σ σ ¯ − σ ε ( ε ε ; R θ m ) ‖ 2 (21) = argmin R θ m ∈ ℝ 1 2 ‖ N N ¯ ( ε σ ) − N N ( ε ε ; R θ m ) ‖ 2 , (22) where R θ m denotes the NNCM connection weights and the overline is used to indicate a constant term. N N ¯ is the same NNCM used to solve FEA _σ and FEA _ε whereas N N ( ε ε ; θ ) changes each training iteration as the NNCM weights are updated. Applying the aforementioned linear approximations, Eq. 22 can be expressed as R ^ θ m = argmin R θ m ∈ ℝ 1 2 ‖ N N ¯ [ f u ( u ε + Δ u ) ] − N N [ f u ( u ε ; R θ m ) ] ‖ 2 (23) ≈ argmin R θ m ∈ ℝ 1 2 ‖ D N N ¯ ⋅ [ f u ( u ε + Δ u ) ] − D N N ( R θ m ) ⋅ [ f u ( u ε ) ] ‖ 2 (24)

If we consider the first iteration of training when D N N ¯ = D N N ( R θ m ) , the cost function further reduces to ≈ argmin R θ m ∈ ℝ 1 2 ‖ D N N ⋅ f u ( Δ u ) ‖ 2 , (25) and reverts back to Eq. 24 after the first training iteration and the NNCM weights R θ m change.

The preceding argument does not imply NNCMs trained in AutoP are limited to learning linear-elastic material properties. Rather, the case of linear materials was chosen as an example to best highlight the effect of displacement error in NNCM training. We emphasize that, similar to model-based methods, the displacement error is a driving force in the solution of the inverse problem as approached through AutoP. However, in the case of AutoP, the effect of displacement error emerges from the inconsistency between stresses and strains computed in FEA _σ and FEA _ε . It is a characteristic that arises out of the equilibrium and compatibility requirements rather than an explicitly defined error measure to direct the parameter search.

Much like optimization-based methods, the parameter estimates in the current iteration—the weights of the ANNs—affect the solution of the forward problem. Moreover, the error when comparing the forward problem solution to experimental measurements affects the parameter updates. What sets AutoP apart from typical model-based and machine learning techniques is its self-learning paradigm guided by physical principles. When applied to QUSE, the result is a patient-specific soft-computational material model. Figure 2B aims to illustrate the self-learning property of AutoP. Imposing measurement data as BCs in separate FEAs forces the material properties learned by the ANNs to satisfy physical principles and be self-consistent.

The preceding discussion of NNCMS and AutoP was limited to objects under loads inducing small deformation. Mathematically, the strain tensor in the limit of infinitesimal deformation is expressed as ε = 1 2 ( ∇ u + ∇ u T ) . Under this linear kinematic assumption, appropriate when induced strains are less than 5%, elastic solids can be adequately described by a linear-elastic constitutive model.

Probing the non-linear mechanical properties of deep structures in QUSE requires application of large loads to sufficiently deform distal tissues. In these scenarios, a large deformation framework must be invoked wherein the strain tensor includes geometrically non-linear terms. As an example, one potential definition of geometrically non-linear strain is ε = 1 2 ( ∇ u + ∇ u T + ∇ u T ∇ u ) . Application of large compressive loads coupled with the non-linear and time-dependent mechanical behavior of soft tissues implies that one or two linear-elastic parameters are ineffective for clinical QUSE. More importantly, the complex non-linear and viscoelastic properties of soft tissues may provide a significant amount of diagnostic information [42], reinforcing the value of reconstructing elasticity images beyond shear modulus.

The problem then arises of selecting an appropriate non-linear and/or viscoelastic constitutive model that describes tissue behavior and maximizes the diagnostic information in QUSE images. Complexity is further increased when considering the likelihood that different tissues or tissue states (i.e., healthy tissue, benign lesions, and malignant tumors) are best described by different material models. Current model-based approaches assume homogeneity of the underlying constitutive model for the entire imaged tissue section and estimate the spatial variation of the corresponding model parameters. There exists no model-based QUSE inverse method capable of discovering the set of constitutive models—and estimating values of the corresponding parameters—that adequately describes a heterogeneous group of tissues. The combination of AutoP with CaNNCMs can fill this void.

AutoP has been applied to build soft-computational models of complex materials undergoing small and large deformation when the total geometry was known (e.g. [19, 27, 28, 30, 45, 58]). To achieve non-linear, time-dependent material property imaging with AutoP, CaNNCMs must be modified to 1) accommodate geometrically non-linear deformation to facilitate non-linear elasticity imaging and 2) incorporate additional stress-strain state information as network input to properly capture a heterogeneous distribution of complex mechanical behaviors. Prior successes with AutoP lead us to believe there exists an adaptation to the CaNNCM architecture and/or AutoP that will accomplish both tasks.

The difficulty of this problem can be understood by comparing the information required for a NNCM to compute a stress vector for materials of varying complexity. For linear-elastic materials, the NNCM output is defined as σ N N = N N ( ε ) . Non-linear, viscoelastic materials require time information to fully define the stress-strain response, σ N N = N N ( ε , t ) . It is further complicated for objects under large deformation because of the need to account for differences in Lagrangian and Eulerian descriptions of stress and strain. It is clear the NNCM inputs must be changed. Our current work is toward the goal of identifying a NNCM architecture capable of learning the stress-strain response of hyperelastic materials.

Projecting further, adjustments to the CaNNCM architecture will also be necessary to characterize a heterogeneous distribution of non-linear, viscoelastic materials. If the operating principle of linear-elastic CaNNCMs is retained, the stress-strain response would take the form σ N N = N N ( ε / S x ε , t ) , where spatial information is again encoded within the spatial scaling values S x ε . Less obvious from this relationship is the necessity of stress-strain state information at the spatial network input to account for the non-linear and time-dependent stress responses. This requirement can be understood by expanding on the explanation provided in [24] for the operation of spatial scaling values. When all materials are linear-elastic, the stiffness matrix D N N (recall its use in Eq. 18) computed from the weights of the MPN is constant. The role of S x ε is to vary the effective “stiffness” described by D N N based on position. Because the materials considered were linear-elastic, the modified stiffness matrix is independent of time and state of deformation, meaning S x ε is constant at x . Conversely, D N N computed from a non-linear MPN will not be constant, nor will the rate at which it changes be the same at all points. Therefore, values of S x ε will need to change based not only on position, but also the current (or previous) stress-strain states.

Quantitative elasticity imaging is a challenging problem. Much of the difficulty arises from the limited measurement data that can be acquired during a clinical US imaging exam. Model-based inverse methods address the ill-posed inverse problem by constraining the solutions through geometric and constitutive model assumptions, well-defined force loads, and application of regularization to the solution. While effective, deviation of measurement data from the model assumptions can introduce large artifacts in the reconstructed material property images, potentially corrupting diagnostic information, and the regularization method is almost never grounded in physical laws. Furthermore, parameters associated with the pre-selected constitutive model may only capture a portion of the tissue mechanical properties, missing information that could be important to clinical decision making.

AutoP and NNCMs are a fundamentally different approach to the inverse problem in elasticity imaging. Rather than constraining the solution through model assumptions, AutoP extracts material property information from force-displacement measurements through a combination of physical modeling and machine learning. In this review, we presented AutoP in a new way to highlight similarities of the procedure with well-known optimization methods. Specifically, we demonstrated how the principles of stress equilibrium and strain compatibility are exploited through application of measurement data as BCs in separate FEAs, from which naturally arises an implicit displacement error that drives NNCM training.

The primary benefit of AutoP over model-based inverse methods is the potential to discover 3-D mechanical properties of soft tissues in vivo. Full computational modeling of applied force loads means measurements need not be acquired in a constrained manner; e.g., uniaxial compression by a large platen. Nor is 3-D displacement data necessary to reconstruct material properties throughout a volume, as we demonstrated in [25]. However, the use of arbitrary force-displacement measurements as input to AutoP significantly increases the computational load. Furthermore, we expect intelligent sampling strategies will need to be developed in order to maximize the information content of measurement data and reduce the computational burden of AutoP. Good sampling strategies will likely be essential for effectively imaging non-linear, time-dependent material properties.

One particularly exciting prospect of CaNNCMs and AutoP is the ability to identify mechanical properties from a heterogeneous distribution of material types. Classic model-based inverse methods must first assume the form of the constitutive model for the entire tissue section. The solution is the spatial variation of corresponding parameters that best fits the measurement data. AutoP reverses this order to first build a non-parametric, soft-computational model that describes the spatially varying material properties encoded in force-displacement data. Parameter estimation occurs after relevant information has been extracted from measurements. Successfully developing this capability would represent a paradigm shift in not only QUSE, but inverse identification of material properties in general.

AutoP is at its core a method to solve boundary value inverse problems. It is naturally suited for elasticity imaging because the physical principles of stress equilibrium and strain compatibility are built into the forward model. These principles relate the available force-displacement measurement data to the desired stress and strain distributions. By combining the physical principles through forward modeling with the flexibility of ANNs, virtually any type of continuum material can be described non-parametrically. AutoP could be applicable to other inverse problems with well-defined physical laws relating measurement data to the desired quantities. Nevertheless, AutoP is just one example of integrating physical modeling with machine learning to build powerful methods of solving inverse problems.