Revealing the magnetic coupling mechanism is often a critical step toward understanding the role of magnetism in intriguing phenomena such as colossal magnetoresistance (CMR), high T_C superconductivity, multiferroicity or frustration in correlated electron materials [1–5]. By way of example, the indirect double- and super-exchange interactions were successfully elaborated in qualitatively explaining the CMR effect and associated magnetic orders based only on the spin and charge degrees of freedom [1]. In 4f-based insulators, the indirect oscillating interaction [4–7] between pairs of localized 4f moments via the intermediary of valence electrons is blocked. Therefore, possible super-, dipole-dipole and multipolar, and Dzyaloshinsky-Moriya (DM) exchange interactions are primarily responsible for potential magnetic ordering [4]. Without detailed knowledge of the structural and magnetic parameters, it is hard to uniquely determine which interaction acts as the major exchange mechanism [8]. In this case, the origins of the related incommensurable spin structures become elusive [4]. In addition, the competition between spin-orbital coupling and crystal electric field (CEF) at low temperatures largely affects the highly-degenerate Hund's rule ground state, and besides the anisotropic dipolar and DM interactions, determine the magnitude of the magnetic anisotropy [9, 10]. This anisotropy strongly influences the degree of magnetic frustration. Sometimes, it may disorder or even quench potential magnetic moments, leading to a virtually nonmagnetic ground state [11].

Magnetic frustration can lead to novel quantum states such as spin liquid, spin ice, cooperative paramagnetism or the magnetic Coulomb phase based on magnetic monopole excitations, providing an excellent testing ground for theories [4, 12–25]. A Monte Carlo simulation indicates that the observed diffuse scattering in SrEr₂O₄ originates from a ladder of Er triangles [26]. A computation of the crystal-field levels demonstrates site-dependent anisotropic single-ion magnetism in the compounds of SrHo₂O₄ and SrDy₂O₄ [27]. Lanthanide-based magnetic compounds, e.g., edge-sharing tetrahedra, corner-sharing spinels, or triangular Kagomé and pyrochlore lattices, often show anomalous magnetic properties due to geometric frustration [28–31]. The family of SrRE₂O₄ (RE = Y, Gd, Ho, Yb) compounds was first synthesized in 1967 [32]. Recently, a study on polycrystalline SrRE₂O₄ (RE = Gd, Dy, Ho, Er, Tm, Yb) samples demonstrates that they adopt the orthorhombic structure [33] with a geometric frustration for the magnetic ions revealed by the existence of magnetic short-range orders down to ~1.5 K [34]. Subsequently, single crystals of SrRE₂O₄ (RE = Y, Lu, Dy, Ho, Er) were successfully grown [35]. Single-crystal neutron-scattering studies on SrRE₂O₄ (RE = Ho, Er, Yb) compounds with respective antiferromagnetic (AFM) transition temperatures at 0.62, 0.73, and 0.9 K were reported [36–40], generally confirming that there exists a coexistence of long- and short-range magnetic orders. It is pointed out that for the case of SrHo₂O₄, Young et al. [41] observed only a short-range spin order inconsistent with other reports [36, 37, 40]. Further experimental tests would be necessary to address this discrepancy. Since the adopted orthorhombic structure accommodates two RE sites (RE1 and RE2), it is hard to derive the crystallographic origins of the two types of spin ordering. In addition, single-crystal SrDy₂O₄ displays only weak diffuse magnetic scattering which persists down to ~20 mK [42]. The low transition temperatures of the magnetic orders, to some extent, prevent a complete understanding of the nature of magnetic interactions and frustrations in the family. To overcome these problems and address the relevant interesting physics necessitate a search in the SrRE₂O₄ (RE = rare earth) family for a new compound that displays a higher Néel temperature, thus permitting a technically easier study of the two coupling mechanisms.

In this study, we report on a new frustrated member to the family of SrRE₂O₄, namely SrTb₂O₄, which has not been studied yet by neutron scattering. The single-crystal SrTb₂O₄ displays a long-range magnetic order relative to the underlying lattice. The noncollinear incommensurate AFM structure forms at T_N = 4.28(2) K upon cooling. The synthesis of SrTb₂O₄ with the highest Néel temperature in the family opens up an easier route to elucidate the magnetic coupling and frustrating mechanisms. By polarized and unpolarized neutron scattering we uniquely determine the detailed structural and magnetic parameters to understand the magnetism in SrTb₂O₄.

Polycrystalline samples of SrTb₂O₄ were synthesized from stoichiometric mixtures of SrCO₃ (99.99%) and Tb₄O₇ (99.99%) compounds by standard solid-state reaction [43]. Both raw materials were preheated at 800° for 12 h and weighted at ~200°. The mixed and milled raw materials were calcined twice at 1473 and 1573 K for 48 h each in air in order to perform decarbonization and prereaction. The resulting powder was pressed into cylindrical rods with an isostatical pressure of ~78 MPa. The rods were sintered two times at 1573 and 1673 K for 48 h at each temperature in air. After each round of the isostatic pressing and subsequent firing, the product was reground and ball-remilled, which results in a dense and homogenous sample and ensures a complete chemical reaction. The single crystal of SrTb₂O₄ was grown by optical floating-zone method with an atmosphere of ~98% Ar and ~2% O₂. The growing speed is ~4 mm/h with rotations of the feed and seed rods at +32 and −28 rpm, respectively. The phase purity of the polycrystalline and single-crystalline samples was checked by in-house X-ray powder diffraction. The electrical resistivity of a bar-shaped single crystal by standard dc four-probe technique was measured on a commercial physical property measurement system.

High-resolution neutron powder diffraction (NPD) patterns were collected with a pulverized SrTb₂O₄ single crystal (~5 g) mounted in a ³He insert on the structure powder diffractometer (SPODI) [44] with constant wavelength λ = 2.54008(2) Å at the FRM-II research reactor in Garching, Germany.

The SrTb₂O₄ single crystal (~2.2 g) for the neutron-scattering studies was oriented in the (H, K, 0) scattering plane with the neutron Laue diffractometer OrientExpress [45] and the IN3 thermal triple-axis spectrometer at the Institut Laue-Langevin (ILL), Grenoble, France. The mosaic of this single crystal is 0.494(5)^o full width at half maximum (FWHM) for the nuclear (2, 0, 0) Bragg reflection at 1.5 K. Longitudinal XYZ neutron polarization analysis [46] was carried out on the D7 (ILL) diffractometer with a dilution fridge and λ = 4.8 Å. Unpolarized elastic neutron-scattering studies were performed at the two-axis D23 diffractometer (ILL) with incident wavelength 1.277 Å and the IN12 (ILL) cold triple-axis spectrometer with fixed final energy of 5.3 meV and the beam collimation set as open-40′-sample-60′-open.

Here the wave vector Q_(HKL) (Å⁻¹) = (Q_H, Q_K, Q_L) is defined through (H, K, L) = (a2πQH,b2πQK,c2πQL) quoted in units of r.l.u., where a, b, and c are relevant lattice constants referring to the orthorhombic [33] unit cell.

Figure 1 shows the neutron polarization analysis in the spin-flip (SF, i.e., Z-flipper on) and non-spin-flip (NSF, i.e., Z-flipper off) channels. Compared with the maps at 300 K (Figure 1A), it is clear that extra fourfold Bragg peaks around (±1.6, ±1, 0) appear symmetrically in both SF and NSF reciprocal space maps at 50 mK (Figure 1B) due to a long-range magnetic transition. Polarized neutron magnetic scattering depends on the direction of the neutron polarization P^ with respect to the scattering vector Q^, and also the direction of the ordered-moments μ^. In our case, P^ (Z-component) ‖ c-axis [46], and the magnetic Bragg reflections are observed in the (H, K, 0) plane, i.e., P^⊥Q^. In this case, the neutron-scattering cross sections of the NSF and SF channels are

respectively. The first and the second terms in each equation refer to the magnetic and spin-incoherent scatterings, respectively. The third term in Eq. (1) denotes nuclear and isotope incoherent contributions [46]. The presence of the incommensurable AFM Bragg peaks in the NSF channel (Figure 1B) indicates that one component of μ^ is parallel to the c axis, while their appearances in the SF channel imply a μ^ component lying in the ab plane.

We observe the magnetic Bragg peak only at 0.5 K in our NPD study (Figure 2). We thereby refine the AFM wave vector exactly as Q_AFM = (0.5924(1), 0.0059(1), 0) by the profile-matching mode [47] and a total moment |μ^| = 1.92(6) μ_B at the maximum amplitude for the Tb1 ions only with the b- and c-components equalling to +1.88(8) and +0.40(23) μ_B (Table 1), respectively. The moment size of the Tb2 site is negligible. Figure 3 schematically shows the resulting crystal and magnetic structures as well as the structural parameters for the bent Tb₆ honeycombs. The temperature dependence of the AFM (1.6, 1, 0) Bragg peak is shown in Figure 4. The extracted integrated intensity (I) was fit to a power law I=I0(1−TTN)β, which produces a Néel temperature T_N = 4.28(2) K, and a critical exponent β = 0.55(2) probably indicative of a second-order type phase transition and possible three-dimensional Heisenberg-like spin interactions [48].

We record a reciprocal space map (Figure 5A) around the AFM (1.6, 1, 0) Bragg peak at 1.7 K using D23, and the central scans along the q_H and q_K directions (Figure 5B) were measured at IN12. In both figures, the FWHM of the magnetic Bragg peak along the q_H and q_K directions is sharply different. Both magnetic Bragg peaks are broader than the nuclear Bragg (2, 0, 0) reflection in the reciprocal space as shown in Figure 5B, which indicates that the observed magnetic Bragg peaks are beyond the instrument resolution. Therefore, Figure 5B shows a real in-plane magnetic anisotropy.

To quantitatively estimate the in-plane anisotropy, we take the FWHM of the nuclear Bragg (2, 0, 0) peak as the detecting accuracy which is convoluted in fitting the magnetic peaks by a Gaussian function shown as the solid lines in Figure 5B. This results in FWHM = 0.0183(1) and 0.0492(2) Å⁻¹ along the q_H and q_K directions, respectively, implying highly anisotropic in-plane spin correlations consistent with the observation that strong magnetic frustration exists in SrTb₂O₄. We roughly estimate the spin-correlation length (ξ) by ξ=2πFWHM, i.e., ξ_H = 343.7(22) Å and ξ_K = 127.6(4) Å. Therefore, ξHξK=2.69(2). Similar in-plane anisotropic magnetic correlations were also observed in the iron-based superconductors [49–53] that are highly frustrated, too, where its microscopic origin, from the ellipticity of the electron pockets or the competing exchange interactions associated with the local-moment picture, is still highly debated [54–57]. It is undoubted that the observed in-plane magnetic anisotropy in SrTb₂O₄ indicates an appearance of the competing spin exchanges and is certainly associated with a description of the purely-localized magnetism of ionic Tb³⁺ ions. A deeper understanding of the insulating state necessitates theoretical band structure calculations. We tentatively estimate the compatibility between ordered magnetic and nuclear crystalline domains based on the non-deconvoluted FWHM (κ) of the Bragg (1.6, 1, 0) (κ_m = 0.0300(7) Å⁻¹) and (2, 0, 0) (κ_n = 0.0238(2) Å⁻¹) peaks, i.e., κ_n/κ_m = 79(2)%, which implies that the incommensurate AFM structure orders in a long-range fashion relative to the underlying lattice of the single crystal.

We further analyze the spin-correlation length with our NPD data (Figure 2C). Firstly, it is pointed out that the positive and negative momenta cannot technically be differentiated in a NPD study. As shown in Figure 6A, taking into account the corresponding SPODI instrument resolution (dashed line) [46], a Gaussian fit (solid line) to the AFM Bragg (1.5924, 1.0059, 0) peak (squares) results in an average ξ_AFM = 864(36) Å in real space. This indicates that the AFM ordering observed in SrTb₂O₄ is indeed of long range in character in comparison with the reported extremely-broad magnetic diffuse scattering which was attributed to the presence of short-ranged magnetic ordering in polycrystalline SrRE₂O₄ (RE = Ho, Er, Dy) samples in the study of Ref. [34]. With the same method utilized in Figure 6A, we also analyze the NPD peak of the nuclear Bragg (2, 0, 0) reflection as shown in Figure 6B and extract that ξ₍₂₀₀₎ = 1304(34) Å. This indicates that ξ_AFM/ξ₍₂₀₀₎ = 66(3)% basically in accord with the compatibility between ordered magnetic and nuclear crystalline domains extracted with our single-crystal neutron-scattering data. Since our NPD data were collected from a pulverized SrTb₂O₄ single crystal, that ξ_AFM is ~2.5 times larger than ξ_H may indicate that there have strong magnetic and crystalline domain effects in single-crystal SrTb₂O₄, or a large part of spins are blocked probably due to a pining effect by strains accumulated during single crystal growth. In any case, this difference between single-crystalline and polycrystalline samples in turn supports the fact that there is a strong magnetic frustration in single-crystal SrTb₂O₄. Further studies with high pressures would be of great interest.

In most cases, the strength of the indirect magnetic interactions such as conventional double- or super-exchange [1] can be influenced more or less by the value of the revelent bond angle [58–61], e.g., the ∠Tb-O-Tb bond angles in SrTb₂O₄ as listed in Table 1 (see also Figure 7). However, the respective values of ∠Tb-O-Tb display no appreciable difference within accuracy between 0.5 and 10 K (Table 1), below and above the T_N, respectively, which may indicate an invalidity of the two conventional magnetic coupling mechanisms (double- or super-exchange) in SrTb₂O₄. This is consistent with the study of SrTm₂O₄ [11] and in excellent agreement with our transport study, where any attempts to measure possible resistivity in SrTb₂O₄ from 2 to 300 K were fruitless. We estimate that the resistance of the single crystal measured is beyond at least 10⁶ ohm. We thus conclude that SrTb₂O₄ is a robust insulator, and the electrons responsible for the incommensurable antiferromagnetism are mainly from the localized 4f⁸ shell of the ionic Tb³⁺ ions. In this localized picture, the interionic exchange interactions dominate for the formation of the magnetic structure [4, 6]. The nearest Tb neighbors are stacked linearly along the c axis (Figure 3B). The shortest Tb1-Tb1 and Tb2-Tb2 have the same bond length. However, the NN Tb1 ions have a ferromagnetic (FM) arrangement. By contrast, the interaction between the NN Tb2 ions is blocked unexpectedly (Figure 3B). There is no appreciable difference in the NN Tb-Tb bond length, i.e., the c lattice constant, between 0.5 and 10 K (Table 1), which probably rules out the potential direct exchange interaction consistent with the fact that unpaired 4f electrons are deeply embedded under the 5s²p⁶ shells and also indicates that the prevalent dipole-dipole interaction is subjected to some condition, i.e., the octahedral distortion as discussed below, in agreement with the study of SrTm₂O₄ [11].

As a non-Kramers ion, Tb³⁺ (S = 3, L = 3, J = 6, g_J = 1.5) in principle keeps the time reversal symmetry and doesn't show any energy degeneracy in the presence of the purely-localized electric field. However, we refine two kinds of octahedra as shown in Figure 7: Tb1O₆ and Tb2O₆, corresponding to the partially-ordered and totally-frozen Tb1 and Tb2 ions, respectively. The average octahedral distortion [58, 59] can be quantitatively measured by the parameter Δ defined as: Δ=16∑n = 16[ (dn−〈d〉)〈d〉 ]2, where d_n and 〈d〉 are the six Tb-O bond lengths along the six crossed directions (Figure 7) and the mean Tb-O bond length (Table 1), respectively. It is noteworthy that the Δ values of the Tb1 and Tb2 ions are in the same magnitudes as those of the Mn³⁺ Kramers and Mn⁴⁺ non-Kramers ions, respectively, in the Jahn-Teller (JT) distorted regime of single-crystal La78Sr18MnO₃ [60]. This sharp contrast implies that the Tb1 ions are strongly distorted, while the Tb2 ions behave normally within the non-Kramers scheme. Therefore, the Δ magnitude that reflects the ion local symmetry and thus the strength of the surrounding CEF directly determines the existence of the magnetic ordering, which is supported by the observation that below T_N the respective Δ values of the Tb1 and Tb2 ions change oppositely with temperature (Figure 8). We therefore infer that one possible reason for the formation of the incommensurable magnetic structure is the modulated distribution of the 4f¹6s² valence electrons which modify the surrounding environment experienced by the localized unpaired 4f electrons. The corresponding modulation of the local symmetry may plausibly be attributed to the spatial zigzag-type Tb arrangements along the a and b axes in the process of forming the crystallographic domains. This is supported by the fact that the honeycomb columns run straightly along the c axis, and there is no spin modulation at all in that direction.

Based on the refined Tb-O bond lengths, we deduce two distortion modes for the Tb1O₆ and Tb2O₆ octahedra (Figure 7), respectively. The possible product of the Tb1 subjected stress-vectors (small arrows) should point qualitatively to the direction of the Tb1 moment, implying a strong single-ion anisotropy. This JT-like distortion mode leads to the large Δ value of the Tb1 ions, and possibly lifts further the degenerate multiplets. By contrast, the Tb2 ions are subjected to opposing stresses in all the three pair-directions. In this case, the octahedral distortion strongly depends on their competing strengths. This mode makes the small Δ value of the Tb2 ions and their potential total magnetic moments quenched vitally.

The maximum Tb1 moment size is mere 1.92(6) μ_B, 21.3(7)% of the theoretical saturation value (g_J J = 9 μ_B). It is of particular interest to explore the frustrating mechanism. The virtual non-Kramers state of the Tb2 site reduces the total moment size per molar formula by 50%. The Tb1 moment fluctuates like a wave defined as μ^=|μ^max|cos(QAFM·Rx+ϕ), where R_x is a spin coordinate along the a axis, and ϕ is a phase parameter. The existence of the strong single-ion anisotropy indicates a large CEF effect which should be comparable to the energy scale of the magnetic interactions. We have shown the clear evidence for a large magnetic exchange anisotropy (Figure 5), which is ascribed to the anisotropic dipole-dipole interaction. The NN magnetic arrangement is FM (Figure 3B), implying no possibility for a magnetic frustration. The NNN magnetic configurations display a dual character, i.e., FM and AFM for the equivalent Tb11 and Tb12 sites, respectively. This sharp difference may frustrate the Heisenberg-exchange coupled NNN spins.

To summarize, we have synthesized large enough SrTb₂O₄ single crystals suitable for neutron scattering and revealed a modulated spin structure in SrTb₂O₄ with the highest AFM transition temperature at T_N = 4.28(2) K in the SrRE₂O₄ family, which provides a technically friendly platform to explore the related magnetic coupling and frustrating mechanisms. Our studies show that the localized Tb1 moments lie in the bc plane with the FM chains along both the b and c directions and the AFM modulation mainly along the a axis. We have found two distinct octahedra for the non-Kramers Tb³⁺ ions: Tb1O₆ being strongly distorted, corresponding to the partially-ordered moments; Tb2O₆ being frustrated entirely in the non-Kramers state. Therefore, the octahedral distortion has a decisive influence on the Hund's rule magnetic ground state (⁷F₆) and the related frustrations. The magnetocrystalline anisotropy is crucial in determining the direction of the ordered moments. The direct NN interaction results in a FM arrangement for the Tb1 ions along the c axis, and the different NNN Tb configurations (FM and AFM) further lift the magnetic frustration. The present results make SrTb₂O₄ a particularly significant compound in the family for theoretical and further experimental studies. Inelastic neutron-scattering studies to determine the detailed crystal-field and magnetic-interaction parameters would be of great interest. The factors that influence the value of the AFM transition temperature would be further explored in combination with theoretical calculations.

Cong Zhang, Paul Meuffels, and Hai-Feng Li prepared the polycrystals and grew the single crystals. Anatoliy Senyshyn and Hai-Feng Li performed the SPODI experiments and analyzed the data. Martin Boehm, Binyang Hou, and Hai-Feng Li performed the IN3 experiments. Andrew Wildes and Hai-Feng Li performed the D7 experiments and analyzed the data. Karin Schmalzl, Wolfgang Schmidt, and Hai-Feng Li performed the IN12 experiments and analyzed the data. Wolfgang Schmidt, Karin Schmalzl, Eric Ressouche, and Hai-Feng Li performed the D23 experiments. Hai-Feng Li, Cong Zhang, Anatoliy Senyshyn, Andrew Wildes, Karin Schmalzl, Wolfgang Schmidt, Martin Boehm, Eric Ressouche, Binyang Hou, Paul Meuffels, Georg Roth, and Thomas Brückel discussed and analyzed the results. Hai-Feng Li wrote the main manuscript text. Cong Zhang, Anatoliy Senyshyn, Andrew Wildes, Karin Schmalzl, Binyang Hou, Georg Roth, and Thomas Brückel commented on the manuscript and all authors reviewed the paper. Hai-Feng Li conceived and directed the project.