Novel magnetic domain structure, which is observed in various magnetic materials, is the consequence of competition among various magnetic interactions that are executed on the materials both from intrinsic and extrinsic circumstances [1–4]. In particular, the intrinsic magnetic interactions that occur in materials with kagome crystal is of great complexity and results in complicated magnetic domains, for example, a frustrated circumstance would occur and lead to noncollinear magnetic spin structure [5–10]. Recently, topologically nontrivial skyrmions have been predicted to generate in kagome crystal when the crystal owns both Heisenberg and Dzyaloshinskii–Moriya interactions [11–13], and finally, the skyrmion bubbles are experimented and reported in Fe₃Sn₂ without any Dzyaloshinskii–Moriya interaction [14–17], showing that the uniaxial magnetic anisotropy in kagome crystal can play a vital role in stabilized novel spin structure. On the other hand, the magnetic and electronic structures in kagome crystals are highly entangled, making them show both fruitful spin-orbit and electronic corrected effects. Consequently, the conduction electrons show both an intrinsic anomalous Hall effect which links to reciprocal band structure and a topological Hall effect which related to real space nontrivial spin textures [18–22]. The exotic phenomena are confirmed in many kagome crystals like Fe₃Sn₂ [23, 24], Co₃Sn₂S₂ [25, 26], YMn₆Sn₆ [27], etc. Evenly, the kagome lattices may also host Dirac electronic states, which leads to topological and Chern insulating phases, which have been confirmed both by theoretical and experimental investigation on Fe₃Sn₂ [23, 24] and TbMn₆Sn₆ [28]. Recently, not only TbMn₆Sn₆ but also RMn₆Sn₆ (R = Y, Dy, Ho, etc.) have been reported to show a very large anomalous Hall effect and a large topological Hall effect [27, 29, 30], stimulating vast investigations in the field and materials of kagome lattices.

DyMn₆Sn₆ is a member of kagome metal RMn₆Sn₆ family (R = rare element) which consists of a two-dimensional (2D) Mn layer kagome lattice. It has a hexagonal HfFe₆Ge₆-type structure with P6/mmm space group [31]. It has a ferrimagnetic behavior below T _C = 393 K with Dy and Mn sublattices ordering simultaneously due to the strong Mn-Dy antiferromagnetic coupling [32]. Neutron diffraction study shows that above room temperature, it possesses a planar anisotropy for the Mn sublattice but with a negative value for Dy, resulting in a partial spin reorientation below T _t ∼ 240 K [31]. Considering both the spontaneous magnetization with kagome lattice in DyMn₆Sn₆, a conical arrangement of the spin magnetic moment is inferred in the previous report. Recently, the transport, and therefore the electronic structure, has been intensively studied to uncover the possible Dirac and Chern insulating state nature [28]; however, the directional observation of the spin texture has not been reported. The relationship between the complex electronic correlated novel state and the novel spin structure, in other words, the link between the degrees of charge and spin for the electrons in kagome lattices, is highly elusive.

Microscopic magneto-optical Kerr effect (micro-MOKE), which refers to the polarization rotation of a linearly polarized light when reflected by magnetized materials, is widely used to map the magnetic domain patterns and magnetic phase transitions in various materials [33–35]. In this study, the magnetic domain in DyMn₆Sn₆ crystal and its evolution with temperature and magnetic field was investigated using the micro-MOKE imaging technique. We evidenced that the spin reorientation occurs between 255 and 235 K. In addition, we observed two types of magnetic domain structures that coexist and compete with each other simultaneously in DyMn₆Sn₆. Our results can provide a platform to understand the novel magnetic domain structure in the kagome metal family.

DyMn₆Sn₆ single crystals were grown by the flux method [36]. The constituent elements Dy, Mn, and Sn of better than 3N purity were weighed in the mole ratio of Dy:Mn:Sn = 1:6:30. The sample was placed in an alumina crucible and sealed in an evacuated silica ampoule. The ampoule was heated to 1,273 K and kept at the temperature for 24 h, and then slowly cooled to 1,223 K. It was heated again up to 1,263 K and finally cooled down slowly to 873 K at a rate of 6 K/h. The ampoule was quickly removed from the furnace. Sn flux was removed using a centrifuge. After cooling down to room temperature, the ampoule was broken by tools, and plate-like DyMn₆Sn₆ single crystals of about 2 × 2 × 0.5 mm³ in size were obtained from the alumina crucible. The quality of the crystals was checked by X-ray diffraction (XRD). Magnetization was carried out using the quantum design magnetic property measurement system (MPMS-XL5) for 1.8 K < T < 400 K and the H < 7 T.

The crystal was exfoliated using Torr seal glue to get smooth sample surface. Then micro-MOKE images of the surface magnetic domains were obtained at several temperatures using a polarizing microscope (Olympus, BX53M) equipped with a homemade option which could provide low temperature circumstances with liquid nitrogen. The magnetic fields executed on the sample during measurements were generated by calibrated permanent magnets.

Figure 1A shows the crystal structure of DyMn₆Sn₆. It has an hexagonal HfFe₆Ge₆-type structure with P6/mmm space group with two-dimensional (2D) Mn layer kagome lattice. Figure 1B shows the XRD pattern of a DyMn₆Sn₆ single crystal. The DyMn₆Sn₆ single crystal is high quality due to the very sharp diffraction peaks. Temperature-dependent magnetization (M–T) curves with B//c and B//ab plane and magnetic field-dependent magnetization (M–H) curves with B//c are shown in Figure 1C,D, respectively. There is an obvious spin reorientation transition in DyMn₆Sn_6, the same as that in TbMn₆Sn₆ and HoMn₆Sn₆ [32]. The curve at 300 K in Figure 1D is obviously different from the curve at 250 K. It means that the spin reorientation happens between 250 and 300 K.

In order to understand the material behavior of magnetic materials, the direct visualization of the domain structure is required. Figure 2A is a MOKE image of the magnetic domain of DyMn₆Sn₆ crystal at 173 K. In Figure 2A, we discover that there are two types of magnetic domains in DyMn₆Sn₆. Type-I domain is the belt domain, while type-II domain is the new stripe domain with complex construction, as shown in Figure 2B,C, respectively. The type-I domain shows that there are many white elliptic domains in the black belt domains. The type-II domain shows that the main part is a wide and long domain and the other part is a thin, short domain with the same direction. The thin, short domain locates in the middle of the black main domain and the white main domain. Figure 2D,E shows the line profile of the brightness distribution as marked by the dotted lines in Figure 2B,C. The intensity of black elliptic domains is stronger than the black belt domains in Figure 2D. Inferred from the theory of two-phase branching [37], the elliptic domains could be considered as floating on the belt domains, as shown in Figure 2F. Similar to the type-I domains, the intensity of black stripe domains is stronger than the black main domains in Figure 2E. It is quite likely that the stripe domains float on the main domains as shown in Figure 2G. From the brightness of the magnetic domains, the magnetic structure in DyMn₆Sn₆ is reasonably clear, where the sketch map of two types of magnetic domain structures are shown in Figure 2F,G, respectively.

From the above observation, one can note that for DyMn₆Sn₆, the domain shows refinement structure, reminiscent of the domain branching phenomenon in very large crystals with strongly misoriented surfaces. For DyMn₆Sn₆, it shows three-dimensional branching, as shown in Figure 2. Based on the theory of two-phase branching, the refinement structure that occurs in DyMn₆Sn₆ could be understood when one takes the uniaxial magnetic anisotropy. Usually, the theory of two-phase branching consists of three energy terms [37]: 1) the wall energy that increases toward the surface with increasing wall density, 2) the generalized closure energy depending only on the domain width of the last generation, and 3) the energy connected with the internal stray fields. It is rather complicated and difficult to extract explicit calculation, especially for the three-dimensional structure, as in the case of DyMn₆Sn₆. However, the theory applies to both Co- and Dy-modified NdFeB crystals with thickness of about several hundred micrometers. The theory and experimental result confirmed that the occurrence of domain branching has the relationship to both sample thickness and strong enough uniaxial magnetic anisotropy. Based on these previous results, we propose that the branching phenomenon observed in DyMn₆Sn₆ has the same mechanism with Dy-modified NdFeB, since DyMn₆Sn₆ is a strong uniaxial ferromagnet.

Figure 3 displays the magnetic domain structure as a function of temperature. From 255 to 235 K, the spin reorientation (SR) is observed and shown in Figure 3A–C. The magnetic domain is not clear until the temperature decreases to 245 K. When the temperature is below 245 K, the domain appears but is misty. It becomes obvious when the temperature continues decreasing. It is also shown that the spin reorientation in DyMn₆Sn₆ occurs gradually.

For investigating these two types of domains, we study the magnetic domains in DyMn₆Sn₆ with the temperature decrease. In Figure 3D–G, we observe the coexistence of two types of magnetic domains from 230 to 140 K. At the beginning, the type-II domains are in the vast majority at 230 K. When the temperature keeps decreasing, the type-II domains transform into type-I domains. The type-I domain also changes, in which the white elliptic domain in the black belt domain becomes little and disappears with the temperature decrease. At 140 K, all the type-II domains disappear and only a few white elliptic domains are left. The white elliptic domains disappear and the type-I domain does not show any edges nor branches at 110 K. As the temperature continues decreasing, the magnetic domain does not change any more. As shown above, the magnetic domain structure at 110 K is a kind of stripe domain due to the strong perpendicular anisotropy. At the beginning, the type-II domains are in the vast majority due to the weak perpendicular anisotropy. When the temperature decreases, the perpendicular anisotropy becomes strong making the magnetic moment apt to move along the perpendicular plane. We find that type-I domain is less affected than the type-II domain. The domains floating on other domains may reverse under the influence of the perpendicular anisotropy. It is obvious that the type-II domain is more sensitive to the perpendicular anisotropy than that of type-I domain. Consequently, the type-II domains transform into type-I domain when the temperature continues decreasing. The type-I domain also changes because of the perpendicular anisotropy. At last, the perpendicular anisotropy only makes the stripe domain left. The stripe domain is affected by the demagnetizing field.

For investigating the type-II domain in detail, from 180 to 160 K, the structure of the type-II domains is observed. In Figure 3I,J, we find that our description of the type-II domains is the same as the domain in Figure 2A. It is shown that the structure of the type-II domains changes with temperature. As the temperature decreases, the main domain becomes wider, but the other part keeps the same width. The temperature changes the number of the other part domain. The lower the temperature is, the lower the number is. It is the main reason that the perpendicular anisotropy is enhanced as the temperature decreases. The perpendicular anisotropy can make the magnetic moment apt to move along the perpendicular plane. So, the main domain becomes wider and the number of the other stripe domains decreases. The type-I domain is a kind of manifestation of the branch domain at low temperature [37]. As the temperature increases, the branch structure grows on the type-I domain. The physics behind the domain transformation is the temperature dependence of magnetic anisotropy, especially in kagome lattice with strong uniaxial anisotropy. When the temperature decreases, the coefficients for anisotropy changes, leading to spin reorientation and domain variations, as shown in Figure 3.

To make sense of the domains clearly, we perform the experiment with the various magnetic fields at two representative temperatures, 223 and 173 K, as shown in Figure 4. The variable magnetic field of 0 or 510 Oe pointing up (+) or down (−) was applied by a permanent magnet. At 223 K, type-I domains sparsely appear while type-II domains are in the vast majority. One side of the white stripe domains flip to the black domain with the magnetic field increase and the other side of space between the white stripe domains becomes wider. With the magnetic field on, some of the type-II domains transform into the type-I domains. When the magnetic field is removed, the domain recovers. However, the type-I domain to the type-II domain transform does not recover, showing very strong irreversible effect. With the magnetic field reverse, a similar phenomenon happens. One side of the black stripe domains flips to the white domains and the other side of space between the black stripe domains becomes wider. Some type-II domains transform into type-I domain and do not recover without magnetic field assistance. At 173 K, we discover that the type-II domains in Figure 4F are far less than the type-II domains in Figure 2A. This is because the image in Figure 2A is obtained during zero field cooling (ZFC) process, but the image in Figure 4F is obtained during cooling process from the state in Figure 4E. Both the magnetic field and the temperature can make the type-II domain transform into type-I domain. At 173 K, we observe the same phenomenon. After a loop in the magnetic field, the area of the type-II domain decreases. From the above results, we can conclude that the type-I domain is more stable than the type-II domain.

The phase diagram of magnetic domains is obtained in Figure 4H. At T > 235 K, the spin reorientation happens. There is no domain in the gray area. Type-II domain appears, as the temperature decreases, as shown in the light blue part. Type-II domain transforms into type-I domain under the influence of temperature and magnetic field. Type-I domain is shown in the yellow part. Type-II domain and type-I domain coexist locating at the intermediate region, as shown in the figure. As the temperature continues decreasing, only stripe domains are left, as marked by red color in the figure.

In summary, the magnetic domains in kagome lattice metal DyMn₆Sn₆ are observed using the micro-MOKE technique. Spin reorientation in DyMn₆Sn₆ is observed directly by the micro-MOKE technique from 255 to 235 K. There are two types of magnetic domains in DyMn₆Sn₆. Type-I domain is the belt domain. Type-II domain is the new stripe domain. The type-II domain exists from 160–235 K (ZFC). We analyze the structure of two types of domains. As the temperature decreases, the perpendicular anisotropy becomes strong, making the type-II domain transform into the type-I domain and even disappear. With a varying magnetic field experiment, we find that the type-II domain is less stable than the type-I domain. The phase diagram of magnetic domains in DyMn₆Sn₆ is obtained. However, the formation mechanism of new stripe domain in DyMn₆Sn₆ needs to be further investigated in the future.