Frustrated out-of-plane Dzyaloshinskii-Moriya interaction and the onset of atomic-scale 3$q$ magnetic textures in 2D Fe$_{3}$GeXTe (X = Te, Se, S) monolayers

We theoretically study the effect of in- and out-of-plane Dzyaloshinskii-Moriya interaction (DMI) on the magnetic ground states of two-dimensional (2D) Fe$_3$GeXTe (X=Te, Se, S) monolayers, where X=Se, S correspond to antisymmetric Janus structures with nonvanishing in-plane DMI. We perform atomistic spin simulations with the extended Heisenberg Hamiltonian parametrized by first principles calculations. While we find that the base DMI in all systems is too weak to stabilize noncollinear states, we show how the frustrated out-of-plane DMI tends to favor atomic-scale $3q$ magnetic textures at the edge of the Brillouin zone. Owing to the ability to tune the DMI in 2D magnets via applied strain or electric field, we study the evolution of the systems' ground state with increasing DMI amplitude. We find that nonplanar $3q$ states are favored under scaling factors as low as 3, while larger DMI tends to stabilize states reminiscent of nanoskyrmion lattices at the atomic-scale.