Nematic spin liquid phase in a frustrated spin-1 system on the square lattice

Competing interactions in quantum spin systems promote a variety of unusual phases. An important example is the frustrated spin-$1$ model on the square lattice with the nearest-neighbor bilinear ($J_1$) and biquadratic ($K_1$) interactions, which has an enlarged SU(3) symmetry at $K_1=J_1$. We find evidence for a nematic spin liquid phase for a range of $K_1/J_1$ near the SU(3)-symmetric point, based on the linear flavor-wave theory (LFWT) and the density matrix renormalization group (DMRG) method. This phase displays no spin dipolar or quadrupolar order, preserves translational symmetry but spontaneously breaks $C_4$ lattice rotational symmetry, and possesses fluctuations peaked at the wavevector $(\pi, 2\pi/3)$. The excitation spectrum appears gapless, and the nematic order is attributed to the dominant $(\pi, 2\pi/3)$ fluctuations. Our results provide a novel mechanism for electronic nematic order and, more generally, open up a new avenue to explore frustration-induced exotic ground states.