Quantum Transitions of Nematic Phases in a Spin-$1$ Bilinear-Biquadratic Model and Their Implications for FeSe

Since its discovery, iron-based superconductivity has been known to develop near an antiferromagnetic order, but this paradigm fails in the iron chalcogenide FeSe, whose single-layer version holds the record for the highest superconducting transition temperature in the iron-based superconductors. The striking puzzle that FeSe displays nematic order (spontaneously broken lattice rotational symmetry) while being non-magnetic, has led to several competing proposals for its origin in terms of either the $3d$-electron's orbital degrees of freedom or spin physics in the form of frustrated magnetism. Here we argue that the phase diagram of FeSe under pressure could be qualitatively described by a quantum spin model with highly frustrated interactions. We implement both the site-factorized wave-function analysis and the large-scale density matrix renormalization group (DMRG) in cylinders to study the spin-$1$ bilinear-biquadratic model on the square lattice, and identify quantum transitions from the well-known $(π,0)$ antiferromagnetic state to an exotic $(π,0)$ antiferroquadrupolar order, either directly or through a $(π/2,π)$ antiferromagnetic state. These many phases, while distinct, are all nematic. We also discuss our theoretical ground-state phase diagram for the understanding of the experimental low-temperature phase diagram obtained by the NMR [P. S. Wang {\it et al.}, Phys. Rev. Lett. 117, 237001 (2016)] and X-ray scattering [K. Kothapalli {\it et al.}, Nature Communications 7, 12728 (2016)] measurements in pressurized FeSe. Our results suggest that superconductivity in a wide range of iron-based materials has a common origin in the antiferromagnetic correlations of strongly correlated electrons.