Noncollinear antiferromagnetic order and effect of spin-orbit coupling in spin-1 honeycomb lattice

Effective pseudospin-1/2 honeycomb lattice materials such as Li2IrO3, Na2IrO3 and $\alpha$-RuCl3 have received much attention due to the presence of bond-dependent spin interactions such as in Kitaev's compass model. By comparison, spin-1 materials have received less attention. Motivated by the recently synthesized insulating nickelate Ni2Mo3O8, we consider spin-1 effective model on the honeycomb lattice. The Ni$^{2+}$ spin-1 moments form a complex noncollinear order with a nontrivial angle between adjacent spins, according to the recent neutron scattering measurements. In this work, we successfully explain the observed ordering using an effective spin model with the exchange parameters determined with the help of first principles electronic structure calculations. We find that spin-orbit coupling in the form of the Dzyaloshinskii-Moriya interaction, coupled with magnetic frustrations, is key to explain the observed noncollinear spin structure in this material.