Multiple magnetic transitions and complex magnetic structures in Fe$_2$SiSe$_4$ with the sawtooth lattice

The sawtooth lattice shares some structural similarities with the kagome lattice and may attract renewed research interest. Here, we report a comprehensive study on the physical properties of Fe$_2$SiSe$_4$, an unexplored member in the olivine chalcogenides with the sawtooth lattice of Fe. Our results show that Fe$_2$SiSe$_4$ is a magnetic semiconductor with band gap of 0.66~eV. It first undergoes an antiferromagnetic transition at T$_{m1}$=110~K, then an ferrimagnetic-like one at T$_{m2}$=50~K and finally a magnetic transition at T$_{m3}$=25~K which is likely driven by the thermal populations of spin-orbit manifold on the Fe site. Neutron diffraction analysis reveals a non-collinear antiferromagnetic structure with propagation vector $\mathbf{q_1}$=(0,0,0) at T$_{m2}$<T<T$_{m1}$. Interestingly, below T$_{m2}$, an additional antiferromagnetic structure with $\mathbf{q_2}$=(0,0.5,0) appears and Fe$_2$SiSe$_4$ exhibits a complex double-$\mathbf{q}$ magnetic structure which has never been observed in sawtooth olivines. Density functional theory calculations suggest this complex noncollinear magnetic structure may originate from the competing antiferromagnetic interactions for both intra- and inter-chain in the sawtooth lattice. Furthermore, band structural calculations show that Fe$_2$SiSe$_4$ has quasi-flat band features near the valence and conduction bands. Based on the above results, we propose Fe$_2$SiSe$_4$ as a new material platform to condensed matter researches.