From a quasimolecular band insulator to a relativistic Mott insulator in $t_{2g}^5$ systems with a honeycomb lattice structure

The $t_{2g}$ orbitals of an edge-shared transition-metal oxide with a honeycomb lattice structure form dispersionless electronic bands when only hopping mediated by the edge-sharing oxygens is accessible. This is due to the formation of isolated quasimolecular orbitals (QMOs) in each hexagon, introduced recently by Mazin et al. [Phys. Rev. Lett. 109, 197201 (2012)], which stabilizes a band insulating phase for $t_{2g}^5$ systems. However, with help of the exact diagonalization method to treat the electron kinetics and correlations on an equal footing, we find that the QMOs are fragile against not only the spin-orbit coupling (SOC) but also the Coulomb repulsion. We show that the electronic phase of $t_{2g}^5$ systems can vary from a quasimolecular band insulator to a relativistic $J_{\rm eff}=1/2$ Mott insulator with increasing the SOC as well as the Coulomb repulsion. The different electronic phases manifest themselves in electronic excitations observed in optical conductivity and resonant inelastic x-ray scattering. Based on our calculations, we assert that the currently known Ru$^{3+}$- and Ir$^{4+}$-based honeycomb systems are far from the quasimolecular band insulator but rather the relativistic Mott insulator.