Anisotropic RKKY interactions mediated by $j=3/2$ quasiparticles in half-Heusler topological semimetal

We theoretically explore the RKKY interaction mediated by spin-3/2 quasiparticles in half-Heusler topological semimetals in quasi-two-dimensional geometries. We find that while the Kohn-Luttinger terms gives rise to generalized Heisenberg coupling of the form ${\cal H}_{\rm RKKY} \propto σ_{1,i} {\cal I}_{ij} σ_{2,j}$ with a symmetric matrix ${\cal I}_{ij}$, addition of small antisymmetric linear spin-orbit coupling term leads to Dzyaloshinskii-Moriya (DM) coupling with an antisymmetric matrix ${\cal I}'_{ij}$. We demonstrate that besides the oscillatory dependence on the distance, all coupling strengths strongly depend on the relative orientation of the two impurities with respect to the lattice. This yields a strongly anisotropic behavior for ${\cal I}_{ij}$ such that by only rotating one impurity around another at a constant distance, we can see further oscillations of the RKKY couplings. This unprecedented effect is unique to our system which combines spin-orbit coupling with strongly anisotropic Fermi surfaces. We further find that all of the RKKY terms have two common features: a tetragonal warping in their map of spatial variations, and a complex beating pattern. Intriguingly, all these features survive in all dopings and we see them in both electron- and hole-doped cases. In addition, due to the lower dimensionality combined with the effects of different spin-orbit couplings, we see that only one symmetric off-diagonal term, ${\cal I}_{xy}$ and two DM components ${\cal I}'_{xz}$ and ${\cal I}'_{yz}$ are nonvanishing, while the remaining three off-diagonal components are identically zero. This manifests another drastic difference of RKKY interaction in half-Heusler topological semimetals compared to the electronic systems with spin-1/2 effective description.