Radiative Transfer Distortions of Lyman-$α$ Emitters: a New Fingers-of-God Damping in the Clustering in Redshift Space

Complex radiative transfer (RT) of the Lyman-$α$ photons poses a theoretical challenge to galaxy surveys which infer the large-scale structure with Lyman-$α$ emitters (LAEs). Guided by RT simulations, prior studies investigated the impact of RT on the large-scale LAE clustering, and claimed that RT induces a selection effect which results in an anisotropic distortion even in real space but in an otherwise negligible effect in redshift space. However, our previous study, which relies on a full radiative transfer code run on the Illustris simulations, shows that the anisotropic selection effect was drastically reduced with higher spatial resolution. Adopting the same simulation framework, we further study the impact of RT on the LAE clustering in redshift space. Since we measure LAE's radial position through a spectral peak of a Lyman-$α$ emission, the frequency shift due to RT contaminates the redshift measurement and hence the inferred radial position in redshift space. We demonstrate that this additional RT offset suppresses the LAE clustering along the line of sight, which can be interpreted as a novel Fingers-of-God (FoG) effect. To assess the FoG effect, we develop a theoretical framework with particular emphasis on its connection with the underlying one-point velocity distribution which simultaneously takes into account the RT offset as well as the peculiar velocity that is commonly studied in the context of the Redshift Space Distortion (RSD). Although our findings strongly encourage a more careful RSD modeling in LAE surveys, we also seek a method to mitigate the additional FoG effect due to RT by making use of other information in a Lyman-$α$ spectrum.