A numerical study of 21-cm signal suppression and noise increase in direction-dependent calibration of LOFAR data

We investigate systematic effects in direction dependent gain calibration in the context of the Low-Frequency Array (LOFAR) 21-cm Epoch of Reionization (EoR) experiment. The LOFAR EoR Key Science Project aims to detect the 21-cm signal of neutral hydrogen on interferometric baselines of $50-250 λ$. We show that suppression of faint signals can effectively be avoided by calibrating these short baselines using only the longer baselines. However, this approach causes an excess variance on the short baselines due to small gain errors induced by overfitting during calibration. We apply a regularised expectation-maximisation algorithm with consensus optimisation (sagecal-co) to real data with simulated signals to show that overfitting can be largely mitigated by penalising spectrally non-smooth gain solutions during calibration. This reduces the excess power with about a factor 4 in the simulations. Our results agree with earlier theoretical analysis of this bias-variance trade off and support the gain-calibration approach to the LOFAR 21-cm signal data.