Distinguishing high-mass binary neutron stars from binary black holes with second- and third-generation gravitational wave observatories

(Abridged) While the gravitational-wave (GW) signal GW170817 was accompanied by a variety of electromagnetic (EM) counterparts, sufficiently high-mass binary neutron star (BNS) mergers are expected to be unable to power bright EM counterparts. The putative high-mass binary BNS merger GW190425, for which no confirmed EM counterpart has been identified, may be an example of such a system. It is thus important to understand how well we will be able to distinguish high-mass BNSs and low-mass binary black holes (BBHs) solely from their GW signals. To do this, we consider the imprint of the tidal deformability of the neutron stars on the GW signal for systems undergoing prompt black hole formation after merger. We model the BNS signals using hybrid numerical relativity -- tidal effective-one-body waveforms. Specifically, we consider a set of five nonspinning equal-mass BNS signals with masses of 2.7, 3.0, 3.2 Msun and with three different equations of state, as well as the analogous BBH signals. We perform parameter estimation on these signals in three networks: Advanced LIGO-Advanced Virgo and Advanced LIGO-Advanced Virgo-KAGRA with sensitivities similar to O3 and O4, respectively, and a 3G network of two Cosmic Explorers (CEs) and one Einstein Telescope, with a CE sensitivity similar to Stage 2. Our analysis suggests that we cannot distinguish the signals from high-mass BNSs and BBHs at a 90% credible level with the O3-like network even at 40 Mpc. However, we can distinguish all but the most compact BNSs that we consider in our study from BBHs at 40 Mpc at a >= 95% credible level using the O4-like network, and can even distinguish them at a > 99.2% (>= 97%) credible level at 369 (835) Mpc using the 3G network. Additionally, we present a simple method to compute the leading effect of the Earth's rotation on the response of a gravitational wave detector in the frequency domain.