531 new spectroscopic redshifts from the CDFS and a test on the cosmological relevance of the GOODS-South field

(Abbrev.) This paper prepares a series of papers analysing the Intermediate MAss Galaxy Evolution Sequence (IMAGES) up to z=1. Intermediate mass galaxies (MJ <=-20.3) are selected from the Chandra Deep Field South (CDFS) for which we identify a serious lack of spectroscopically determined redshifts..... We have spectroscopically identified 691 objects including 580 gal., 7 QSOs, and 104 stars. This study provides 531 new redshifts in the CDFS. It confirms the presence of several large scale structures in the CDFS. To test the impact of these structures in the GOODS-South field, we ... compare the evolution of rest-frame U, B, V and K galaxy luminosity densities to that derived from the CFRS. The CDFS field shows a significant excess of luminosity densities in the z=0.5-0.75 range, which increases with the wavelength, reaching up to 0.5 dex at 2.1 um. Stellar mass and specific star formation evolutions might be significantly affected by the presence of the peculiar large scale structures at z= 0.668 and at z= 0.735, that contain a significant excess of evolved, massive galaxies when compared to other fields. This leads to a clear warning to results based on the CDFS/GOODS South fields, especially those related to the evolution of red luminosity densities, i.e. stellar mass density and specific star formation rate. Photometric redshift techniques, when applied to that field, are producing quantities which are apparently less affected by cosmic variance (0.25 dex at 2.1 um), however at the cost of the difficulty in disentangling between evolutionary and cosmic variance effects.