Directionally Locked Heteroepitaxy with a Structurally Modulated van der Waals Material

Precise orientation of symmetry-mismatched epilayers on van der Waals (vdW) substrates via heteroepitaxy has commonly been achieved through surface treatment processes to accommodate weak interlayer registry and bonding strength, thereby limiting the range of material combinations for heterostructure design. In this study, we investigate the influence of lattice instabilities in a structurally modulated vdW TaCo2Te2 substrate on the growth and alignment of a symmetry-mismatched bulk CoxTey epilayer using in situ heating in a transmission electron microscope (TEM). We show that a Peierls-like lattice instability occurs in TaCo2Te2 at a transition temperature of ~523 K, which was corroborated by phonon calculations. Post-heat-treated samples reveal a thermally induced surface diffusion process and the dominant lateral growth of the CoxTey epilayer on the TaCo2Te2 vdW layers, as observed in cross-sectional TEM images. Temperature-dependent selected area electron diffraction (SAED) patterns reveal that the quasi-vdW CoxTey/TaCo2Te2 heterointerface acquires directional locking by aligning larger interlayer lattice mismatch along the lattice instability axis of TaCo2Te2, while preserving a strong lattice matching along the orthogonal direction. This heterostructure exhibits precise interlayer registry with one-dimensional lattice incommensuration along the lattice instability axis, resulting from structural distortion to accommodate lattice-mismatch strain. Moreover, the interfacial reconstruction of TaCo2Te2 back to the distorted phase stabilizes the lattice-locking of the quasi-vdW heterointerface at elevated temperatures. These findings encourage the expansion of material diversity for designing and predicting novel multi-dimensional heterostructures by leveraging lattice instabilities to guide epitaxy.