Learning Depth with Convolutional Spatial Propagation Network

In this paper, we propose the convolutional spatial propagation network (CSPN) and demonstrate its effectiveness for various depth estimation tasks. CSPN is a simple and efficient linear propagation model, where the propagation is performed with a manner of recurrent convolutional operations, in which the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). Compare to the previous state-of-the-art (SOTA) linear propagation model, i.e., spatial propagation networks (SPN), CSPN is <inline-formula><tex-math notation="LaTeX">$\mathbf 2$</tex-math><alternatives><mml:math><mml:mn mathvariant="bold">2</mml:mn></mml:math><inline-graphic xlink:href="cheng-ieq1-2947374.gif"/></alternatives></inline-formula> to <inline-formula><tex-math notation="LaTeX">$\mathbf 5\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn mathvariant="bold">5</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cheng-ieq2-2947374.gif"/></alternatives></inline-formula> faster in practice. We concatenate CSPN and its variants to SOTA depth estimation networks, which significantly improve the depth accuracy. Specifically, we apply CSPN to two depth estimation problems: depth completion and stereo matching, in which we design modules which adapts the original 2D CSPN to embed sparse depth samples during the propagation, operate with 3D convolution and be synergistic with spatial pyramid pooling. In our experiments, we show that all these modules contribute to the final performance. For the task of depth completion, our method reduce the depth error over 30 percent in the NYU v2 and KITTI datasets. For the task of stereo matching, our method currently ranks <inline-formula><tex-math notation="LaTeX">$\mathbf 1st$</tex-math><alternatives><mml:math><mml:mrow><mml:mn mathvariant="bold">1</mml:mn><mml:mi>s</mml:mi><mml:mi>t</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="cheng-ieq3-2947374.gif"/></alternatives></inline-formula> on both the KITTI Stereo 2012 and 2015 benchmarks.