Improving Deformable Image Registration Accuracy through a Hybrid Similarity Metric and CycleGAN Based Auto-Segmentation

Purpose: Deformable image registration (DIR) plays a critical role in adaptive radiation therapy (ART) to accommodate anatomical changes. However, conventional intensity-based DIR methods face challenges when registering images with unequal image intensities. In these cases, DIR accuracy can be improved using a hybrid image similarity metric which matches both image intensities and the location of known structures. This study aims to assess DIR accuracy using a hybrid similarity metric and leveraging CycleGAN-based intensity correction and auto-segmentation and comparing performance across three DIR workflows. Methods: The proposed approach incorporates a hybrid image similarity metric combining a point-to-distance (PD) score and intensity similarity score. Synthetic CT (sCT) images were generated using a 2D CycleGAN model trained on unpaired CT and CBCT images, improving soft-tissue contrast in CBCT images. The performance of the approach was evaluated by comparing three DIR workflows: (1) traditional intensity-based (No PD), (2) auto-segmented contours on sCT (CycleGAN PD), and (3) expert manual contours (Expert PD). A 3D U-Net model was then trained on two datasets comprising 56 3D images and validated on 14 independent cases to segment the prostate, bladder, and rectum. DIR accuracy was assessed using Dice Similarity Coefficient (DSC), 95% Hausdorff Distance (HD), and fiducial separation metrics. Results: The hybrid similarity metric significantly improved DIR accuracy. For the prostate, DSC increased from 0.61 ± 0.18 (No PD) to 0.82 ± 0.13 (CycleGAN PD) and 0.89 ± 0.05 (Expert PD), with corresponding reductions in 95% HD from 11.75 mm to 4.86 mm and 3.27 mm, respectively. Fiducial separation was also reduced from 8.95 mm to 4.07 mm (CycleGAN PD) and 4.11 mm (Expert PD) (p < 0.05). Improvements in alignment were also observed for the bladder and rectum, highlighting the method’s robustness. Conclusion: A hybrid similarity metric that uses CycleGAN-based auto-segmentation presents a promising avenue for advancing DIR accuracy in ART. The study’s findings suggest the potential for substantial enhancements in DIR accuracy by combining AI-based image correction and auto-segmentation with classical DIR.