EgoSurgery‐HTS: A Dataset for Egocentric Hand–Tool Segmentation in Open Surgery Videos

ABSTRACT Egocentric open‐surgery videos capture rich, fine‐grained details essential for accurately modelling surgical procedures and human behaviour in the operating room. A detailed, pixel‐level understanding of hands and surgical tools is crucial for interpreting a surgeon action and intention. We introduce EgoSurgery‐HTS, a new dataset with pixel‐wise annotations and a benchmark suite for segmenting surgical tools, hands, and interacting tools in egocentric open‐surgery videos. Specifically, we provide a labelled dataset for (1) tool instance segmentation of 14 distinct surgical tools, (2) hand instance segmentation, and (3) hand–tool segmentation to label main operating hands and the tools they manipulate. Using EgoSurgery‐HTS, we conduct extensive evaluations of state‐of‐the‐art segmentation methods and demonstrate significant improvements in the accuracy of hand and hand–tool segmentation in egocentric open‐surgery videos compared to existing datasets. The dataset will be released upon acceptance.