Sensitivity to new physics phenomena in anomaly detection: A study of untunable hyperparameters

The search for physics beyond the Standard Model (BSM) at collider experiments requires model-independent strategies to avoid missing possible discoveries of unexpected signals. Anomaly detection (AD) techniques offer a promising approach by identifying deviations from the Standard Model (SM) and have been extensively studied. The sensitivity of these methods to untunable hyperparameters has not been systematically compared, however. This study addresses it by investigating four semi-supervised AD methods -- Auto-Encoders, Deep Support Vector Data Description, Histogram-based Outlier Score, and Isolation Forest -- trained on simulated SM background events. In this paper, we study the sensitivity of these methods to BSM benchmark signals as a function of these untunable hyperparameters. Such a study is complemented by a proposal of a non-parametric permutation test using signal-agnostic statistics, which can provide a robust statistical assessment.