StatTestCalculator: A New General Tool for Statistical Analysis in High Energy Physics

Abstract

We present StatTestCalculator (STC), a new open-source statistical analysis tool designed for analysis high energy physics experiments. STC provides both asymptotic calculations and Monte Carlo simulations for computing the exact statistical significance of a discovery or for setting upper limits on signal model parameters. We review the underlying statistical formalism, including profile likelihood ratio test statistics for discovery and exclusion hypotheses, and the asymptotic distributions that allow quick significance estimates. We explain the relevant formulas for the likelihood functions, test statistic distributions, and significance metrics (both with and without incorporating systematic uncertainties). The implementation and capabilities of STC are described, and we validate its performance against the widely-used CMS Combine tool. We find excellent agreement in both the expected discovery significances and upper limit calculations. STC is a flexible framework that can accommodate systematic uncertainties and user-defined statistical models, making it suitable for a broad range of analyses.