Uncertainty in Ranking

Ranks estimated from data are uncertain and this poses a challenge in many applications. The need to measure the uncertainty in sample ranks has been recognized for some time, but the previous literature considering this problem has been concentrated on measuring the uncertainty of individual ranks and not the joint uncertainty. We characterize the relationship between parameter uncertainty and rank uncertainty in terms of linear extensions of a partial order and use this characterization to propose a measure of the joint uncertainty in a sample ranking. We provide efficient algorithms for several questions of interest and also derive valid simultaneous confidence intervals for the individual ranks. We apply our methods to both simulated and real data and make them available through the R package rankUncertainty.