A Distribution-Free Test of Covariate Shift Using Conformal Prediction

Covariate shift is a common and important assumption in transfer learning and domain adaptation to treat the distributional difference between the training and testing data. We propose a nonparametric test of covariate shift using the conformal prediction framework. The construction of our test statistic combines recent developments in conformal prediction with a novel choice of conformity score, resulting in a valid and powerful test statistic under very general settings. To our knowledge, this is the first successful attempt of using conformal prediction for testing statistical hypotheses. Our method is suitable for modern machine learning scenarios where the data has high dimensionality and large sample sizes, and can be effectively combined with existing classification algorithms to find good conformity score functions. The performance of the proposed method is demonstrated in synthetic and real data examples.