Causal Regularization: On the trade-off between in-sample risk and out-of-sample risk guarantees

Invariant prediction uses the prediction stability of causal relationships across different environments to identify causal variables. Conversely, using causal variables gives prediction guarantees even in out-of-sample data settings. In this paper, we investigate the identification of causal-like models from in-sample data that ensure out-of-sample risk guarantees when predicting a target variable from an arbitrary set of covariates. Ordinary least squares minimizes in-sample risk but offers limited out-of-sample guarantees, while causal models optimize out-of-sample guarantees at the expense of in-sample performance. We introduce a form of \textit{causal regularization} to balance these properties. In the population setting, higher regularization yields estimators with greater risk stability, albeit with increased in-sample risk. Empirically, however, there is a further trade-off to consider, as finite in-sample data reduced the ability to correctly identify models with high out-of-sample risk guarantees. We show how in such empirical settings the optimal causal regularizer can be found via cross-validation.