Conformal Risk Control

We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split <PRE_TAG>conformal prediction</POST_TAG> together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an O(1/n) factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.