Quantifying Local Model Validity using Active Learning

Sven Lämmle^1,2, Can Bogoclu³, Robert Vosshall⁴, Anselm Haselhoff⁵, Dirk Roos²

¹ZF Friedrichshafen AG, Germany;²Niederrhein University of Applied Sciences, Germany;³Zalando SE, Germany;⁴auxmoney GmbH, Germany;⁵Ruhr West University of Applied Sciences, Germany Conference on Uncertainty in Artificial Intelligence (UAI), 2024

Paper Code arXiv

Abstract

Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data. We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.

Illustration of a locally valid model: The trained model is marginally valid with 80% probability in certain regions of the input space. Shows the marginal distribution of the true absolute error, with the 80% quantile corresponding to the tolerance level. Depicts the learned error model and 90% confidence interval, along with the predicted local valid set. Samples are sequentially placed to reduce misclassification probability, with most samples near the limit state.

Illustration of a locally valid model: The trained model is marginally valid with 80% probability in certain regions of the input space. Shows the marginal distribution of the true absolute error, with the 80% quantile corresponding to the tolerance level. Depicts the learned error model and 90% confidence interval, along with the predicted local valid set. Samples are sequentially placed to reduce misclassification probability, with most samples near the limit state.

Prediction for the modified Rastrigin function after 20 initial and 70 adaptive observations, with specified parameters. The true limit state is shown by the black line.

Poster

BibTeX

@inproceedings{
laemmle2024quantifying,
title={Quantifying Local Model Validity using Active Learning},
author={Sven L{\"a}mmle and Can Bogoclu and Robert Vosshall and Anselm Haselhoff and Dirk Roos},
booktitle={The 40th Conference on Uncertainty in Artificial Intelligence},
year={2024},
url={https://openreview.net/forum?id=muDcwOKf50}
}