Classifier Monitoring and Updating from Unlabelled Data Event

Event details

Abstract: In a dynamic world, the distributions on which prediction models are based are not stationary but change over time. The question whether, when and how to incorporate effects of such concept drift into a prediction model is highly relevant. A classic approach is to monitor the predictive performance on labelled data. However, such data is often scarce and delayed, and the predictive performance is not necessarily a good indicator of concept drift. Recently, techniques have been proposed that either monitor the data distributions directly, extrapolate continuous change patterns, or even incorporate information from yet unlabelled data. In this talk, I will review those techniques and propose an approach that combines monitoring and the prediction of gradual, continuous changes in the data by applying drift mining techniques to unlabelled data. It aims to assess the type of drift in the data and monitors in particular for unexpected changes. Such changes denote changes that are not predictable by extrapolating change patterns. The approach monitors the distribution of recent, unlabelled data and delayed, labelled data to indicate moments and regions of unexpected change in the spatio-temporal distribution.