Retrain or not retrain: Conformal test martingales for change-point detection

Vladimir Vovk; Ivan Petej; Ilia Nouretdinov; Ernst Ahlberg; Lars Carlsson; Alex Gammerman

Retrain or not retrain: Conformal test martingales for change-point detection

Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Ernst Ahlberg, Lars Carlsson, Alex Gammerman

Research output: Chapter in Book/Report/Conference proceeding › Chapter

Abstract

We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.

Original language	English
Title of host publication	Proceedings of Machine Learning Research
Editors	Lars Carlsson, Zhiyuan Luo, Giovanni Cherubin, Khuong Nguyen
Pages	191-210
Number of pages	20
Volume	152
Publication status	Published - 2021

Keywords

conformal test martingales
dataset shift
IID assumption
retraining

Access to Document

https://arxiv.org/abs/2102.10439

Cite this

@inbook{46ff40bdcff742f0bab50d4eba55b150,

title = "Retrain or not retrain: Conformal test martingales for change-point detection",

abstract = "We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.",

keywords = "conformal test martingales, dataset shift, IID assumption, retraining",

author = "Vladimir Vovk and Ivan Petej and Ilia Nouretdinov and Ernst Ahlberg and Lars Carlsson and Alex Gammerman",

year = "2021",

language = "English",

isbn = "ISSN: 2640-3498",

volume = "152",

pages = "191--210",

editor = "Lars Carlsson and Zhiyuan Luo and Giovanni Cherubin and Khuong Nguyen",

booktitle = "Proceedings of Machine Learning Research",

}

TY - CHAP

T1 - Retrain or not retrain

T2 - Conformal test martingales for change-point detection

AU - Vovk, Vladimir

AU - Petej, Ivan

AU - Nouretdinov, Ilia

AU - Ahlberg, Ernst

AU - Carlsson, Lars

AU - Gammerman, Alex

PY - 2021

Y1 - 2021

N2 - We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.

AB - We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.

KW - conformal test martingales

KW - dataset shift

KW - IID assumption

KW - retraining

M3 - Chapter

SN - ISSN: 2640-3498

VL - 152

SP - 191

EP - 210

BT - Proceedings of Machine Learning Research

A2 - Carlsson, Lars

A2 - Luo, Zhiyuan

A2 - Cherubin, Giovanni

A2 - Nguyen, Khuong

ER -