Abstract
A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since the algorithms have been trained for a certain data distribution and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic and many other applications. It has been a subject of intensive research, with many publications in the statistical literature. Among well-known methods, there are CumulativeSum (CUSUM) and Shiryaev-Roberts procedures for the detection of changes.
However, many of the methods would require complete or partial knowledge of the dis- tribution of observed variables before and after the distribution has changed. Recent work in Conformal Testing and the introduction of Conformal Test Martingales (CTM) allows us to avoid this limitation and obtained valid results without information about used distri- butions Vovk et al. (2022). This is done in online mode with the assumption that data are exchangeable and the corresponding martingale accumulates evidence against this assump- tion. This paper considers an approach to the problem using the Venn-ABERS testing. It
allows us to find deviations in the distributions when IID was violated.The Venn-ABERS approach satisfies the property of validity, makes calibrated probabilistic predictions and would allow reducing the number of false alarms. In application to the change-point pre- diction we also use e-values instead of p-values which makes some computational savings Using e-values allows us to avoid making an additional step from the non-conformity scores to the p-values. This makes it more convenient to combine with Venn-ABERS scores. As it has been pointed out in the above reference, a large e-value can be interpreted as evi- dence against the IID (or exchangeability) assumption: the evidence is strong when e-value exceeds 10 and decisive when it exceeds 100.
However, many of the methods would require complete or partial knowledge of the dis- tribution of observed variables before and after the distribution has changed. Recent work in Conformal Testing and the introduction of Conformal Test Martingales (CTM) allows us to avoid this limitation and obtained valid results without information about used distri- butions Vovk et al. (2022). This is done in online mode with the assumption that data are exchangeable and the corresponding martingale accumulates evidence against this assump- tion. This paper considers an approach to the problem using the Venn-ABERS testing. It
allows us to find deviations in the distributions when IID was violated.The Venn-ABERS approach satisfies the property of validity, makes calibrated probabilistic predictions and would allow reducing the number of false alarms. In application to the change-point pre- diction we also use e-values instead of p-values which makes some computational savings Using e-values allows us to avoid making an additional step from the non-conformity scores to the p-values. This makes it more convenient to combine with Venn-ABERS scores. As it has been pointed out in the above reference, a large e-value can be interpreted as evi- dence against the IID (or exchangeability) assumption: the evidence is strong when e-value exceeds 10 and decisive when it exceeds 100.
| Original language | English |
|---|---|
| Media of output | Poster |
| Publisher | Proceedings of Machine Learning Research: COPA 2023 12th Symposium on Conformal and Probabilistic Prediction with Applications |
| Number of pages | 2 |
| Volume | 204 |
| Publication status | Published - Aug 2023 |