Reliable Confidence Measures and Well-Calibrated Probabilistic Outputs in Classification Algorithms

Antonis Lamprou

Research output: ThesisDoctoral Thesis

210 Downloads (Pure)

Abstract

The Machine Learning research area is widely used in several predictive systems,
where observations from the past can be used to create predictions about
future events. Machine Learning can be applied to any area where classification
or regression is used. Nonetheless, most Machine Learning algorithms
do not provide any measures of valid confidence. Conformal Prediction (CP)
is a framework which uses underlying Machine Learning algorithms, and can
provide valid measures of confidence for predictions. Additionally, the Venn
Prediction (VP) framework, which is an extension to the CP framework, provides
well-calibrated probabilistic outputs. This thesis explores and provides
new methods for valid measures of confidence and probabilistic outputs, based
on the Conformal and Venn Prediction frameworks. We introduce a new Conformal
Predictor based on Genetic Algorithms and compare our approach with
other methods. Additionally, the CP framework is extended for multi-label applications
where predictions can contain more than one possible classifications.
Furthermore, a new Venn Predictor based on Inductive CP is introduced, which
greatly improves the computational eciency of VP. We conduct experiments
on our methods and examine their performance and validity. Finally, we examine
the applications of osteoporosis risk assessment, the diagnosis of childhood
abdominal pain, and the evaluation of the risk of stroke based on ultrasound
images of atherosclerotic carotid plaques. Our experimental results on all our
methods demonstrate the reliability and usefulness of our conndence and
probabilistic outputs.
Original languageEnglish
QualificationPh.D.
Awarding Institution
  • Royal Holloway, University of London
Supervisors/Advisors
  • Papadopoulos, Harris, Supervisor, External person
  • Gammerman, Alex, Supervisor
  • Vovk, Vladimir, Advisor
Thesis sponsors
Award date21 Jul 2016
Publication statusUnpublished - 2016

Keywords

  • confidence measures
  • conformal prediction
  • venn machines
  • machine learning

Cite this