Reliable Confidence Measures and Well-Calibrated Probabilistic Outputs in Classification Algorithms

The Machine Learning research area is widely used in several predictive systems,
where observations from the past can be used to create predictions about
future events. Machine Learning can be applied to any area where classification
or regression is used. Nonetheless, most Machine Learning algorithms
do not provide any measures of valid confidence. Conformal Prediction (CP)
is a framework which uses underlying Machine Learning algorithms, and can
provide valid measures of confidence for predictions. Additionally, the Venn
Prediction (VP) framework, which is an extension to the CP framework, provides
well-calibrated probabilistic outputs. This thesis explores and provides
new methods for valid measures of confidence and probabilistic outputs, based
on the Conformal and Venn Prediction frameworks. We introduce a new Conformal
Predictor based on Genetic Algorithms and compare our approach with
other methods. Additionally, the CP framework is extended for multi-label applications
where predictions can contain more than one possible classifications.
Furthermore, a new Venn Predictor based on Inductive CP is introduced, which
greatly improves the computational eciency of VP. We conduct experiments
on our methods and examine their performance and validity. Finally, we examine
the applications of osteoporosis risk assessment, the diagnosis of childhood
abdominal pain, and the evaluation of the risk of stroke based on ultrasound
images of atherosclerotic carotid plaques. Our experimental results on all our
methods demonstrate the reliability and usefulness of our conndence and
probabilistic outputs.