| Summary: | <p>In robotics, a classifier is often a core component of the decision-making framework. <em>Precision</em> and <em>recall</em> have been widely adopted as canonical metrics to quantify the performance of a classifier, but for applications involving mission-critical decision making, good performance in relation to these metrics is insufficient. The use of a classification framework which produces scores with inappropriate confidences will ultimately lead to the robot making bad decisions, thereby compromising robot or user safety. In order to select a classifier which will make decisions reflecting the nature of the costs, we should pay careful attention to the ways in which it generates scores. We introduce and motivate the importance of a classifier's <em>introspective</em> capacity: the ability to give an appropriate assessment of confidence with any test case. Classification made confidently must be correct, and mistakes should be made with high uncertainty. A classifier's capacity to do so must remain consistent despite unusual or surprising test cases. We propose that a key ingredient for introspection is a classifier's potential to increase its uncertainty with the distance between a test datum and its training data.</p> <p>We define the ideal introspective behaviour, and derive idealised classifiers which serve to benchmark a number of commonly used classification frameworks in a variety of decision-making tasks. We show that classifiers that offer predictive variance at test-time are more cautious and less over-confident than those which consider a single hypothesis or discriminant. However, in high-cost (or high-risk) decision making, none of the classifiers evaluated in this thesis are sufficiently introspective to prevent all potential catastrophic mistakes. We show that in sequential decision-making, when the mapping from score to class is explicitly stated, a classifier's ability to behave consistently despite non-stationary test data is of primary importance. </p>
|
|---|