Information theoretic approach for performance evaluation of multi-class assignment systems
Multi-class assignment is often used to aid in the exploitation of data in the Intelligence, Surveillance, and Reconnaissance (ISR) community. For example, tracking systems collect detections into tracks and recognition systems classify objects into various categories. The reliability of these syste...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
SPIE
2010
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/58584 |
| _version_ | 1826217965443350528 |
|---|---|
| author | Holt, Ryan S. Mastromarino, Peter A. Kao, Edward K. Hurley, Michael B. |
| author2 | Lincoln Laboratory |
| author_facet | Lincoln Laboratory Holt, Ryan S. Mastromarino, Peter A. Kao, Edward K. Hurley, Michael B. |
| author_sort | Holt, Ryan S. |
| collection | MIT |
| description | Multi-class assignment is often used to aid in the exploitation of data in the Intelligence, Surveillance, and Reconnaissance (ISR) community. For example, tracking systems collect detections into tracks and recognition systems classify objects into various categories. The reliability of these systems is highly contingent upon the correctness of the assignments. Conventional methods and metrics for evaluating assignment correctness only convey partial information about the system performance and are usually tied to the specific type of system being evaluated. Recently, information theory has been successfully applied to the tracking problem in order to develop an overall performance evaluation metric. In this paper, the information-theoretic framework is extended to measure the overall performance of any multiclass assignment system, specifically, any system that can be described using a confusion matrix. The performance is evaluated based upon the amount of truth information captured and the amount of false information reported by the system. The information content is quantified through conditional entropy and mutual information computations using numerical estimates of the association probabilities. The end result is analogous to the Receiver Operating Characteristic (ROC) curve used in signal detection theory. This paper compares these information quality metrics to existing metrics and demonstrates how to apply these metrics to evaluate the performance of a recognition system. |
| first_indexed | 2024-09-23T17:11:52Z |
| format | Article |
| id | mit-1721.1/58584 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T17:11:52Z |
| publishDate | 2010 |
| publisher | SPIE |
| record_format | dspace |
| spelling | mit-1721.1/585842022-09-30T00:20:55Z Information theoretic approach for performance evaluation of multi-class assignment systems Holt, Ryan S. Mastromarino, Peter A. Kao, Edward K. Hurley, Michael B. Lincoln Laboratory Hurley, Michael B. Holt, Ryan S. Mastromarino, Peter A. Kao, Edward K. Hurley, Michael B. information theory measures of performance, recognition systems classification Multi-class assignment is often used to aid in the exploitation of data in the Intelligence, Surveillance, and Reconnaissance (ISR) community. For example, tracking systems collect detections into tracks and recognition systems classify objects into various categories. The reliability of these systems is highly contingent upon the correctness of the assignments. Conventional methods and metrics for evaluating assignment correctness only convey partial information about the system performance and are usually tied to the specific type of system being evaluated. Recently, information theory has been successfully applied to the tracking problem in order to develop an overall performance evaluation metric. In this paper, the information-theoretic framework is extended to measure the overall performance of any multiclass assignment system, specifically, any system that can be described using a confusion matrix. The performance is evaluated based upon the amount of truth information captured and the amount of false information reported by the system. The information content is quantified through conditional entropy and mutual information computations using numerical estimates of the association probabilities. The end result is analogous to the Receiver Operating Characteristic (ROC) curve used in signal detection theory. This paper compares these information quality metrics to existing metrics and demonstrates how to apply these metrics to evaluate the performance of a recognition system. United States. Dept. of the Air Force (FA8721-05-C-0002) 2010-09-17T14:30:37Z 2010-09-17T14:30:37Z 2010-04 2010-04 Article http://purl.org/eprint/type/JournalArticle 0277-786X http://hdl.handle.net/1721.1/58584 Ryan S. Holt, Peter A. Mastromarino, Edward K. Kao, and Michael B. Hurley (2010). Information theoretic approach for performance evaluation of multi-class assignment systems. Proc. SPIE 7697: 76970R/1-12. ©2010 COPYRIGHT SPIE--The International Society for Optical Engineering en_US http://dx.doi.org/10.1117/12.851019 Proceedings of SPIE--the International Society for Optical Engineering; v. 7697 Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf SPIE SPIE |
| spellingShingle | information theory measures of performance, recognition systems classification Holt, Ryan S. Mastromarino, Peter A. Kao, Edward K. Hurley, Michael B. Information theoretic approach for performance evaluation of multi-class assignment systems |
| title | Information theoretic approach for performance evaluation of multi-class assignment systems |
| title_full | Information theoretic approach for performance evaluation of multi-class assignment systems |
| title_fullStr | Information theoretic approach for performance evaluation of multi-class assignment systems |
| title_full_unstemmed | Information theoretic approach for performance evaluation of multi-class assignment systems |
| title_short | Information theoretic approach for performance evaluation of multi-class assignment systems |
| title_sort | information theoretic approach for performance evaluation of multi class assignment systems |
| topic | information theory measures of performance, recognition systems classification |
| url | http://hdl.handle.net/1721.1/58584 |
| work_keys_str_mv | AT holtryans informationtheoreticapproachforperformanceevaluationofmulticlassassignmentsystems AT mastromarinopetera informationtheoreticapproachforperformanceevaluationofmulticlassassignmentsystems AT kaoedwardk informationtheoreticapproachforperformanceevaluationofmulticlassassignmentsystems AT hurleymichaelb informationtheoreticapproachforperformanceevaluationofmulticlassassignmentsystems |