PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic
Trust and confidence in autonomous behavior is required to send autonomous vehicles into operational missions. The authors introduce the Performance Evaluation and Review Framework Of Robotic Missions (PERFORM), a framework to enable a rigorous and replicable autonomy test environment, thereby filli...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-12-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/24/11940 |
| _version_ | 1827674205146054656 |
|---|---|
| author | Allisa J. Dalpe May-Win L. Thein Martin Renken |
| author_facet | Allisa J. Dalpe May-Win L. Thein Martin Renken |
| author_sort | Allisa J. Dalpe |
| collection | DOAJ |
| description | Trust and confidence in autonomous behavior is required to send autonomous vehicles into operational missions. The authors introduce the Performance Evaluation and Review Framework Of Robotic Missions (PERFORM), a framework to enable a rigorous and replicable autonomy test environment, thereby filling the void between that of merely simulating autonomy and that of completing true field missions. A generic architecture for defining the missions under test is proposed and a unique Interval Type-2 Fuzzy Logic approach is used as the foundation for the mathematically rigorous autonomy evaluation framework. The test environment is designed to aid in (1) new technology development (i.e., providing direct comparisons and quantitative evaluations between autonomy algorithms), (2) the validation of the performance of specific autonomous platforms, and (3) the selection of the appropriate robotic platform(s) for a given mission type (e.g., for surveying, surveillance, search and rescue). Three case studies are presented to apply the metric to various test scenarios. Results demonstrate the flexibility of the technique with the ability to tailor tests to the user’s design requirements accounting for different priorities related to acceptable risks and goals of a given mission. |
| first_indexed | 2024-03-10T04:36:19Z |
| format | Article |
| id | doaj.art-0aaedec093df4210a92c03bf0a9134f7 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T04:36:19Z |
| publishDate | 2021-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-0aaedec093df4210a92c03bf0a9134f72023-11-23T03:40:44ZengMDPI AGApplied Sciences2076-34172021-12-0111241194010.3390/app112411940PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy LogicAllisa J. Dalpe0May-Win L. Thein1Martin Renken2School of Marine Science and Engineering, University of New Hampshire, Durham, NH 03824, USASchool of Marine Science and Engineering, University of New Hampshire, Durham, NH 03824, USANaval Undersea Warfare Center—Division Keyport, Keyport, WA 98345, USATrust and confidence in autonomous behavior is required to send autonomous vehicles into operational missions. The authors introduce the Performance Evaluation and Review Framework Of Robotic Missions (PERFORM), a framework to enable a rigorous and replicable autonomy test environment, thereby filling the void between that of merely simulating autonomy and that of completing true field missions. A generic architecture for defining the missions under test is proposed and a unique Interval Type-2 Fuzzy Logic approach is used as the foundation for the mathematically rigorous autonomy evaluation framework. The test environment is designed to aid in (1) new technology development (i.e., providing direct comparisons and quantitative evaluations between autonomy algorithms), (2) the validation of the performance of specific autonomous platforms, and (3) the selection of the appropriate robotic platform(s) for a given mission type (e.g., for surveying, surveillance, search and rescue). Three case studies are presented to apply the metric to various test scenarios. Results demonstrate the flexibility of the technique with the ability to tailor tests to the user’s design requirements accounting for different priorities related to acceptable risks and goals of a given mission.https://www.mdpi.com/2076-3417/11/24/11940autonomous systemsmarine vehiclesevaluation toolsperformance metricsfuzzy logicInterval Type-2 Fuzzy Logic |
| spellingShingle | Allisa J. Dalpe May-Win L. Thein Martin Renken PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic Applied Sciences autonomous systems marine vehicles evaluation tools performance metrics fuzzy logic Interval Type-2 Fuzzy Logic |
| title | PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic |
| title_full | PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic |
| title_fullStr | PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic |
| title_full_unstemmed | PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic |
| title_short | PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic |
| title_sort | perform a metric for evaluating autonomous system performance in marine testbed environments using interval type 2 fuzzy logic |
| topic | autonomous systems marine vehicles evaluation tools performance metrics fuzzy logic Interval Type-2 Fuzzy Logic |
| url | https://www.mdpi.com/2076-3417/11/24/11940 |
| work_keys_str_mv | AT allisajdalpe performametricforevaluatingautonomoussystemperformanceinmarinetestbedenvironmentsusingintervaltype2fuzzylogic AT maywinlthein performametricforevaluatingautonomoussystemperformanceinmarinetestbedenvironmentsusingintervaltype2fuzzylogic AT martinrenken performametricforevaluatingautonomoussystemperformanceinmarinetestbedenvironmentsusingintervaltype2fuzzylogic |