Human detection of computer simulation mistakes in engineering experiments
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2010.
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | eng |
| Published: |
Massachusetts Institute of Technology
2011
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/61526 |
| _version_ | 1826210965767585792 |
|---|---|
| author | Savoie, Troy Brendon |
| author2 | Daniel D. Frey and Brenan C. McCarragher. |
| author_facet | Daniel D. Frey and Brenan C. McCarragher. Savoie, Troy Brendon |
| author_sort | Savoie, Troy Brendon |
| collection | MIT |
| description | Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2010. |
| first_indexed | 2024-09-23T14:58:35Z |
| format | Thesis |
| id | mit-1721.1/61526 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T14:58:35Z |
| publishDate | 2011 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/615262019-04-11T06:59:41Z Human detection of computer simulation mistakes in engineering experiments Savoie, Troy Brendon Daniel D. Frey and Brenan C. McCarragher. Massachusetts Institute of Technology. Dept. of Mechanical Engineering. Massachusetts Institute of Technology. Dept. of Mechanical Engineering. Mechanical Engineering. Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2010. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (p. 97-104). This thesis investigates the notion that the more complex the experimental plan, the less likely an engineer is to discover a simulation mistake in a computer-based experiment. The author used an in vitro methodology to conduct an experiment with 54 engineers completing a design task to find the optimal configuration for a device with seven two-level control factors. Participants worked individually using a prescribed design approach dependent upon the randomly assigned experimental condition -- an adaptive one-factor-at-a-time plan for the control group or a resolution III fractional factorial plan for the treatment group -- with a flawed computer simulation of the device. A domain knowledge score was measured by quiz, and success or failure in discovering the flaw was measured by questioning during debriefing. About half (14 of 17) of the participants using the one-factor-at-a-time plan discovered the flaw, while nearly none (1 of 27) using the fractional factorial plan did so. Logistic regression analysis of the dichotomous outcome on treatment condition and domain knowledge score showed that flaw detection ability improved with increased domain knowledge, but that an advantage of two standard deviations in domain knowledge was insufficient to overcome the disadvantage of using the fractional factorial plan. Participant reactions to simulation results were judged by two independent raters for surprise as an indicator of expectation violation. Contingency analysis of the surprise rating results showed that participants using the fractional factorial plan were significantly less likely (risk ratio ~ 0.57) to appear surprised when the anomaly was elicited, but there was no difference in tendency to display surprise otherwise. The observed phenomenon has ramifications beyond simulation mistake detection. Cognitive psychologists have shown that the most effective way to learn a new concept is to observe unexpected behavior, investigate the cause, then integrate the new concept into one's mental model. If using a complex experimental plan hinders an engineer's ability to recognize anomalous data, the engineer risks losing opportunities to develop expertise. Initial screening and sensitivity analysis are recommended as countermeasures when using complex experiments, but more study is needed for verification. by Troy Brendon Savoie. Ph.D. 2011-03-07T14:39:44Z 2011-03-07T14:39:44Z 2010 2010 Thesis http://hdl.handle.net/1721.1/61526 704706161 eng M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 275 p. application/pdf Massachusetts Institute of Technology |
| spellingShingle | Mechanical Engineering. Savoie, Troy Brendon Human detection of computer simulation mistakes in engineering experiments |
| title | Human detection of computer simulation mistakes in engineering experiments |
| title_full | Human detection of computer simulation mistakes in engineering experiments |
| title_fullStr | Human detection of computer simulation mistakes in engineering experiments |
| title_full_unstemmed | Human detection of computer simulation mistakes in engineering experiments |
| title_short | Human detection of computer simulation mistakes in engineering experiments |
| title_sort | human detection of computer simulation mistakes in engineering experiments |
| topic | Mechanical Engineering. |
| url | http://hdl.handle.net/1721.1/61526 |
| work_keys_str_mv | AT savoietroybrendon humandetectionofcomputersimulationmistakesinengineeringexperiments |