The timing mega-study: comparing a range of experiment generators, both lab-based and online
Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of pa...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2020-07-01
|
| Series: | PeerJ |
| Subjects: | |
| Online Access: | https://peerj.com/articles/9414.pdf |
| _version_ | 1797424609518157824 |
|---|---|
| author | David Bridges Alain Pitiot Michael R. MacAskill Jonathan W. Peirce |
| author_facet | David Bridges Alain Pitiot Michael R. MacAskill Jonathan W. Peirce |
| author_sort | David Bridges |
| collection | DOAJ |
| description | Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration. |
| first_indexed | 2024-03-09T08:04:37Z |
| format | Article |
| id | doaj.art-9a93a265c76e4c56aa615e0089b0be4e |
| institution | Directory Open Access Journal |
| issn | 2167-8359 |
| language | English |
| last_indexed | 2024-03-09T08:04:37Z |
| publishDate | 2020-07-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ |
| spelling | doaj.art-9a93a265c76e4c56aa615e0089b0be4e2023-12-03T00:24:57ZengPeerJ Inc.PeerJ2167-83592020-07-018e941410.7717/peerj.9414The timing mega-study: comparing a range of experiment generators, both lab-based and onlineDavid Bridges0Alain Pitiot1Michael R. MacAskill2Jonathan W. Peirce3School of Psychology, University of Nottingham, Nottingham, UKLaboratory of Image and Data Analysis, Ilixa Ltd., London, UKDepartment of Medicine, University of Otago, Christchurch, New ZealandSchool of Psychology, University of Nottingham, Nottingham, UKMany researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.https://peerj.com/articles/9414.pdfTimingStimuliPrecisionExperimentsSoftwareOpen-source |
| spellingShingle | David Bridges Alain Pitiot Michael R. MacAskill Jonathan W. Peirce The timing mega-study: comparing a range of experiment generators, both lab-based and online PeerJ Timing Stimuli Precision Experiments Software Open-source |
| title | The timing mega-study: comparing a range of experiment generators, both lab-based and online |
| title_full | The timing mega-study: comparing a range of experiment generators, both lab-based and online |
| title_fullStr | The timing mega-study: comparing a range of experiment generators, both lab-based and online |
| title_full_unstemmed | The timing mega-study: comparing a range of experiment generators, both lab-based and online |
| title_short | The timing mega-study: comparing a range of experiment generators, both lab-based and online |
| title_sort | timing mega study comparing a range of experiment generators both lab based and online |
| topic | Timing Stimuli Precision Experiments Software Open-source |
| url | https://peerj.com/articles/9414.pdf |
| work_keys_str_mv | AT davidbridges thetimingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT alainpitiot thetimingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT michaelrmacaskill thetimingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT jonathanwpeirce thetimingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT davidbridges timingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT alainpitiot timingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT michaelrmacaskill timingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline AT jonathanwpeirce timingmegastudycomparingarangeofexperimentgeneratorsbothlabbasedandonline |