A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics
There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we c...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2021-02-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2021.604522/full |
| _version_ | 1818329491060031488 |
|---|---|
| author | Sharath Koorathota Sharath Koorathota Kaveri Thakoor Linbi Hong Yaoli Mao Patrick Adelman Paul Sajda |
| author_facet | Sharath Koorathota Sharath Koorathota Kaveri Thakoor Linbi Hong Yaoli Mao Patrick Adelman Paul Sajda |
| author_sort | Sharath Koorathota |
| collection | DOAJ |
| description | There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal. |
| first_indexed | 2024-12-13T12:48:54Z |
| format | Article |
| id | doaj.art-74d80979c73740c58e5193cf631460b3 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-12-13T12:48:54Z |
| publishDate | 2021-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-74d80979c73740c58e5193cf631460b32022-12-21T23:45:24ZengFrontiers Media S.A.Frontiers in Psychology1664-10782021-02-011210.3389/fpsyg.2021.604522604522A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil DynamicsSharath Koorathota0Sharath Koorathota1Kaveri Thakoor2Linbi Hong3Yaoli Mao4Patrick Adelman5Paul Sajda6Department of Biomedical Engineering, Columbia University, New York, NY, United StatesFovea Inc., New York, NY, United StatesDepartment of Biomedical Engineering, Columbia University, New York, NY, United StatesDepartment of Biomedical Engineering, Columbia University, New York, NY, United StatesDepartment of Cognitive Science, Columbia University, New York, NY, United StatesFovea Inc., New York, NY, United StatesDepartment of Biomedical Engineering, Columbia University, New York, NY, United StatesThere is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal.https://www.frontiersin.org/articles/10.3389/fpsyg.2021.604522/fullrecurrent neural networkpupil diametereye trackingvideo viewingpupil response |
| spellingShingle | Sharath Koorathota Sharath Koorathota Kaveri Thakoor Linbi Hong Yaoli Mao Patrick Adelman Paul Sajda A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics Frontiers in Psychology recurrent neural network pupil diameter eye tracking video viewing pupil response |
| title | A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics |
| title_full | A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics |
| title_fullStr | A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics |
| title_full_unstemmed | A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics |
| title_short | A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics |
| title_sort | recurrent neural network for attenuating non cognitive components of pupil dynamics |
| topic | recurrent neural network pupil diameter eye tracking video viewing pupil response |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2021.604522/full |
| work_keys_str_mv | AT sharathkoorathota arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT sharathkoorathota arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT kaverithakoor arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT linbihong arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT yaolimao arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT patrickadelman arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT paulsajda arecurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT sharathkoorathota recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT sharathkoorathota recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT kaverithakoor recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT linbihong recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT yaolimao recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT patrickadelman recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics AT paulsajda recurrentneuralnetworkforattenuatingnoncognitivecomponentsofpupildynamics |