Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be!
The rich data that Massive Open Online Courses (MOOCs) platforms collect on the behavior of millions of users provide a unique opportunity to study human learning and to develop data-driven methods that can address the needs of individual learners. This type of research falls into the emerging field...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Published: |
Springer Science and Business Media LLC
2020
|
| Online Access: | https://hdl.handle.net/1721.1/124002 |
| _version_ | 1811079602160795648 |
|---|---|
| author | Alexandron, Giora Yoo, Lisa Y. Ruiperez Valiente, Jose Antonio Lee, Sunbok Pritchard, David E. |
| author2 | Massachusetts Institute of Technology. Department of Physics |
| author_facet | Massachusetts Institute of Technology. Department of Physics Alexandron, Giora Yoo, Lisa Y. Ruiperez Valiente, Jose Antonio Lee, Sunbok Pritchard, David E. |
| author_sort | Alexandron, Giora |
| collection | MIT |
| description | The rich data that Massive Open Online Courses (MOOCs) platforms collect on the behavior of millions of users provide a unique opportunity to study human learning and to develop data-driven methods that can address the needs of individual learners. This type of research falls into the emerging field of learning analytics. However, learning analytics research tends to ignore the issue of the reliability of results that are based on MOOCs data, which is typically noisy and generated by a largely anonymous crowd of learners. This paper provides evidence that learning analytics in MOOCs can be significantly biased by users who abuse the anonymity and open-nature of MOOCs, for example by setting up multiple accounts, due to their amount and aberrant behavior. We identify these users, denoted fake learners, using dedicated algorithms. The methodology for measuring the bias caused by fake learners’ activity combines the ideas of Replication Research and Sensitivity Analysis. We replicate two highly-cited learning analytics studies with and without fake learners data, and compare the results. While in one study, the results were relatively stable against fake learners, in the other, removing the fake learners’ data significantly changed the results. These findings raise concerns regarding the reliability of learning analytics in MOOCs, and highlight the need to develop more robust, generalizable and verifiable research methods. Keywords: Learning Analytics; MOOCs; Replication research; Sensitivity analysis; Fake learners |
| first_indexed | 2024-09-23T11:17:36Z |
| format | Article |
| id | mit-1721.1/124002 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T11:17:36Z |
| publishDate | 2020 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1240022024-06-25T23:19:23Z Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! Alexandron, Giora Yoo, Lisa Y. Ruiperez Valiente, Jose Antonio Lee, Sunbok Pritchard, David E. Massachusetts Institute of Technology. Department of Physics Massachusetts Institute of Technology. Program in Comparative Media Studies/Writing The rich data that Massive Open Online Courses (MOOCs) platforms collect on the behavior of millions of users provide a unique opportunity to study human learning and to develop data-driven methods that can address the needs of individual learners. This type of research falls into the emerging field of learning analytics. However, learning analytics research tends to ignore the issue of the reliability of results that are based on MOOCs data, which is typically noisy and generated by a largely anonymous crowd of learners. This paper provides evidence that learning analytics in MOOCs can be significantly biased by users who abuse the anonymity and open-nature of MOOCs, for example by setting up multiple accounts, due to their amount and aberrant behavior. We identify these users, denoted fake learners, using dedicated algorithms. The methodology for measuring the bias caused by fake learners’ activity combines the ideas of Replication Research and Sensitivity Analysis. We replicate two highly-cited learning analytics studies with and without fake learners data, and compare the results. While in one study, the results were relatively stable against fake learners, in the other, removing the fake learners’ data significantly changed the results. These findings raise concerns regarding the reliability of learning analytics in MOOCs, and highlight the need to develop more robust, generalizable and verifiable research methods. Keywords: Learning Analytics; MOOCs; Replication research; Sensitivity analysis; Fake learners 2020-03-03T21:49:16Z 2020-03-03T21:49:16Z 2019-07 Article http://purl.org/eprint/type/JournalArticle 1560-4292 1560-4306 https://hdl.handle.net/1721.1/124002 Alexandron, Giora et al. "Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be!" International Journal of Artificial Intelligence in Education 29, 4 (July 2019): 484–506 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 10.1007/s40593-019-00183-1 http://dx.doi.org/10.1007/s40593-019-00183-1 International Journal of Artificial Intelligence in Education Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Springer Science and Business Media LLC Prof Pritchard |
| spellingShingle | Alexandron, Giora Yoo, Lisa Y. Ruiperez Valiente, Jose Antonio Lee, Sunbok Pritchard, David E. Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title | Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title_full | Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title_fullStr | Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title_full_unstemmed | Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title_short | Are MOOC Learning Analytics Results Trustworthy? With Fake Learners, They Might Not Be! |
| title_sort | are mooc learning analytics results trustworthy with fake learners they might not be |
| url | https://hdl.handle.net/1721.1/124002 |
| work_keys_str_mv | AT alexandrongiora aremooclearninganalyticsresultstrustworthywithfakelearnerstheymightnotbe AT yoolisay aremooclearninganalyticsresultstrustworthywithfakelearnerstheymightnotbe AT ruiperezvalientejoseantonio aremooclearninganalyticsresultstrustworthywithfakelearnerstheymightnotbe AT leesunbok aremooclearninganalyticsresultstrustworthywithfakelearnerstheymightnotbe AT pritcharddavide aremooclearninganalyticsresultstrustworthywithfakelearnerstheymightnotbe |