Evaluating the Robustness of Learning Analytics Results Against Fake Learners
Massive Open Online Courses (MOOCs) collect large amounts of rich data. A primary objective of Learning Analytics (LA) research is studying these data in order to improve the pedagogy of interactive learning environments. Most studies make the underlying assumption that the data represent truthful...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
HTTC e.V.
2018
|
| Online Access: | http://hdl.handle.net/1721.1/116511 https://orcid.org/0000-0002-2304-6365 https://orcid.org/0000-0001-5697-1496 |
| _version_ | 1811070669231751168 |
|---|---|
| author | Alexandron, Giora Lee, Sunbok Ruiperez Valiente, Jose Antonio Pritchard, David E. |
| author2 | Massachusetts Institute of Technology. Department of Physics |
| author_facet | Massachusetts Institute of Technology. Department of Physics Alexandron, Giora Lee, Sunbok Ruiperez Valiente, Jose Antonio Pritchard, David E. |
| author_sort | Alexandron, Giora |
| collection | MIT |
| description | Massive Open Online Courses (MOOCs) collect large amounts of rich data. A primary objective of Learning Analytics (LA) research is studying these data in order to improve the pedagogy of interactive
learning environments. Most studies make the underlying assumption that the data represent truthful and honest learning activity. However, previous studies showed that MOOCs can have large cohorts of users that
break this assumption and achieve high performance through behaviors such as Cheating Using Multiple Accounts or unauthorized collaboration, and we therefore denote them fake learners. Because of their aberrant
behavior, fake learners can bias the results of Learning Analytics (LA) models. The goal of this study is to evaluate the robustness of LA results when the data contain a considerable number of fake learners. Our
methodology follows the rationale of ‘replication research’. We challenge the results reported in a well-known, and one of the first LA/PedagogicEfficacy MOOC papers, by replicating its results with and without the fake learners (identified using machine learning algorithms). The results show that fake learners exhibit very different behavior compared to true learners. However, even though they are a significant portion of the student
population (∼15%), their effect on the results is not dramatic (does not change trends). We conclude that the LA study that we challenged was robust against fake learners. While these results carry an optimistic
message on the trustworthiness of LA research, they rely on data from one MOOC. We believe that this issue should receive more attention within the LA research community, and can explain some ‘surprising’ research results in MOOCs. Keywords: Learning Analytics, Educational Data Mining, MOOCs, Fake Learners, Reliability, IRT |
| first_indexed | 2024-09-23T08:39:41Z |
| format | Article |
| id | mit-1721.1/116511 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T08:39:41Z |
| publishDate | 2018 |
| publisher | HTTC e.V. |
| record_format | dspace |
| spelling | mit-1721.1/1165112022-09-23T13:41:31Z Evaluating the Robustness of Learning Analytics Results Against Fake Learners Alexandron, Giora Lee, Sunbok Ruiperez Valiente, Jose Antonio Pritchard, David E. Massachusetts Institute of Technology. Department of Physics Ruipérez-Valiente, Jose A. Ruiperez Valiente, Jose Antonio Pritchard, David E Massive Open Online Courses (MOOCs) collect large amounts of rich data. A primary objective of Learning Analytics (LA) research is studying these data in order to improve the pedagogy of interactive learning environments. Most studies make the underlying assumption that the data represent truthful and honest learning activity. However, previous studies showed that MOOCs can have large cohorts of users that break this assumption and achieve high performance through behaviors such as Cheating Using Multiple Accounts or unauthorized collaboration, and we therefore denote them fake learners. Because of their aberrant behavior, fake learners can bias the results of Learning Analytics (LA) models. The goal of this study is to evaluate the robustness of LA results when the data contain a considerable number of fake learners. Our methodology follows the rationale of ‘replication research’. We challenge the results reported in a well-known, and one of the first LA/PedagogicEfficacy MOOC papers, by replicating its results with and without the fake learners (identified using machine learning algorithms). The results show that fake learners exhibit very different behavior compared to true learners. However, even though they are a significant portion of the student population (∼15%), their effect on the results is not dramatic (does not change trends). We conclude that the LA study that we challenged was robust against fake learners. While these results carry an optimistic message on the trustworthiness of LA research, they rely on data from one MOOC. We believe that this issue should receive more attention within the LA research community, and can explain some ‘surprising’ research results in MOOCs. Keywords: Learning Analytics, Educational Data Mining, MOOCs, Fake Learners, Reliability, IRT 2018-06-21T20:27:56Z 2018-06-21T20:27:56Z 2018-09 Article http://purl.org/eprint/type/ConferencePaper http://hdl.handle.net/1721.1/116511 Alexandron, Giora et al. "Evaluating the Robustness of Learning Analytics Results Against Fake Learners." EC-TEL 2018, Thirteenth European Conference on Technology Enhanced Learning, 3-6 September, 2018, Leeds, United Kingdom, HTTC e.V., 2018. https://orcid.org/0000-0002-2304-6365 https://orcid.org/0000-0001-5697-1496 en_US http://www.ec-tel.eu/index.php?id=791 EC-TEL 2018, Thirteenth European Conference on Technology Enhanced Learning Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf HTTC e.V. Ruipérez-Valiente |
| spellingShingle | Alexandron, Giora Lee, Sunbok Ruiperez Valiente, Jose Antonio Pritchard, David E. Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title | Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title_full | Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title_fullStr | Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title_full_unstemmed | Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title_short | Evaluating the Robustness of Learning Analytics Results Against Fake Learners |
| title_sort | evaluating the robustness of learning analytics results against fake learners |
| url | http://hdl.handle.net/1721.1/116511 https://orcid.org/0000-0002-2304-6365 https://orcid.org/0000-0001-5697-1496 |
| work_keys_str_mv | AT alexandrongiora evaluatingtherobustnessoflearninganalyticsresultsagainstfakelearners AT leesunbok evaluatingtherobustnessoflearninganalyticsresultsagainstfakelearners AT ruiperezvalientejoseantonio evaluatingtherobustnessoflearninganalyticsresultsagainstfakelearners AT pritcharddavide evaluatingtherobustnessoflearninganalyticsresultsagainstfakelearners |