A critical study on MovieLens dataset for recommender systems
The growth in recommendation systems (RecSys) research has led to the development of many toolkits, which provide users, who may have varying levels of knowledge in the field, with the necessary tools to build, test, evaluate and benchmark different algorithms. The MovieLens datasets have garnered w...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project (FYP) |
| Language: | English |
| Published: |
Nanyang Technological University
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/171942 |
| _version_ | 1811695044806049792 |
|---|---|
| author | Tan, Ernest Yan Heng |
| author2 | Sun Aixin |
| author_facet | Sun Aixin Tan, Ernest Yan Heng |
| author_sort | Tan, Ernest Yan Heng |
| collection | NTU |
| description | The growth in recommendation systems (RecSys) research has led to the development of many toolkits, which provide users, who may have varying levels of knowledge in the field, with the necessary tools to build, test, evaluate and benchmark different algorithms. The MovieLens datasets have garnered widespread popularity as a benchmark dataset for RecSys research, but exploratory analysis has shown that the datasets elicit certain issues such as popularity bias and data sparsity as a result. Therefore, evaluation results of baseline algorithms trained on this dataset may pick up these inherent signals present in the data, and therefore should not be generalised across other recommendation scenarios. A comprehensive and consistent experiment involving 3 Python-based Top-N recommendation toolkits: LensKit, RecPack, and daisyRec have shown that toolkits are often built with different purposes or to solve specific issues, which leads to inconsistency in implementation methodology and hence evaluation results. This can be attributed to several main factors: (1) unclear or inconsistent definition of concepts such as evaluation metrics and (2) differences in default preprocessing and splitting strategies being the most significant. The experiments also highlight the disadvantages of using a global time-aware split on the MovieLens dataset, such as eliminating unseen users which are present in the test set but not in the train set. Additionally, analysis showed that having a low absolute number of train interactions, e.g., less than 15, is detrimental to the performance of a model than having a low train to test interaction ratio, with the evaluation metrics showing relatively poorer performance on 2 out of 3 of the toolkits discussed. Lastly, this study proposes some possible improvements to the toolkits based on the issues highlighted, such as clearly defined default dataset preprocessing, fully customisable hyperparameters, and frameworks which allow for quick development of algorithms and metrics, with a possible future work of producing an actively managed, open source toolkit which can solve the problems surfaced during this study. |
| first_indexed | 2024-10-01T07:17:13Z |
| format | Final Year Project (FYP) |
| id | ntu-10356/171942 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:17:13Z |
| publishDate | 2023 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1719422023-11-17T15:38:06Z A critical study on MovieLens dataset for recommender systems Tan, Ernest Yan Heng Sun Aixin School of Computer Science and Engineering AXSun@ntu.edu.sg Engineering::Computer science and engineering::Information systems::Information storage and retrieval The growth in recommendation systems (RecSys) research has led to the development of many toolkits, which provide users, who may have varying levels of knowledge in the field, with the necessary tools to build, test, evaluate and benchmark different algorithms. The MovieLens datasets have garnered widespread popularity as a benchmark dataset for RecSys research, but exploratory analysis has shown that the datasets elicit certain issues such as popularity bias and data sparsity as a result. Therefore, evaluation results of baseline algorithms trained on this dataset may pick up these inherent signals present in the data, and therefore should not be generalised across other recommendation scenarios. A comprehensive and consistent experiment involving 3 Python-based Top-N recommendation toolkits: LensKit, RecPack, and daisyRec have shown that toolkits are often built with different purposes or to solve specific issues, which leads to inconsistency in implementation methodology and hence evaluation results. This can be attributed to several main factors: (1) unclear or inconsistent definition of concepts such as evaluation metrics and (2) differences in default preprocessing and splitting strategies being the most significant. The experiments also highlight the disadvantages of using a global time-aware split on the MovieLens dataset, such as eliminating unseen users which are present in the test set but not in the train set. Additionally, analysis showed that having a low absolute number of train interactions, e.g., less than 15, is detrimental to the performance of a model than having a low train to test interaction ratio, with the evaluation metrics showing relatively poorer performance on 2 out of 3 of the toolkits discussed. Lastly, this study proposes some possible improvements to the toolkits based on the issues highlighted, such as clearly defined default dataset preprocessing, fully customisable hyperparameters, and frameworks which allow for quick development of algorithms and metrics, with a possible future work of producing an actively managed, open source toolkit which can solve the problems surfaced during this study. Bachelor of Engineering (Computer Science) 2023-11-17T03:14:06Z 2023-11-17T03:14:06Z 2023 Final Year Project (FYP) Tan, E. Y. H. (2023). A critical study on MovieLens dataset for recommender systems. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/171942 https://hdl.handle.net/10356/171942 en application/pdf Nanyang Technological University |
| spellingShingle | Engineering::Computer science and engineering::Information systems::Information storage and retrieval Tan, Ernest Yan Heng A critical study on MovieLens dataset for recommender systems |
| title | A critical study on MovieLens dataset for recommender systems |
| title_full | A critical study on MovieLens dataset for recommender systems |
| title_fullStr | A critical study on MovieLens dataset for recommender systems |
| title_full_unstemmed | A critical study on MovieLens dataset for recommender systems |
| title_short | A critical study on MovieLens dataset for recommender systems |
| title_sort | critical study on movielens dataset for recommender systems |
| topic | Engineering::Computer science and engineering::Information systems::Information storage and retrieval |
| url | https://hdl.handle.net/10356/171942 |
| work_keys_str_mv | AT tanernestyanheng acriticalstudyonmovielensdatasetforrecommendersystems AT tanernestyanheng criticalstudyonmovielensdatasetforrecommendersystems |