A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data
Business organizations experience cut-throat competition in the e-commerce era, where a smart organization needs to come up with faster innovative ideas to enjoy competitive advantages. A smart user decides from the review information of an online product. Data-driven smart machine learning applicat...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-11-01
|
| Series: | Computers |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-431X/11/11/158 |
| _version_ | 1797468663331160064 |
|---|---|
| author | Kamal Uddin Sarker Mohammed Saqib Raza Hasan Salman Mahmood Saqib Hussain Ali Abbas Aziz Deraman |
| author_facet | Kamal Uddin Sarker Mohammed Saqib Raza Hasan Salman Mahmood Saqib Hussain Ali Abbas Aziz Deraman |
| author_sort | Kamal Uddin Sarker |
| collection | DOAJ |
| description | Business organizations experience cut-throat competition in the e-commerce era, where a smart organization needs to come up with faster innovative ideas to enjoy competitive advantages. A smart user decides from the review information of an online product. Data-driven smart machine learning applications use real data to support immediate decision making. Web scraping technologies support supplying sufficient relevant and up-to-date well-structured data from unstructured data sources like websites. Machine learning applications generate models for in-depth data analysis and decision making. The Internet Movie Database (IMDB) is one of the largest movie databases on the internet. IMDB movie information is applied for statistical analysis, sentiment classification, genre-based clustering, and rating-based clustering with respect to movie release year, budget, etc., for repository dataset. This paper presents a novel clustering model with respect to two different rating systems of IMDB movie data. This work contributes to the three areas: (i) the “grey area” of web scraping to extract data for research purposes; (ii) statistical analysis to correlate required data fields and understanding purposes of implementation machine learning, (iii) k-means clustering is applied for movie critics rank (<i>Metascore</i>) and users’ star rank (<i>Rating</i>). Different python libraries are used for web data scraping, data analysis, data visualization, and k-means clustering application. Only 42.4% of records were accepted from the extracted dataset for research purposes after cleaning. Statistical analysis showed that votes, ratings, <i>Metascore</i> have a linear relationship, while random characteristics are observed for income of the movie. On the other hand, experts’ feedback (<i>Metascore</i>) and customers’ feedback (<i>Rating</i>) are negatively correlated (−0.0384) due to the biasness of additional features like genre, actors, budget, etc. Both rankings have a nonlinear relationship with the income of the movies. Six optimal clusters were selected by elbow technique and the calculated silhouette score is 0.4926 for the proposed k-means clustering model and we found that only one cluster is in the logical relationship of two rankings systems. |
| first_indexed | 2024-03-09T19:10:37Z |
| format | Article |
| id | doaj.art-abc39934407347109f4b232fc5e2d044 |
| institution | Directory Open Access Journal |
| issn | 2073-431X |
| language | English |
| last_indexed | 2024-03-09T19:10:37Z |
| publishDate | 2022-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Computers |
| spelling | doaj.art-abc39934407347109f4b232fc5e2d0442023-11-24T04:14:08ZengMDPI AGComputers2073-431X2022-11-01111115810.3390/computers11110158A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie DataKamal Uddin Sarker0Mohammed Saqib1Raza Hasan2Salman Mahmood3Saqib Hussain4Ali Abbas5Aziz Deraman6Department of Computer Science, American International University Bangladesh, 408/1, Kuratoli, Khilkhet, Dhaka 1229, BangladeshBusiness Administration Department, Jumeira University, Latifa Bint Hamdan Street (West), Exit Number 24, Al Khail Road, Dubai P.O. Box 555532, United Arab EmiratesDepartment of Computing and IT, Global College of Engineering and Technology, Muscat 112, OmanDepartment of Information Technology, School of Science and Engineering, Malaysia University of Science and Technology, Petaling Jaya 47810, Selangor, MalaysiaDepartment of Computing and IT, Global College of Engineering and Technology, Muscat 112, OmanDepartment of Computing, Middle East College, Knowledge Oasis Muscat, P.B. No. 79, Al Rusayl 124, OmanDepartment of Informatics, University Malaysia Terengganu, Kuala Terengganu 21030, Terengganu, MalaysiaBusiness organizations experience cut-throat competition in the e-commerce era, where a smart organization needs to come up with faster innovative ideas to enjoy competitive advantages. A smart user decides from the review information of an online product. Data-driven smart machine learning applications use real data to support immediate decision making. Web scraping technologies support supplying sufficient relevant and up-to-date well-structured data from unstructured data sources like websites. Machine learning applications generate models for in-depth data analysis and decision making. The Internet Movie Database (IMDB) is one of the largest movie databases on the internet. IMDB movie information is applied for statistical analysis, sentiment classification, genre-based clustering, and rating-based clustering with respect to movie release year, budget, etc., for repository dataset. This paper presents a novel clustering model with respect to two different rating systems of IMDB movie data. This work contributes to the three areas: (i) the “grey area” of web scraping to extract data for research purposes; (ii) statistical analysis to correlate required data fields and understanding purposes of implementation machine learning, (iii) k-means clustering is applied for movie critics rank (<i>Metascore</i>) and users’ star rank (<i>Rating</i>). Different python libraries are used for web data scraping, data analysis, data visualization, and k-means clustering application. Only 42.4% of records were accepted from the extracted dataset for research purposes after cleaning. Statistical analysis showed that votes, ratings, <i>Metascore</i> have a linear relationship, while random characteristics are observed for income of the movie. On the other hand, experts’ feedback (<i>Metascore</i>) and customers’ feedback (<i>Rating</i>) are negatively correlated (−0.0384) due to the biasness of additional features like genre, actors, budget, etc. Both rankings have a nonlinear relationship with the income of the movies. Six optimal clusters were selected by elbow technique and the calculated silhouette score is 0.4926 for the proposed k-means clustering model and we found that only one cluster is in the logical relationship of two rankings systems.https://www.mdpi.com/2073-431X/11/11/158movie dataweb scrapingstatistical analysismachine learningk-means clustering |
| spellingShingle | Kamal Uddin Sarker Mohammed Saqib Raza Hasan Salman Mahmood Saqib Hussain Ali Abbas Aziz Deraman A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data Computers movie data web scraping statistical analysis machine learning k-means clustering |
| title | A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data |
| title_full | A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data |
| title_fullStr | A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data |
| title_full_unstemmed | A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data |
| title_short | A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data |
| title_sort | ranking learning model by k means clustering technique for web scraped movie data |
| topic | movie data web scraping statistical analysis machine learning k-means clustering |
| url | https://www.mdpi.com/2073-431X/11/11/158 |
| work_keys_str_mv | AT kamaluddinsarker arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT mohammedsaqib arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT razahasan arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT salmanmahmood arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT saqibhussain arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT aliabbas arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT azizderaman arankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT kamaluddinsarker rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT mohammedsaqib rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT razahasan rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT salmanmahmood rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT saqibhussain rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT aliabbas rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata AT azizderaman rankinglearningmodelbykmeansclusteringtechniqueforwebscrapedmoviedata |