Scheduling on a budget: Avoiding stale recommendations with timely updates
Recommendation systems usually create static models from historical data. Due to concept drift and changes in the environment, such models are doomed to become stale, which causes their performance to degrade. In live production environments, models are therefore typically retrained at fixed time-in...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-03-01
|
| Series: | Machine Learning with Applications |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666827023000087 |
| _version_ | 1797903557740986368 |
|---|---|
| author | Robin Verachtert Olivier Jeunen Bart Goethals |
| author_facet | Robin Verachtert Olivier Jeunen Bart Goethals |
| author_sort | Robin Verachtert |
| collection | DOAJ |
| description | Recommendation systems usually create static models from historical data. Due to concept drift and changes in the environment, such models are doomed to become stale, which causes their performance to degrade. In live production environments, models are therefore typically retrained at fixed time-intervals. Of course, every retraining comes at a significant computational cost, making very frequent model updates unrealistic in practice. In some cases, the cost is worth it, but in other cases an update could be redundant and the cost an unnecessary loss. The research question then consists of finding an acceptable update schedule for your recommendation system, given a limited budget. This work provides a pragmatic analysis of model staleness for a variety of collaborative filtering algorithms in news and retail domains, where concept drift is a known impediment. We highlight that the rate at which models become stale is highly dependent on the environment they perform in and that this property can be derived from data. These findings are corroborated by empirical observations from four large-scale online experiments. Instead of retraining at regular intervals, we propose an adaptive scheduling method that aims to maximise the accuracy of the recommendations within a fixed resource budget. Offline experiments show that our proposed approach improves recommendation performance while keeping the cost constant. Our findings can guide practitioners to spend their available resources more efficiently. |
| first_indexed | 2024-04-10T09:34:51Z |
| format | Article |
| id | doaj.art-f4a91fb62ea248d2af830590687a9a90 |
| institution | Directory Open Access Journal |
| issn | 2666-8270 |
| language | English |
| last_indexed | 2024-04-10T09:34:51Z |
| publishDate | 2023-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Machine Learning with Applications |
| spelling | doaj.art-f4a91fb62ea248d2af830590687a9a902023-02-18T04:17:43ZengElsevierMachine Learning with Applications2666-82702023-03-0111100455Scheduling on a budget: Avoiding stale recommendations with timely updatesRobin Verachtert0Olivier Jeunen1Bart Goethals2Froomle NV, Posthofbrug 6-8, 2600 Antwerpen, Belgium; University of Antwerp, Prinsstraat 13, 2000 Antwerpen, Belgium; Corresponding author at: Froomle NV, Posthofbrug 6-8, 2600 Antwerpen, Belgium.University of Antwerp, Prinsstraat 13, 2000 Antwerpen, BelgiumFroomle NV, Posthofbrug 6-8, 2600 Antwerpen, Belgium; University of Antwerp, Prinsstraat 13, 2000 Antwerpen, Belgium; Monash University, Victoria 3800, Melbourne, AustraliaRecommendation systems usually create static models from historical data. Due to concept drift and changes in the environment, such models are doomed to become stale, which causes their performance to degrade. In live production environments, models are therefore typically retrained at fixed time-intervals. Of course, every retraining comes at a significant computational cost, making very frequent model updates unrealistic in practice. In some cases, the cost is worth it, but in other cases an update could be redundant and the cost an unnecessary loss. The research question then consists of finding an acceptable update schedule for your recommendation system, given a limited budget. This work provides a pragmatic analysis of model staleness for a variety of collaborative filtering algorithms in news and retail domains, where concept drift is a known impediment. We highlight that the rate at which models become stale is highly dependent on the environment they perform in and that this property can be derived from data. These findings are corroborated by empirical observations from four large-scale online experiments. Instead of retraining at regular intervals, we propose an adaptive scheduling method that aims to maximise the accuracy of the recommendations within a fixed resource budget. Offline experiments show that our proposed approach improves recommendation performance while keeping the cost constant. Our findings can guide practitioners to spend their available resources more efficiently.http://www.sciencedirect.com/science/article/pii/S2666827023000087Recommender systemsSchedulingOnline trials |
| spellingShingle | Robin Verachtert Olivier Jeunen Bart Goethals Scheduling on a budget: Avoiding stale recommendations with timely updates Machine Learning with Applications Recommender systems Scheduling Online trials |
| title | Scheduling on a budget: Avoiding stale recommendations with timely updates |
| title_full | Scheduling on a budget: Avoiding stale recommendations with timely updates |
| title_fullStr | Scheduling on a budget: Avoiding stale recommendations with timely updates |
| title_full_unstemmed | Scheduling on a budget: Avoiding stale recommendations with timely updates |
| title_short | Scheduling on a budget: Avoiding stale recommendations with timely updates |
| title_sort | scheduling on a budget avoiding stale recommendations with timely updates |
| topic | Recommender systems Scheduling Online trials |
| url | http://www.sciencedirect.com/science/article/pii/S2666827023000087 |
| work_keys_str_mv | AT robinverachtert schedulingonabudgetavoidingstalerecommendationswithtimelyupdates AT olivierjeunen schedulingonabudgetavoidingstalerecommendationswithtimelyupdates AT bartgoethals schedulingonabudgetavoidingstalerecommendationswithtimelyupdates |