Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments
© 2020 INFORMS Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and th...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute for Operations Research and the Management Sciences (INFORMS)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/135950 |
| _version_ | 1826213486735130624 |
|---|---|
| author | Simester, Duncan Timoshenko, Artem Zoumpoulis, Spyros I |
| author2 | Sloan School of Management |
| author_facet | Sloan School of Management Simester, Duncan Timoshenko, Artem Zoumpoulis, Spyros I |
| author_sort | Simester, Duncan |
| collection | MIT |
| description | © 2020 INFORMS Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies. |
| first_indexed | 2024-09-23T15:49:58Z |
| format | Article |
| id | mit-1721.1/135950 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:49:58Z |
| publishDate | 2021 |
| publisher | Institute for Operations Research and the Management Sciences (INFORMS) |
| record_format | dspace |
| spelling | mit-1721.1/1359502023-01-11T19:40:56Z Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments Simester, Duncan Timoshenko, Artem Zoumpoulis, Spyros I Sloan School of Management © 2020 INFORMS Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies. 2021-10-27T20:30:05Z 2021-10-27T20:30:05Z 2020 2021-04-08T14:53:56Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/135950 en 10.1287/MNSC.2019.3379 Management Science Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute for Operations Research and the Management Sciences (INFORMS) SSRN |
| spellingShingle | Simester, Duncan Timoshenko, Artem Zoumpoulis, Spyros I Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title | Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title_full | Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title_fullStr | Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title_full_unstemmed | Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title_short | Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments |
| title_sort | efficiently evaluating targeting policies improving on champion vs challenger experiments |
| url | https://hdl.handle.net/1721.1/135950 |
| work_keys_str_mv | AT simesterduncan efficientlyevaluatingtargetingpoliciesimprovingonchampionvschallengerexperiments AT timoshenkoartem efficientlyevaluatingtargetingpoliciesimprovingonchampionvschallengerexperiments AT zoumpoulisspyrosi efficientlyevaluatingtargetingpoliciesimprovingonchampionvschallengerexperiments |