Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size
© 2019 by ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the o...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
ASME International
2021
|
| Online Access: | https://hdl.handle.net/1721.1/136664 |
| _version_ | 1811085572256563200 |
|---|---|
| author | Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D |
| author2 | Massachusetts Institute of Technology. Department of Mechanical Engineering |
| author_facet | Massachusetts Institute of Technology. Department of Mechanical Engineering Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D |
| author_sort | Sudarsanam, Nandan |
| collection | MIT |
| description | © 2019 by ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds, the presence of unequal prior distributions, and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget. |
| first_indexed | 2024-09-23T13:11:43Z |
| format | Article |
| id | mit-1721.1/136664 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:11:43Z |
| publishDate | 2021 |
| publisher | ASME International |
| record_format | dspace |
| spelling | mit-1721.1/1366642023-02-23T20:59:00Z Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D Massachusetts Institute of Technology. Department of Mechanical Engineering © 2019 by ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds, the presence of unequal prior distributions, and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget. 2021-10-27T20:36:31Z 2021-10-27T20:36:31Z 2020 2020-07-09T16:35:10Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/136664 en 10.1115/1.4045603 Journal of Mechanical Design Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf ASME International ASME |
| spellingShingle | Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title | Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_full | Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_fullStr | Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_full_unstemmed | Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_short | Conducting Non-adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_sort | conducting non adaptive experiments in a live setting a bayesian approach to determining optimal sample size |
| url | https://hdl.handle.net/1721.1/136664 |
| work_keys_str_mv | AT sudarsanamnandan conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize AT chandranramya conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize AT freydanield conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize |