Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size
Copyright © 2019 ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results o...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
ASME International
2021
|
| Online Access: | https://hdl.handle.net/1721.1/138011 |
| _version_ | 1811094123997822976 |
|---|---|
| author | Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D. |
| author2 | Massachusetts Institute of Technology. Department of Mechanical Engineering |
| author_facet | Massachusetts Institute of Technology. Department of Mechanical Engineering Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D. |
| author_sort | Sudarsanam, Nandan |
| collection | MIT |
| description | Copyright © 2019 ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget. |
| first_indexed | 2024-09-23T15:55:27Z |
| format | Article |
| id | mit-1721.1/138011 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:55:27Z |
| publishDate | 2021 |
| publisher | ASME International |
| record_format | dspace |
| spelling | mit-1721.1/1380112022-10-02T05:06:40Z Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D. Massachusetts Institute of Technology. Department of Mechanical Engineering Copyright © 2019 ASME. This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget. 2021-11-09T18:16:34Z 2021-11-09T18:16:34Z 2019-08 2020-07-09T16:28:14Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/138011 Sudarsanam, Nandan, Chandran, Ramya and Frey, Daniel D. 2019. "Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size." Proceedings of the ASME Design Engineering Technical Conference, 7. en 10.1115/detc2019-98335 Proceedings of the ASME Design Engineering Technical Conference Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf ASME International ASME |
| spellingShingle | Sudarsanam, Nandan Chandran, Ramya Frey, Daniel D. Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title | Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_full | Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_fullStr | Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_full_unstemmed | Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_short | Conducting Non-Adaptive Experiments in a Live Setting: A Bayesian Approach to Determining Optimal Sample Size |
| title_sort | conducting non adaptive experiments in a live setting a bayesian approach to determining optimal sample size |
| url | https://hdl.handle.net/1721.1/138011 |
| work_keys_str_mv | AT sudarsanamnandan conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize AT chandranramya conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize AT freydanield conductingnonadaptiveexperimentsinalivesettingabayesianapproachtodeterminingoptimalsamplesize |