On mitigating the analytical limitations of finely stratified experiments
Although attractive from a theoretical perspective, finely stratified experiments such as paired designs suffer from certain analytical limitations that are not present in block-randomized experiments with multiple treated and control individuals in each block. In short, when using a weighted differ...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Article |
| Published: |
Royal Statistical Society
2019
|
| Online Access: | http://hdl.handle.net/1721.1/120515 |
| _version_ | 1811079231175655424 |
|---|---|
| author | Fogarty, Colin B |
| author2 | Sloan School of Management |
| author_facet | Sloan School of Management Fogarty, Colin B |
| author_sort | Fogarty, Colin B |
| collection | MIT |
| description | Although attractive from a theoretical perspective, finely stratified experiments such as paired designs suffer from certain analytical limitations that are not present in block-randomized experiments with multiple treated and control individuals in each block. In short, when using a weighted difference in means to estimate the sample average treatment effect, the traditional variance estimator in a paired experiment is conservative unless the pairwise average treatment effects are constant across pairs; however, in more coarsely stratified experiments, the corresponding variance estimator is unbiased if treatment effects are constant within blocks, even if they vary across blocks. Using insights from classical least squares theory, we present an improved variance estimator that is appropriate in finely stratified experiments. The variance estimator remains conservative in expectation but is asymptotically no more conservative than the classical estimator and can be considerably less conservative. The magnitude of the improvement depends on the extent to which effect heterogeneity can be explained by observed covariates. Aided by this estimator, a new test for the null hypothesis of a constant treatment effect is proposed. These findings extend to some, but not all, superpopulation models, depending on whether the covariates are viewed as fixed across samples. |
| first_indexed | 2024-09-23T11:11:55Z |
| format | Article |
| id | mit-1721.1/120515 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T11:11:55Z |
| publishDate | 2019 |
| publisher | Royal Statistical Society |
| record_format | dspace |
| spelling | mit-1721.1/1205152022-09-27T17:47:17Z On mitigating the analytical limitations of finely stratified experiments Fogarty, Colin B Sloan School of Management Fogarty, Colin B Although attractive from a theoretical perspective, finely stratified experiments such as paired designs suffer from certain analytical limitations that are not present in block-randomized experiments with multiple treated and control individuals in each block. In short, when using a weighted difference in means to estimate the sample average treatment effect, the traditional variance estimator in a paired experiment is conservative unless the pairwise average treatment effects are constant across pairs; however, in more coarsely stratified experiments, the corresponding variance estimator is unbiased if treatment effects are constant within blocks, even if they vary across blocks. Using insights from classical least squares theory, we present an improved variance estimator that is appropriate in finely stratified experiments. The variance estimator remains conservative in expectation but is asymptotically no more conservative than the classical estimator and can be considerably less conservative. The magnitude of the improvement depends on the extent to which effect heterogeneity can be explained by observed covariates. Aided by this estimator, a new test for the null hypothesis of a constant treatment effect is proposed. These findings extend to some, but not all, superpopulation models, depending on whether the covariates are viewed as fixed across samples. 2019-02-21T14:57:29Z 2019-02-21T14:57:29Z 2018-08 2019-02-12T16:59:03Z Article http://purl.org/eprint/type/JournalArticle 1369-7412 http://hdl.handle.net/1721.1/120515 Fogarty, Colin B. “On Mitigating the Analytical Limitations of Finely Stratified Experiments.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80, no. 5 (August 13, 2018): 1035–1056. http://dx.doi.org/10.1111/rssb.12290 Journal of the Royal Statistical Society: Series B (Statistical Methodology) Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Royal Statistical Society arXiv |
| spellingShingle | Fogarty, Colin B On mitigating the analytical limitations of finely stratified experiments |
| title | On mitigating the analytical limitations of finely stratified experiments |
| title_full | On mitigating the analytical limitations of finely stratified experiments |
| title_fullStr | On mitigating the analytical limitations of finely stratified experiments |
| title_full_unstemmed | On mitigating the analytical limitations of finely stratified experiments |
| title_short | On mitigating the analytical limitations of finely stratified experiments |
| title_sort | on mitigating the analytical limitations of finely stratified experiments |
| url | http://hdl.handle.net/1721.1/120515 |
| work_keys_str_mv | AT fogartycolinb onmitigatingtheanalyticallimitationsoffinelystratifiedexperiments |