An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours
Abstract Background In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2020-10-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s12874-020-01134-4 |
| _version_ | 1818295369094660096 |
|---|---|
| author | Andrew Copas Sarah Burkill Fred Conrad Mick P. Couper Bob Erens |
| author_facet | Andrew Copas Sarah Burkill Fred Conrad Mick P. Couper Bob Erens |
| author_sort | Andrew Copas |
| collection | DOAJ |
| description | Abstract Background In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys. Methods Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average ‘absolute’ odds ratio (inverse of the odds ratio if less than 1) across 22 pre-specified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance. Results Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased. Conclusions Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates. |
| first_indexed | 2024-12-13T03:46:33Z |
| format | Article |
| id | doaj.art-a32833280f864e3ba46e95ecd7670627 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-12-13T03:46:33Z |
| publishDate | 2020-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-a32833280f864e3ba46e95ecd76706272022-12-22T00:00:49ZengBMCBMC Medical Research Methodology1471-22882020-10-0120111010.1186/s12874-020-01134-4An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behavioursAndrew Copas0Sarah Burkill1Fred Conrad2Mick P. Couper3Bob Erens4Institute for Global Health, University College LondonInstitute for Global Health, University College LondonSurvey Research Center, University of MichiganSurvey Research Center, University of MichiganInstitute for Global Health, University College LondonAbstract Background In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys. Methods Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average ‘absolute’ odds ratio (inverse of the odds ratio if less than 1) across 22 pre-specified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance. Results Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased. Conclusions Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates.http://link.springer.com/article/10.1186/s12874-020-01134-4Internet surveyWeb surveySurvey methodsSampling biasSelection biasSexual behaviour |
| spellingShingle | Andrew Copas Sarah Burkill Fred Conrad Mick P. Couper Bob Erens An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours BMC Medical Research Methodology Internet survey Web survey Survey methods Sampling bias Selection bias Sexual behaviour |
| title | An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours |
| title_full | An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours |
| title_fullStr | An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours |
| title_full_unstemmed | An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours |
| title_short | An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours |
| title_sort | evaluation of whether propensity score adjustment can remove the self selection bias inherent to web panel surveys addressing sensitive health behaviours |
| topic | Internet survey Web survey Survey methods Sampling bias Selection bias Sexual behaviour |
| url | http://link.springer.com/article/10.1186/s12874-020-01134-4 |
| work_keys_str_mv | AT andrewcopas anevaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT sarahburkill anevaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT fredconrad anevaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT mickpcouper anevaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT boberens anevaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT andrewcopas evaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT sarahburkill evaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT fredconrad evaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT mickpcouper evaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours AT boberens evaluationofwhetherpropensityscoreadjustmentcanremovetheselfselectionbiasinherenttowebpanelsurveysaddressingsensitivehealthbehaviours |