A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys
Abstract The value of cross-country comparisons is at the heart of large-scale international surveys. Yet the validity of such comparisons is often challenged, particularly in the case of latent traits whose estimates are based on self-reported answers to a small number of questionnaire items. Many...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PsychOpen GOLD/ Leibniz Institute for Psychology
2019-08-01
|
| Series: | Measurement Instruments for the Social Sciences |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s42409-019-0010-z |
| _version_ | 1827853489854742528 |
|---|---|
| author | Francesco Avvisati Noémie Le Donné Marco Paccagnella |
| author_facet | Francesco Avvisati Noémie Le Donné Marco Paccagnella |
| author_sort | Francesco Avvisati |
| collection | DOAJ |
| description | Abstract The value of cross-country comparisons is at the heart of large-scale international surveys. Yet the validity of such comparisons is often challenged, particularly in the case of latent traits whose estimates are based on self-reported answers to a small number of questionnaire items. Many believe self-reports to be unreliable and not comparable, and indeed, formal statistical procedures very often reject the assumption that the questions are understood and answered in the same way in different countries (measurement invariance). A methodological conference on the comparability of questionnaire scales was hosted by the OECD on 8 and 9 November 2018. This meeting report summarises the discussions held at the conference about measurement invariance testing and instrument design. The report first provides a brief introduction to the measurement models and the accompanying invariance analyses typically used in the industry of large-scale international surveys and points to the main limitations of these current standard approaches. It then presents classical and novel ways to deal with imperfect comparability of measurements when scaling and reporting on continuous traits and on categorical latent variables. It finally discusses the extent to which item design can improve the cross-country comparability of the measured constructs (e.g. by adopting innovative item formats such as anchoring vignettes and situational judgement test items). It concludes with some general considerations for survey design and reporting on invariance analyses and survey results. |
| first_indexed | 2024-03-12T11:06:26Z |
| format | Article |
| id | doaj.art-57b6534aed2548758ffcabb76d2fe2c7 |
| institution | Directory Open Access Journal |
| issn | 2523-8930 |
| language | English |
| last_indexed | 2024-03-12T11:06:26Z |
| publishDate | 2019-08-01 |
| publisher | PsychOpen GOLD/ Leibniz Institute for Psychology |
| record_format | Article |
| series | Measurement Instruments for the Social Sciences |
| spelling | doaj.art-57b6534aed2548758ffcabb76d2fe2c72023-09-02T03:47:56ZengPsychOpen GOLD/ Leibniz Institute for PsychologyMeasurement Instruments for the Social Sciences2523-89302019-08-011111010.1186/s42409-019-0010-zA meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveysFrancesco Avvisati0Noémie Le Donné1Marco Paccagnella2OECD, Directorate for Education and Skills 2OECD, Directorate for Education and Skills 2OECD, Directorate for Education and Skills 2Abstract The value of cross-country comparisons is at the heart of large-scale international surveys. Yet the validity of such comparisons is often challenged, particularly in the case of latent traits whose estimates are based on self-reported answers to a small number of questionnaire items. Many believe self-reports to be unreliable and not comparable, and indeed, formal statistical procedures very often reject the assumption that the questions are understood and answered in the same way in different countries (measurement invariance). A methodological conference on the comparability of questionnaire scales was hosted by the OECD on 8 and 9 November 2018. This meeting report summarises the discussions held at the conference about measurement invariance testing and instrument design. The report first provides a brief introduction to the measurement models and the accompanying invariance analyses typically used in the industry of large-scale international surveys and points to the main limitations of these current standard approaches. It then presents classical and novel ways to deal with imperfect comparability of measurements when scaling and reporting on continuous traits and on categorical latent variables. It finally discusses the extent to which item design can improve the cross-country comparability of the measured constructs (e.g. by adopting innovative item formats such as anchoring vignettes and situational judgement test items). It concludes with some general considerations for survey design and reporting on invariance analyses and survey results.http://link.springer.com/article/10.1186/s42409-019-0010-zMeasurement invarianceInternational large-scale assessmentsIRTFactor analysisLatent class analysisSituational judgement test |
| spellingShingle | Francesco Avvisati Noémie Le Donné Marco Paccagnella A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys Measurement Instruments for the Social Sciences Measurement invariance International large-scale assessments IRT Factor analysis Latent class analysis Situational judgement test |
| title | A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys |
| title_full | A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys |
| title_fullStr | A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys |
| title_full_unstemmed | A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys |
| title_short | A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys |
| title_sort | meeting report cross cultural comparability of questionnaire measures in large scale international surveys |
| topic | Measurement invariance International large-scale assessments IRT Factor analysis Latent class analysis Situational judgement test |
| url | http://link.springer.com/article/10.1186/s42409-019-0010-z |
| work_keys_str_mv | AT francescoavvisati ameetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys AT noemieledonne ameetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys AT marcopaccagnella ameetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys AT francescoavvisati meetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys AT noemieledonne meetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys AT marcopaccagnella meetingreportcrossculturalcomparabilityofquestionnairemeasuresinlargescaleinternationalsurveys |