Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i>
<i>Background</i>: Researchers frequently use the responses of individuals in clusters to measure cluster-level constructs. Examples are the use of student evaluations to measure teaching quality, or the use of employee ratings of organizational climate. In earlier research, Stapleton an...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-05-01
|
| Series: | Psych |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2624-8611/3/2/12 |
| _version_ | 1797520978531581952 |
|---|---|
| author | Suzanne Jak Terrence D. Jorgensen Yves Rosseel |
| author_facet | Suzanne Jak Terrence D. Jorgensen Yves Rosseel |
| author_sort | Suzanne Jak |
| collection | DOAJ |
| description | <i>Background</i>: Researchers frequently use the responses of individuals in clusters to measure cluster-level constructs. Examples are the use of student evaluations to measure teaching quality, or the use of employee ratings of organizational climate. In earlier research, Stapleton and Johnson (2019) provided advice for measuring cluster-level constructs based on a simulation study with inadvertently confounded design factors. We extended their simulation study using both M<i>plus</i> and lavaan to reveal how their conclusions were dependent on their study conditions. <i>Methods:</i> We generated data sets from the so-called configural model and the simultaneous shared-and-configural model, both with and without nonzero residual variances at the cluster level. We fitted models to these data sets using different maximum likelihood estimation algorithms. <i>Results:</i> Stapleton and Johnson’s results were highly contingent on their confounded design factors. Convergence rates could be very different across algorithms, depending on whether between-level residual variances were zero in the population or in the fitted model. We discovered a worrying convergence issue with the default settings in M<i>plus</i>, resulting in seemingly converged solutions that are actually not. Rejection rates of the normal-theory test statistic were as expected, while rejection rates of the scaled test statistic were seriously inflated in several conditions. <i>Conclusions:</i> The defaults in M<i>plus</i> carry specific risks that are easily checked but not well advertised. Our results also shine a different light on earlier advice on the use of measurement models for shared factors. |
| first_indexed | 2024-03-10T08:05:12Z |
| format | Article |
| id | doaj.art-a8ab0fbe864a4b62b44416d3995af22d |
| institution | Directory Open Access Journal |
| issn | 2624-8611 |
| language | English |
| last_indexed | 2024-03-10T08:05:12Z |
| publishDate | 2021-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Psych |
| spelling | doaj.art-a8ab0fbe864a4b62b44416d3995af22d2023-11-22T11:07:15ZengMDPI AGPsych2624-86112021-05-013213415210.3390/psych3020012Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i>Suzanne Jak0Terrence D. Jorgensen1Yves Rosseel2Research Institute of Child Development and Education, University of Amsterdam, 1012 WX Amsterdam, The NetherlandsResearch Institute of Child Development and Education, University of Amsterdam, 1012 WX Amsterdam, The NetherlandsDepartment of Data Analysis, Ghent University, B-9000 Ghent, Belgium<i>Background</i>: Researchers frequently use the responses of individuals in clusters to measure cluster-level constructs. Examples are the use of student evaluations to measure teaching quality, or the use of employee ratings of organizational climate. In earlier research, Stapleton and Johnson (2019) provided advice for measuring cluster-level constructs based on a simulation study with inadvertently confounded design factors. We extended their simulation study using both M<i>plus</i> and lavaan to reveal how their conclusions were dependent on their study conditions. <i>Methods:</i> We generated data sets from the so-called configural model and the simultaneous shared-and-configural model, both with and without nonzero residual variances at the cluster level. We fitted models to these data sets using different maximum likelihood estimation algorithms. <i>Results:</i> Stapleton and Johnson’s results were highly contingent on their confounded design factors. Convergence rates could be very different across algorithms, depending on whether between-level residual variances were zero in the population or in the fitted model. We discovered a worrying convergence issue with the default settings in M<i>plus</i>, resulting in seemingly converged solutions that are actually not. Rejection rates of the normal-theory test statistic were as expected, while rejection rates of the scaled test statistic were seriously inflated in several conditions. <i>Conclusions:</i> The defaults in M<i>plus</i> carry specific risks that are easily checked but not well advertised. Our results also shine a different light on earlier advice on the use of measurement models for shared factors.https://www.mdpi.com/2624-8611/3/2/12multilevel SEMcluster-level constructsmaximum likelihood estimation |
| spellingShingle | Suzanne Jak Terrence D. Jorgensen Yves Rosseel Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> Psych multilevel SEM cluster-level constructs maximum likelihood estimation |
| title | Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> |
| title_full | Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> |
| title_fullStr | Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> |
| title_full_unstemmed | Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> |
| title_short | Evaluating Cluster-Level Factor Models with <tt>lavaan</tt> and M<i>plus</i> |
| title_sort | evaluating cluster level factor models with tt lavaan tt and m i plus i |
| topic | multilevel SEM cluster-level constructs maximum likelihood estimation |
| url | https://www.mdpi.com/2624-8611/3/2/12 |
| work_keys_str_mv | AT suzannejak evaluatingclusterlevelfactormodelswithttlavaanttandmiplusi AT terrencedjorgensen evaluatingclusterlevelfactormodelswithttlavaanttandmiplusi AT yvesrosseel evaluatingclusterlevelfactormodelswithttlavaanttandmiplusi |