| Summary: | Purpose: While generalizability (G) theory is widely recognized as a method for estimating the reliability (precision) of measures, its unique approach to partitioning and quantifying variance also yields validity (accuracy) evidence. Yet, G theory’s ability to provide validity evidence is much less understood and established in the literature. The purpose of this paper is to demonstrate G theory’s potential for addressing a wide array of health sciences education validity questions. Methods: Using Kane’s validity framework, this paper explores the use of G theory in the health sciences literature by presenting a number of validity applications. The G studies investigate validity-related measurement questions and demonstrate how G theory contribute to one or more of the four types of Kane’s validity inferences (scoring, generalization, extrapolation, and implication). Results: Each G study is linked to one of Kane’s four types of validity inferences. The studies presented in this paper demonstrate how a G theory analysis of score variance, usually within existing (in vivo) assessment data, simultaneously provides researchers with evidence regarding both reliability and validity and offers a more accurate portrayal of the relationship between the two. Discussion: Because each application of G theory is unique, the examples provided do not represent the entire range of potential applications, but rather demonstrate the methodological flexibility of G theory in addressing complex validity questions. Further advances will require researchers to develop and share additional innovations in G study design and work together to develop a consensus regarding its role in validity research.
|
|---|