Impacts of assumption violations on accuracy in test equating: Insights from a simulation study

Abstrakt

In educational testing, comparing individuals’ test scores often requires testequating methods, especially when different test forms are used for different groups. Traditional approaches use an anchor test, where a common set of items is administered to all groups. When this is not feasible, alternative methods, such as the non-equivalent group design with covariates (NEC design), must be applied. This involves using covariates like age, school grades, or other test scores, which are highly correlated with the primary test scores, to adjust for differences in ability between groups. The NEC design relies on the assumption that the conditional distribution of test scores, given the covariates, is the same across groups. However, this assumption may be violated if the covariates’ relationship with test scores varies between groups. One situation where this problem can occur is if one of the covariates is itself measured by multiple test forms. Our study investigates the impact of such violations on the accuracy of resulting equated scores. We conducted a simulation study with two non-equivalent groups differing in ability, using covariates including school type, educational status (as binary variables), and another test score (continuous variable). The simulations reflected real-world scenarios from the Czech Republic’s national matura exam. The results showed minimal bias in equated scores when the assumption of the same conditional distribution was held, especially with large sample sizes. However, notable bias emerged when the assumption was violated, particularly due to different test difficulties for continuous covariate. To address this bias, we proposed an additional equating step: equating the covariate itself before incorporating it into the primary equating process. This adjustment aimed to make the covariate scores comparable across groups, increasing the odds of satisfying the assumption of the same conditional distribution. Results from our simulation show the suggested repeated covariate-equating approach improves equating score accuracy.