Performance of the Mantel-Haenszel statistic and the standardized difference in proportions correct when population ability distributions are incongruent

"A popular method of analyzing test items for differential item functioning (DIF) is to compute a statistic that conditions samples of examinees from different populations on an estimate of ability. This conditioning or matching by ability is intended to produce an appropriate statistic that is sensitive to true differences in item functioning, provided the ability estimate adequately reflects a comparable level of the true ability for these populations. If the observed or number-correct score is used as a conditioning or grouping variable, a problem exists whenever examinees from two different populations are matched on the same level of the observed test score, but actually have quite different levels of the unobserved ability. This occurs whenever the distributions of true abilities for the populations of interest are incongruent or non-overlapping. This situation was investigated in a series of computer simulations using 2,000 White and 2,000 African American examinees and item responses from previous administrations of the American College Testing Mathematics Usage Test. Results indicate that the magnitude of the problem, in terms of being able to detect true DIF with moderate sample sizes when ability distributions are incongruent, may not be that serious for tests that are, on the average, free of DIF. One table and four graphs support the discussion. (Author/SLD)"

"Performance of the Mantel-Haenszel statistic and the standardized difference in proportions correct when population ability distributions are incongruent"@en
"Performance of the Mantel-Haenszel Statistic and the Standardized Difference in Proportions Correct When Population Ability Distributions Are Incongruent"