A Bias Reduction Method for Ability Estimates in Adaptive Online IRT Testing Systems
Abstract
To evaluate the ability of examinees, the item response theory (IRT) gives us an useful information. The IRT evaluates the examinee’s ability and provides the item characteristics. Recently, we have configured the adaptive online testing using the IRT as one of the CBT (Computer Based Testing) methods, thereby we have been able to obtain the examinee’s ability rating in short testing time. However, when the number of items is too small, we observed biases of estimates for the ability parameter. In this study, we have performed simulation studies under various conditions, and have found the magnitude of such biases. In addition, to circumvent the biases due to the Bayes procedure, we have proposed a simple method to reduce the biases.
References
R. De Ayala, The Theory and Practice of Item Response Theory, ser. Methodology in the social sciences. Guilford Press, 2009.
R. K. Hambleton and H. Swaminathan, Item Response Theory: Principles and Applications, 1985th ed. Boston : Hingham, MA, U.S.A: Springer, Dec. 1984.
R. K. Hambleton, Fundamentals of Item Response Theory. Newbury Park, Calif: Sage Publications, Incorporated, Sep. 1991.
W. J. van der Linden and R. K. Hambleton, Eds., Handbook of Modern Item Response Theory. New York, NY: Springer New York, 1997.
D. J. Weiss, “Improving Measurement Quality and Efficiency with Adaptive Testing,” Applied Psychological Measurement, vol. 6, no. 4, pp. 473–492, 1982.
R. D. Bock and R. J. Mislevy, “Adaptive EAP Estimation of Ability in a Microcomputer Environment,” Applied Psychological Measurement, vol. 6, no. 4, pp. 431–444, 1982.
J. R. McBride, “Some Properties of a Bayesian Adaptive Ability Testing Strategy,” Applied Psychological Measurement, vol. 1, no. 1, pp. 121–140, 1977.
D. J. Weiss and J. R. McBride, “Bias and Information of Bayesian Adaptive Testing,” Applied Psychological Measurement, vol. 8, no. 3, pp. 273–285, Jul. 1984.
R. D. Bock and M. Aitkin, “Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm,” Psychometrika, vol. 46, no. 4, pp. 443–459, Dec. 1981.
F. B. Baker and S.-H. Kim, Item response theory: Parameter estimation techniques. CRC Press, 2004.
T. A. Warm, “Weighted Likelihood Estimation of Ability in Item Response Theory,” Psychometrika, vol. 54, no. 3, pp. 427–450, Sep. 1989.
R. J. Patz and B. W. Junker, “A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models,” Journal of Educational and Behaviral Statistics, vol. 24, no. 2, pp. 146–178, 1999.
R. J. Patz and B. W. Junker, “Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses,” Journal of Educational and Behaviral Statistics, vol. 24, no. 4, pp. 342–366, 1999.
W. J. van der Linden and Wim J. Elements of adaptive testing. New York, NY: Springer, 2010.
H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, and R. J. Mislevy, Computerized Adaptive Testing: A Primer, 2nd ed. Mahwah, N.J: Routledge, May 2000.
