Statistics and Its Interface

Volume 13 (2020)

Number 3

Computerized Adaptive Test Using Raw Responses for Item Selection: Theoretical Results and Applications for the Up-and-Down Method

Pages: 317 – 333

DOI: https://dx.doi.org/10.4310/SII.2020.v13.n3.a3

Authors

Cheng-Der Fuh (Fanhai International School of Finance, Fudan University, Shanghai, China)

Edward Haksing Ip (Department of Biostatistics and Data Science, Wake Forest School of Medicine, Winston-Salem, North Carolina, U.S.A.)

Shyh-Huei Chen (Department of Biostatistics and Data Science, Wake Forest School of Medicine, Winston-Salem, North Carolina, U.S.A.)

Abstract

Modern computerized adaptive testing (CAT) is finding applications that contain more intensive assessments, collected over nontraditional devices such as tablets and smartphones. In this paper, we introduce an CAT algorithm that uses raw responses to adaptively select items and does not require updating the ability estimate at every administration of an item. The proposed algorithm is especially useful in adaptive assessment situations in which updating ability estimate at each administration is either not feasible or too costly to implement. Specifically, an $a$-stratified multistage up-and-down method is proposed as an approximation to the commonly used recursive maximum likelihood estimate (R-MLE). Using Markov chain tools, we derive theoretical results for the statistical properties of the up-and-down method. We also report empirical studies for the performance of the proposed method. Both simulation experiments and real data analysis are included. Limitations of the method such as reduced statistical efficiency are also discussed. Overall, despite the limitations, our results show that the up-and-down method is a promising alternative to the classical R-MLE and well-suited for some CAT applications such as ecological momentary assessments.

Keywords

Up-and-down method, computerized adaptive tests, Markov chains, recursive maximum likelihood estimate, ACT, PROMIS.

Full Text (PDF format)

Received 31 October 2018

Accepted 16 January 2020

Published 22 April 2020