On Friday I attended an all-day session about item writing titled Moving from Art to Science: an Item-Writing Course for Test Developers, Researchers, and Teachers. About twenty minutes into the session I was faced with my first challenge of the conference. The challenge was initiated when the speaker provided this quote.
Item-writing is an art. It requires an uncommon combination of special abilities. It is mastered only through extensive and critically supervised practice…. Item-writing is essentially creative. Each item as it is being written presents new problems and new opportunities. (Ebel, 1951, p. 185)
I support the writer’s assertion that item writing is “mastered only through extensive and critically supervised practice.” For me, this is a foundational principle of the test development process and of my profession. When talking about new test developers I call this period of supervised practice the apprenticeship period. So, as you could imagine, I found the quote motivating. The speaker’s follow-up quote became the challenge.
A careful review of any testing program will identify poorly worded test items… written by persons with minimal training and inadequate insights into their audience. (Reckase, 2009) (I don’t have the full reference.)
I don’t for a second believe that the test developers I work with are minimally trained. I think that we go out of our way to find people that have above average insights into their audience. Once they are hired as test developers, we begin an intensive process of supervised practice. The quote challenged me to take a critical look at the supervised practice process. A well-trained test developer is not an end state. Like many things in life becoming an excellent test developer is a journey. I believe that my challenge is to ensure that the process is producing excellent test developers. The process of training test developers and the content used to provide that training must also undergo a journey. I am responsible for ensuring that both are moving – and moving in the right direction.
References
Ebel, R.L. (1951). Writing the test item. In E.F. Linquist (Ed.), Educational measurement (1st ed., pp. 185-249). Washington DC: American Council on Education.
Rodriguez, M., Haladyna, T., & Albano, T. (2010). Moving from art to science: an item-writing course for test developers, researchers, and teachers. Proceedings of the Annual Meeting of the National Council on Measurement in Education.