On October 22, 2014, Dr. YAN Xun fromPurdueUniversitywas invited to deliver a lecture entitled “A Mixed Approach to Examining Rater Reliability and Rater Effects in Language Testing”, which was chaired by Professor XU Hai and attended by faculties and students of the Center.

The lecture started with a discussion of multiple variables that contribute to rater variability in constructed-response items. Four dimensions are identified in the field, all of which could be estimated by either quantitative or qualitative methods. Quantitative approaches comprise (1) consensus estimates looking at exact agreement among independent raters with such indicators as the Exact Agreement and Cohen’s Kappa, (2) consistency estimates probing into raters’ alignment, usually operationalized as internal correlations between two raters and indicated by Pearson’sror Cronbach’s alpha, and (3) measurement estimates used in conjunction with other facets to dig out more detailed information of individual raters and often calculated by Principle Component Analysis, G-Study and Rasch analysis. On the other hand, qualitative approaches are concerned with the “bottom-up” investigation of raters’ performance making use of their comments on students’ work, followed-up interviews or think-aloud protocols. Then, a case study was presented to exemplify the research process of rater performance on OEPT (Oral English Proficiency Test) for ITAs (International Teaching Assistants), utilizing a mixed-method design (a combination of both qualitative and quantitative data). Finally, the lecture concluded with the recommendation of triangulating different data sources on rater performance to inform rater training.

After the lecture, Dr. YAN Xun answered questions raised by attendants and won the appreciation with his in-depth knowledge and elegance.