Presentation Schedule
Developing Validity Evidence for an Analytic Rubric for English Summary Writing: A Multivariate Generalizability Theory Approach (106168)
Session: On Demand
Room: Virtual Video Presentation
Presentation Type: Virtual Presentation
Although many rubrics have been developed to assess English summary writing, most have focused on intermediate to advanced learners. Recognizing that learners with different proficiency levels produce summaries with diverse characteristics, particularly when working with different source texts, this study examines the validity of a four-category analytic rubric designed for learners across a wide range of proficiency levels. Using multivariate generalizability theory, the study analyzed sources of score variance, rating reliability and stability, relationships among rubric categories, and the effects of the number of raters on score reliability. The data consisted of two English summaries written by 70 Japanese university students based on two different source texts. Six experienced English teachers served as raters and were divided into two groups, each evaluating summaries produced by 35 students using the same analytic rubric. The results showed that the contributions of score variance differed across rubric categories and rater groups. In particular, the Paraphrasing category exhibited relatively large variance attributable to examinees, indicating that it consistently reflected learners’ summarization ability. In contrast, other categories were more influenced by examinee-by-task interaction and residual variance, suggesting greater sensitivity to evaluation conditions. Rating reliability also varied by rater group and generally improved as the number of raters increased, while reliability decreased substantially when only one rater was used. Overall, the findings indicate that the analytic rubric is effective for capturing learner differences in specific categories, while highlighting the importance of rater composition and assessment conditions in its application.
Authors:
Makiko Kato, Tohoku University, Japan
About the Presenter(s)
Dr Makiko Kato is a University Assistant Professor/Lecturer at Tohoku University in Japan
See this presentation on the full schedule – On Demand Schedule
Virtual Presentation
Conference Comments & Feedback
Place a comment using your LinkedIn profileShare this Presentation
Comments
Powered by WP LinkPress