A Rasch approach to validation of CEFR-aligned reading tests for testing undergraduate reading comprehension

Abstract

In the ESL context, reading is an important skill necessary for academic success. Similarly, reading tests commonly are conducted in order to find out the students’ ability in comprehending texts so that appropriate teaching and learning instructions are provided to enhance the skill. Applying the latest developments in testing reading and test validation, this study focused on three important objectives. The first was to produce valid and reliable instruments to measure the academic reading

comprehension ability of university students in Sri Lanka by adapting the CEFR- aligned tests. The second was to examine the reading ability of students of the four

faculties at SEUSL, using these validated instruments. The third objective was to investigate the students’ achievement level in the cognitive processes of reading based on Khalifa and Weir’s (2009) model of reading. To achieve these three objectives, 13 texts were adapted along with their (127) items from the CEFR-aligned LRN materials, and four testlets were produced. Eight cognitive processes of reading, namely Word Recognition (WR), Lexical Access (LA), Syntactic Parsing (SP), Establishing Prepositional Meaning (EPM), Inferencing (I), Building a Mental Model (BMM), Creating Text Level Structure (CTLS), and Creating Inter-Textual Representation (CITR), which are arranged hierarchically, were measured. A single test had 40 selected-response objective items including eleven common items, which had been used as anchoring items to horizontally equate four tests. The concurrent analysis of the Rasch measurement model was used to examine the psychometric properties of the tests. The findings revealed the validity and reliability of the tests and the strength of using the Rasch model for test equating. The findings also discovered that, while there was inconsistency in the hierarchical order of the cognitive processes of reading, there was consistency among the LOT (except for EPM) and the HOT processes, and the items within the same process did not have the same difficulty level, which indicates that certain cognitive processes can be used across different difficulty levels. The results also showed that 843 students, 93.5% out of 902, scored the CEFR B1 and B2 levels, which were identified as the minimum requirement for academic success in the ESL context. In addition, students’ reading performance was measured according to their degree programmes with English as a-medium of instruction, and the results showed that students from the FE outperformed their counterparts in FAS, FMC, and FAC in the reading test. The study had several theoretical and practical implications in language testing and validation, and testing reading.