The development and argument-based validation of an analytic rubric for L2 writing electronic portfolios

Abstract

Electronic portfolios are used in many institutions worldwide. Despite the numerous benefits, few studies have investigated the usefulness of the ePortfolio in the writing classroom, in general, and looked into providing reliable and valid scoring rubrics to assess L2 writing ePortfolios, in particular. The purpose of this study is to build a validity argument for WASPER, an analytic rubric developed for L2 writing ePortfolios, using Bachman and Palmer’s (1996) test qualities and Bachman’s (2005) Assessment Use Argument as a basis. Validity is established by evaluating the empirical data that serve as the backing for the inferences of the interpretive argument, in light of evidentiary reasoning. The five claims of the argument-based approach to validity are: (1) WASPER is a reliable tool to assess L2 academic writing ePortfolios; (2) WASPER is relevant to the construct being measured; (3) WASPER corresponds to a more authentic way of assessing academic writing for L2 students in the TLU domain; (4) WASPER has the potential to positively impact on the teaching and learning of L2 academic writing; and, (5) WASPER’s development was practical, and it is a practical scale to use to assess writing ePortfolios. The L2 writing ePortfolio assessment was implemented for one semester with B.Ed. students, at the Institute of Education, International Islamic University Malaysia. Thirty-eight students participated in this study, along with thirteen raters. The mixed method approach, and the Process Model (Chatterji, 2003) for the design and validation of rating scales, were both used to develop and validate WASPER. Both qualitative and quantitative data were collected to serve as evidence to back the warrants or reject the rebuttals in the validity argument. The Many-facet Rasch Measurement (MFRM) was used to evaluate the quantitative data. Anonymous feedback forms, standard open-ended interviews and classroom observation notes were used to elicit the raters’ and students’ perceptions of the usefulness of the sale and the ePortfolio assessment in general. The results indicated that the five-point scoring rubric WASPER is a reliable and valid instrument for the purpose of assessing L2 writing ePortfolios. The FACETS analysis showed that the WASPER rating scale categories were all ordered and functioned appropriately. A reliability of 0.81 with a 19% error variance indicated that the examinees were well differentiated in three levels of ability. However, it was found that the criterion Ease of Navigation should be removed from WASPER as it is related and has an impact on the assessment of the other criteria. Evidence was also found that WASPER corresponds to an authentic way of assessing L2 writing and has a positive impact on students and raters. The limitations and implications of the use of WASPER, as well as suggestions for refinement in future rounds of validations are discussed.