Faculty Spotlight: Carl Whithaus
Dr. Carl Whithaus, Director of UC Davis’ University Writing Program, has been studying automated essay evaluation (AEE). In a recent article, “Challenges of Using Automated Essay Evaluation (AEE) Software for Assessing Multimodal Writing” (in press)*, Whithaus notes that, while various AEE systems (namely, ETS’ E-Rater, Pearson’s Intelligent Essay Assessor, and Vantage’s IntelliMetric) use different criteria and techniques to evaluate essay quality, they all only consider alphabetic writing.
Whithaus thus questions how AEE can exist alongside writing pedagogy that values multimodal composition, which is increasingly common. As Whithaus explains, “[m]ultimodal composing activities are no longer only present in avant-garde writing across media or writing in digital environments courses; rather, students in first-year composition courses, writing across the curriculum classes, and writing in the discipline courses are increasingly being asked to engage in multimodal writing activities.”
As the definition of writing shifts to include digital and multimodal elements, AEE developers would be remiss to maintain systems that only measure alphabetic writing; they need to start revising their measurements to include multimodal composition. While AEE systems are unlikely to properly evaluate completely non-textual compositions (i.e., videos) anytime in the near future, Whithaus argues that “[w]ritten texts that use some multimodal elements provide” a good starting place for adapting AEE software. On possible adaptation, Whithaus says, is to combine AEE evaluations with human raters who rank “both the effectiveness of visuals as well as the relationship of the visual with textual elements.”
Much work remains before AEE systems can properly include multimodal elements, but Whithaus convincingly argues that we are not too far from such developments.
To learn more about the development of multimodal assessment methods and the current attempts to influence education policy, see the work being done by:
- the National Writing Project’s Digital Is… initiative,
- the National Assessment of Educational Progress, and
- the Council of Writing Program Administrators.
* “Challenges of Using Automated Essay Evaluation (AEE) Software for Assessing Multimodal Writing” will appear in Measuring writing: Recent insights into theory, methodology and practices, edited by Marion Tillema, Gert Rijlaarsdam, Huub Van den Bergh, and Elke Van Steendam, published by Emerald Books (UK) in 2013.