Evaluating Adaptive Work-Integrated Learning Systems: From the Lab to the Field

Evaluation frameworks have been presented that suggest layered evaluation of adaptive systems along two dimensions: (i) the software development cycle, and (ii) the component of the adaptive system that shall be looked at. We argue that a third dimension is crucial: the question whether an evaluation should take place in the lab or in the field. We present a refined systematization of evaluation approaches using the evaluation of the adaptive WIL system APOSDLE as an example.

[1]  S. Weibelzahl,et al.  Framework for the Evaluation of Adaptive CBR-Systems , 2001 .

[2]  Francesco Ricci,et al.  User Modeling, Adaptation, and Personalization , 2013, Lecture Notes in Computer Science.

[3]  Judith Masthoff,et al.  Layered evaluation of interactive adaptive systems: framework and formative methods , 2010, User Modeling and User-Adapted Interaction.

[4]  Tobias Ley,et al.  A methodology for eliciting, modelling, and evaluating expert knowledge for an adaptive work-integrated learning system , 2010, Int. J. Hum. Comput. Stud..

[5]  Neil A. M. Maiden,et al.  Evaluating the Adaptation of a Learning System before the Prototype Is Ready: A Paper-Based Lab Study , 2009, UMAP.

[6]  Peter Brusilovsky,et al.  The Benefits of Layered Evaluation of Adaptive Applications and Services , 2001 .

[7]  J. Heckman,et al.  Lab Experiments Are a Major Source of Knowledge in the Social Sciences , 2009, Science.

[8]  Stefanie N. Lindstaedt,et al.  Supporting the Learning Dimension of Knowledge Work , 2009, EC-TEL.

[9]  Tobias Ley,et al.  Scaffolding Self-directed Learning with Personalized Learning Goal Recommendations , 2010, UMAP.

[10]  Ulrike Cress,et al.  Learning in the Synergy of Multiple Disciplines, 4th European Conference on Technology Enhanced Learning, EC-TEL 2009, Nice, France, September 29 - October 2, 2009, Proceedings , 2009, EC-TEL.

[11]  Thea van der Geest,et al.  User-centered evaluation of adaptive and adaptable systems: a literature review , 2008, The Knowledge Engineering Review.