Challenges of Evaluation in Digital Education

Journal Club Article: Cook, D. A., & Ellaway, R. H. (2015). Evaluating technology-enhanced learning: a comprehensive frameworkMedical Teacher37(10), 961-970.

Aim: To outline a comprehensive framework for Technology Enhanced Learning evaluation, which is the process of judging the value of education.

Rationale: Lack of clear focus of evaluation for difitical technology educational activities, including online learning.

“Technology Enhanced Learning can involve web-based technologies, mobile devices and apps, computers, tablets, and other digital devices, and may include activities that are entirely digitally-mediated or those that integrate technology into hybrid or blended activities.

Cook & Ellaway, 2015

General Principles of Evaluation:

  1. Objectives Orientated Focus – how well were course objectives met?
  2. Participant Orientated Approach – the learners experience
  3. Process Orientated Evaluation – consider the life cycle of a course from implentation and delivery.

Special Considerations in Technology Enhanced Learning:

  1. Conduct needs analysis and environmental scan – detrmine the need for change at the start before educational change.
  2. Document processes, decisions, and final product – educational design and framework.
  3. Test usability – “Learnability refers to how easily users can accomplish basic tasks the first time they encounter the technology.”
  4. Document key events during implementation and final product.
  5. Assess participant experience and satisfaction – the value and meaning of the learning.
  6. Assess learning outcomes – Kirkpatrick Model of Evaluation
  7. Estimate cost, reusability, and sustainability

Practical Approaches to Principles of Evaluation:

Three new instruments were developed to assess perceptions and experiences following participation in a Technology Enhanced Learning course for both learners and instructors. The evaluations allows use of a a consistent instrument in evaluation which allows comparisons within and across courses.

Some challenges with evaluating Technology Enhanced Learning:

  • Interactions between teachers and learners are different in Technology Enhanced Learning when compared to face-to-face learning envrionments.
  • Technology Enhanced Learning can generate a lot of data, and different types of data, there is a need to consider how this will be analysed.
  • Is the learning technology as advanced as the technology used in the healthcare workplace?