My Role

Conduct user tests

I conducted the user test with international colleagues from Sweden, Korea, and China in Sweden. I interviewed Swedish clinicians and observed their behaviors when performing the DMSS.

Analyze data

I analyzed the data of videos and interviews. All researchers discussed the results and adjusted it to increase inter-reliability.

The Challenge

Developing a quick formative evaluation tool with the theoretical base

TTo evaluate the individuals’ learning process is challenged because it requires the capability of interpreting users' behaviors. Therefore, the evaluation tool has to be developed in line with cognitive and learning theories. We used Activity Theory and Zone of Proximal Development (ZPD) to design the AAIMA protocol (Assessment of Autonomy in Internet-Mediated Activity) as shown in the following approach section.

The Approach

User recruitment

We set up a workshop of DMSS for physicians after their official meeting. Four physicians with different levels of expertise using the Dementia Diagnosis and Management Support System DMSS. They were asked to think about their patient cases when performing the system.

AAIMA protocol

TThe AAIMA protocol consists of activity levels and independence levels. The activity levels consist of actions and then operational levels.The independence levels includes:

• Autonomy, which means the user is able to perform the system. independently
• ZPD-S, which means the user could be guided by the system as the more capable peer.
• ZPD-H, which means the user needs further education involving more experienced colleagues.

The Result

Experienced physicians

• Apply well-known patient cases to learn how the CDSS works.
• Could on their own, develop their skills in both using the system and assessing dementia using the CDSS as the instrument.
• Were able to assess their dementia cases using the system in the linear, checklist manner.
• Could efficiently utilize the system to evaluate her own assessment, moving back and forth between data capture and diagnosis functionalities.

Novice physicians

• Showed more difficulties mastering DMSS and its purposes during the short time they were observed.
• Found the assessment of each symptom as a challenge, with their limited knowledge in the domain.
• Had difficulties to perceive the overall scope of dementia assessment and its different areas of interest.
• Considered the trust in this system but perceived the system was helpful to diagnose after knowing all assessments in this system were in line with the major clinical guidelines.

Deeper insights

Adapted design

Participants with the novice level of experience need a more substantial introduction to dementia assessment guided by a more experienced physician.

Medical education instrument

The results suggest that DMSS has the potential to function as an instrument for continuing medical education in the dementia domain for clinicians with some experience from dementia assessment, while novices with very minor or no experience need to combine the use with access to medical expertise in clinical practice.

Impact

Proposing a quick evaluation tool

The AAIMA protocol is an efficient tool to help UX researchers and designers to decompose the task and evaluate users’ behaviors and performance. Meanwhile, this project presented a practical case of using the tool.

International impact

The project was presented in one of the three most important medical informatics conferences, Medical Informatics Europe conference (MIE) in Gothenburg, Sweden. More than 500 people related to medical, educational, and technological fields from the world attended the conference. Afterwards, the project was published by IOS Press as an open access ebook in the series “ Studies in Health Technology and Informatics”.