Story Backgroud

The majority of older adults use an increasing number of medications as they become older. 25% to 40% of adults aged 65 years or older are prescribed at least 5 medications. Management of adherence, which is the process of supporting and monitoring patients’ adherence to medication is a challenge for older adults because of their impairments of sensation, perception, and cognition.

This is the story of how the team and I created a possible and innovative augmented reality (AR) system to support the daily task of managing medication adherence by using pill dispensers for older adults in Sweden.

*Med is an abbreviation of medication and “with” in English and Swedish respectively.
*This project result has been published in the Journal of Cognitive System Research.
*The content was partially reinterpreted to help reading.

My Role

I enlisted the cooperation with the research group on Interactive Intelligent Systems in the Department of Computer Science at Umeå University in Sweden.

In Sweden, I led the UX work, conducting context inquiry, designing iterative prototypes, and managing user tests. I cooperated with system engineers from different countries, Swedish field experts and target users between November 2017 and March 2018.

The Challenge

Exploring target users and stakeholders’ needs for medication management with using pillboxes.

Many different kinds of technology have been developed to support older adults’ medication management and improve their medication adherence. However, the intervention technology is unlikely to increase older adults’ medication adherence including correct medication distribution and the evidence is limited. The reason is the designs of technology are technology-driven and lack of situated exploring. Therefore, the team decided to conduct context inquiry. I visited the homebound elderly and interviewed them and field experts to collect and analyze their needs. Afterwards, I shared findings with the team to design the application.

Designing the application with using limited resources and time.

This is a bottom-up project, which means the team had to coordinate the existing resources to make a quick development and test. The users and field experts participated in each design and evaluation process.

Conducting a theoretical-based structure of design

The project was planned in an academic unit so the theoretical-based structure was necessary. In addition, a solid cognitive and artificial intelligence foundation supported the scientific validity of the project and the application. The design process based on Activity theory and formal argumentation theory, which provided the structures of behavior analysis of the medication distributing and reasoning of the application respectively.

The Approach

Deep in the literature

An efficient literature review gave an overall view of design possibilities. I reviewed the literature systematically from device and co-design perspectives. I found:

1. Possible technology
AR can be an efficient technology to improve older adults’ daily independency and control since AR can be located and used in their familiar, daily environments. In addition, the projection-based AR application is especially appropriate to support the medication distribution for older adults because users can practice with their hands the actions and the unlimited display size enables users who have poor vision to read relative instruction clearly.

2. Successful design factors
The keys to succeeding and to avoiding the failures of past eHealth/AI efforts are user-centered design, evaluation from the beginning, and interdisciplinary collaboration with medical experts.

3. The process of using participatory design to design an AI system
Creating design idea, concretizing idea, and evaluating the result.

Stand in users’ shoes

Besides the inputs of literature review, we used participatory design methods to comprehensively understand the process of using pill dispensers to manage medications in older adults. In addition, this method provided the users and stakeholders to present their needs and opinions.

The Discovery

Certain requirements of medication information

The participating older adults relied on medication dispensers to manage their medications. When older adults distribute their medications to the dispensers, they need certain information about the medications and prescriptions, which is the same information that caregivers need.

Multiple sources of medication information

The different sources of information were viewed as sources of stress and confusion, for they caused worries that the patients might miss something important. In addition, sometimes the medication provided might vary in appearance from time to time because the healthcare system provides the cheapest brand, which changes over time. Therefore, older adults relies on the label of medication instruction on medication packages to distribute medication. However, the font size of the medication instruction labels were too small to read easily.

“What is said on the label is really important. Even though the doctor said but you didn’t listen to him, maybe you were thinking about what to do next and you missed some information what doctor said that was not on the label.”

Storing medications in certain cabinets

Older adults removed the medications from the cabinets and then distributed the medications to the dispensers. However, medications in cabinets may belong to different family members who live together and/or visit occasionally. Moreover, the older adults mentioned that some medications that had not recently been used were stored in the cabinets as well.

The Vision

The Design for home-situated medication adherence improvement

The home-situated medication management is individually different from personal prescriptions, the numbers of family members and the environments of storing medications. Therefore, we proposed the following goals of the MED-AR system to improve medication adherence:

1. Personalised support
2. Transparent information with source information
3. The provision of information both when the medicine is distributed and when it is taken
4. Context awareness(e.g., information about whether some medicine is old or prescribed to another person in the household)
5. Coordination between human and Intelligent Assistive Systems (IAS) support

The Framework

The framework of designing the MED-AR based on participatory design with three steps: creating design idea, concretizing idea, evaluating the result. Following the process, we sharpened and integrated our ideas about how to design the interaction and the AI system. Moreover, we used different methods to collect ideas and data in the three steps respectively.

Workshop: Collecting multidisciplinary perspectives

To clarify the possible using scenarios, potential users, and system building, multidisciplinary experts and designers (Medical care, AI, HCI) gathered to discuss the ideas. We summarized that the outcome of the system was to improve users’ medication adherence, which was supporting them to distribute medications correctly and the main target users are older patients.

Interview and prototyping: Integrating interdisciplinary solution

I conducted interviews with older patients and care providers to help the team have a deeper understanding of the using scenarios and demands. A specific design of the system and device was proposed according to the inputs from the interviews. We used the concept of smart cabinet simulating the medication storage environment in home to design the device. The gesture and text recognitions, argument-based reasoning process, agent-based mechanism, and projection-based AR were used to construct the system.

Usability evaluation

I conducted user tests to observe user behavior and evaluate their performance. I used task correctness, task time, the AAIMA (Assessment of Autonomy in Internet-Mediated Activity) protocol, the Raw NASA Task Load Index (RTLX) and the System Usability Scale (SUS) to evaluate the system usability. The follow-up interviews were also conducted to collect users’ suggestions.

Detailed Design

Personalised support

Patients’ prescriptions were constructed in the database. The personal medication instructions information would be provided to the users by the system when the users started to distribute medications. Different languages of the medication instructions can be provides depending on users’ nations.

Transparent information with source information

The system reorganized the medication information layout. The users could understand the required information of medication distribution task at a glance. Afterwards, the abstracted information could easily assist the user to double-check their medication distributions.

Coordination between human and IAS support

Four kinds of support levels were designed to provide adaptive supports for users.

• Independent activity execution
  

In this scenario, the user could perform medication distributing independently. The assistive system does not take any decisions oriented to support the activity. The role of an assistive system is observe and learn without suggesting active support.

• Activities supported by caregiver
  

In this scenario, an individual receives support from another person, for example a caregiver. Similarly to the last scenario, the role of an assistive system is observe and learn without suggesting active support.

• Activities supported by an assistive system
  

In this scenario, an assistive agent takes the decision to support an individual by taking active decisions such as projecting the abstracted medication instructions.

• Activities supported by both caregiver and agent
  

This scenario involves the joint support provided the client by both a caregiver and an assistive system. A main challenge for the assistive system is to detect: (1) actions that the caregiver executes, and (2) observations of the caregiver and client.

Impact

International impact

The project were presented at the Joint Workshop on Artificial Intelligence for Health in Stockholm in Sweden and published in the Journal of Cognitive Systems Research.

Satisfying design

“I think it was really good that it become a such big picture you can see very clearly. Because it is a problem that the text is small on the packages and there is a lot of the text on the package. So I think that is really clear. I also like the picture of the box and pills because the challenge is there are a lot of boxes with different names and the pills can be look different. It confirms that you have the right thing.”---The field expert in health care

“I like that the projected information is clear and helpful. I think older patients with taking lots of different medications need this system.” ---The elder participant of this project