July 24, 2017

EU Stakeholders – What Makes Good Health Apps?

EU stakeholders struggle to find consensus on health apps’ reliability and validity:

Which criteria should be used to assess and evaluate health apps? We want to ensure it is safe, reliable, valid and effective.

What makes good health apps?

To find out, the European Commission convened a Working Group on mHealth Assessment Guidelines. This was in March 2016 and invited 20 organisations (including PatientView/myhealthapps.net) representing patient associations, healthcare professionals, industry, research and academia, payers and developers from across the whole of the EU. As well as 8 national public health authorities, to begin work.

The ‘answers’, unsurprisingly, were varied and nuanced depending on the different stakeholders’ perspectives and experience. There was consensus on much of the baseline criteria. This covers the majority of the working group, diverging definitions of each criterion and its importance/relevance. It was coupled with industry’s wish to keep the scope to a more narrowly focused set of criteria. This meant it was impossible for the group to agree on a set of draft guidelines. The process clearly identified the key sticking points for further work and much existing good practice to build upon.

In addition, The European Commission recently published the working group’s report and annexes. This details different stakeholder positions on the assessment criteria. It also includes case studies and examples of good practice across Europe. The work will now be taken up by the eHealth Network (member states’ representatives responsible for national implementation of digital health rollout).

Objectives for health apps

Firstly, our mission was to develop guidelines for assessing the validity and reliability of the data that health apps collect and process. This is building on existing initiatives and best practices in Europe. According to the Commission’s call, we would:

“seek to provide common quality criteria and assessment methodologies that could help different stakeholders in assessing the validity and reliability of mobile health applications … to link data from these apps to the electronic health records.”

‘Grey zone’ health apps

In essence, we were tasked with developing methodologies for evaluating apps which do not come under the Medical Devices Regulation but have some ‘medical’ functionality. (eg capturing data which can be shared with electronic health records and physicians/clinicians. Those data may be taken into account in diagnosis and treatment decisions). For mobile technology to become mainstream in healthcare, it is recognised that patients, healthcare professionals and health authorities need an agreed assessment framework. This is so health apps and the data collected in order for trust, uptake and rollout to be improved.

Following on, thirteen criteria were initially debated as potential baselines for assessment:

Firstly, privacy
Secondly, transparency
Safety
Reliability
Validity
Interoperability
Technical stability
Effectiveness
Efficacy
Efficiency
Accessibility
Usability/desirability
Lastly, scalability

But straightaway it became clear that finding agreement on the relevance and definitions of each would prove difficult. This is because as these depended on the context, experience and use of apps by each stakeholder group.

For example, the public health authority (PHA) members saw no clear distinction between ‘reliability’ and ‘technical stability’. If an app is not technically stable, how can you trust that it will function and collect data reliably? Moreover, for industry members ‘technical stability of an app has little to do with the app produces reliable and valid data.’

Following on, the PHA members also saw ‘scalability’ as a vital criterion. This is to ensure that an approved app could be rolled out sustainably. This to the relevant populations within their countries or regions. Whereas for industry this was purely an issue for the market to decide. (See Annex 1 to the report for a summary of all stakeholder positions).

PatientView and patient groups

On the ‘transparency’ criterion, PatientView and the patient groups represented (European Cancer Patient Coalition and the European Federation of Allergy and Airways Diseases Patients’ Associations). To a large extent the payers and social health insurers–agreed that:

“Transparency means that the user can easily and clearly identify the developer, the seller/distributor and/or the person or entity that ordered and financed the app, as well as the written purpose of the app that should correspond to what it really does. Moreover, the user should be informed which of his/her personal data the app needs for its smooth function, how his/her data are handled after he registered them in the app, to whom and when they might be transmitted for further processing. What happens to his/her data if he/she stops using the app? Will the app prompt the user to re-use the app or inform him that he may delete his data if he does not intend to re-use it? Moreover, the user should be informed if he/she can delete permanently any data he/she might have entered, if he/she does not agree with how it is used subsequently.”

The scope on health apps

There was also a split between industry and the rest of the group about the scope of the assessment. The original mandate called on us to investigate the reliability and validity of the data collected by apps. There was a growing feeling by the non-industry members that the reliability and validity of the apps should be assessed. If the modules and algorithms that went into making up the app were inaccurate or faulty, how could we trust the data collected by that app?

In conclusion, what is clear is that ‘reliability’ and ‘validity’ mean very different things to different stakeholders. This is depending on their fields of operation. With the rapid evolution of mobile health apps and digital health tools, finding a ‘one-size-fits-all’ method will not be easy. This would be for assessing which ones are safe and effective for patients.

NEXT STEPS:

The European Commission has recently published the working group’s report and annexes detailing different stakeholder positions on the assessment criteria, plus case studies and examples of good practice across Europe. The work will now be taken up by the eHealth Network (member states’ representatives responsible for national implementation of digital health rollout).

The World Health Organization and the International Telecommunications Union have set up a joint mHealth Hub “to champion the uptake of mHealth … [and the] foundation and maintenance of a centralised ‘Knowledge and Innovations Hub for mHealth’ to monitor and enable mHealth adoption and innovation.” This is linked to an EU project (2017-2021) for an EU mHealth hub including evidence for the integration of mHealth in healthcare systems.