Challenging Methodologies in Weight Loss App Trials
Methodologies in App Development
The same challenge applies to applying typical clinical trial methodologies to app development. The few trials tend to be historical. By the time the trial reports, the app may have been changed many times or been retired.
For years we’ve almost become resigned to the fact that digital health is developing faster than our ability to evaluate it. When challenged, app developers’ first line of defence can be to say, “you’re evaluating us against old world criteria using Stone Age processes”.
‘Low quality’ research
A literature review recently published study in the American Journal of Preventative Medicine identifies further challenges with research into health app outcomes.
The review, “Mobile Health applications in weight management” looked at existing evidence on the efficacy of mobile health technology in facilitating weight management behaviours. The authors, Katerina Dounavi from Queen’s University Belfast and Olga Tsoumani from the Vrije Universiteit Brussel, reviewed 39 studies between 2012 and 2017.
Highlighting “the low research quality” of studies reviewed, the authors highlighted that:
Only 23% of studies “presented no risk of bias, with remaining studies presenting a high risk of bias in one or more key domains…This outcome is a cause of concern, given RCTs [Randomised Control Trials] have been considered the gold standard for policy-making and consequent clinical decision making” – Dounavi and Tsoumani
As yet, there seems to be few common methodologies. We appear to lack a common set of metrics, or even common language to evaluate health apps. To address this, the authors make two key recommendations:
“RCTs [Randomised Control Trials] need to be reported in a standardized manner.”
“Behaviour change strategies need to be rated following a coherent behavior analytic framework” – Dounavi and Tsoumani
This last point is key. For example, apps have a huge potential in helping people to manage long-term, chronic health conditions. Yet, we are overwhelmed with diverse ways of measuring and expressing how they do this. How does a patient, let alone a payer, clinician, insurer, or even developer really know what “good” looks like, or make an informed choice?
Which way forward?
Arguably, it’s down to the development community to demonstrate the evidence base to patients, clinicians, regulators and payers, even if few app business models can support this. This literature review in itself is a small example of developer and academic co-operation. It was partly supported by habeats, a weight management app, and one of the authors, Katerina Dounavi, is also a member of the developer’s advisory board.
Getting any app noticed is challenging, but weight management is particularly clogged. This literature review clearly demonstrates that we are a long way from having quick, clear and relevant methodologies for app evaluation. On the other hand, pragmatically, for now, is any evidence better than none?
Download the literature review