But how rigorous is the research that goes into developing health and wellness apps?
Flo’s science team went digging and analyzed if and how these apps are testing how well their products work. What we found was an overwhelming absence of diversity among study participants and a wide range of different study types, highlighting the lack of standard approaches to app testing.
This research is published in the journal BMC Digital Health. Read on to find out more details.
What type of research goes into app development?
You might be surprised to hear that most consumer health and wellness apps are not regulated by health authorities like the United States Food and Drug Administration (FDA). This category of apps includes many diet and nutrition, exercise and fitness, mental health, and sleep apps.
Only apps developed for medical purposes require approval and regulation by the FDA and other health authorities in their respective countries.
When you see your doctor about your health, they will use evidence-based medical guidelines for your care, which are based on scientific research. Yet for health and wellness apps, no such requirements exist. But unlike apps that help you with your food deliveries or your travel, the advice and information you get from these apps should be backed by scientific evidence.
“There are hundreds of thousands of health apps on the market — whether you’re using a wellness app to track your fitness or nutrition or an app to manage a health condition — but only a tiny fraction of these have actually been proven to work in scientific studies,” Liudmila Zhaunova, PhD, director of science at Flo, commented on the team’s research.
But before you start deleting your health apps from your phone, there is positive news.
The team first looked at 2,799 studies that were published on the topic of mobile health app evaluations. Among these, we identified 739 studies for further screening. This included a mix of app types, for example apps designed for health care professionals and for consumers, as well as a mix of study designs and publication types.
Focusing specifically on research that included apps with at least 50,000 downloads and that assessed how much the app improved the users’ health and well-being — as that’s the best way of testing a health intervention — led us to 47 studies published in scientific journals over the past five years.
While this is a small number, it showcases that good-quality research is possible when developing a health app. It was also encouraging to see that companies at all development stages were conducting scientific studies, including start-ups and scale-ups. This investment in science indicates a growing recognition of the importance of scientific validation.
There are different ways to assess efficacy, but the most rigorous way of doing this is to run a randomized controlled trial (RCT). In our analysis, the team found that 30 out of the 47 studies were RCTs.
“RCTs are the gold standard of testing how well an intervention works, whether that’s a new drug, vaccine, or a health app. We were pleasantly surprised to find that among the studies testing health apps in the past five years, there were quite a few RCTs,” Carley Prentice, MPH, lead study author, explained.
“This is encouraging, as it shows that developers who do put their apps to the test are willing to use rigorous and high-standard study designs,” Carley continued.
Our scientists recently put Flo to the test in an RCT. You can read more about what we found here.
Who is included in scientific studies?
Next, our team looked at the participants in the scientific trials. Here, we found that 29 of the 47 studies had been conducted in the United States, and the remainder, in 14 other countries across the globe.
Ethnicity was only reported in 27 of the 47 studies. Of these, four apps were designed for specific populations.
The remaining studies included on average:
- 62% white participants
- 7% Black participants or individuals of African descent
- 0.4% Asian participants or those of Asian descent
- 7% Hispanic or Latinx individuals
- 9% of people who were biracial or multiracial
- 0.3% of participants from Indigenous groups
This analysis clearly highlights the lack of adequate representation in the majority of studies that test health apps.
“It’s important that scientists who investigate how well an app works recruit a diverse group of participants for their studies. Diversity in recruitment is a challenge in many studies, but it’s something that we need to actively work toward,” Liudmila pointed out. “Inclusivity and equitable access to health apps should be everyone’s goal, and research needs to reflect this.”
Why is Flo’s research important?
With this work, the science team hopes to highlight what the paper refers to as an “evaluation crisis”: the fact that the majority of health and wellness apps are released to consumers with little evidence that they truly provide health benefits.
“Looking at the evidence, we see that it is, in fact, possible for health and wellness app companies to run RCTs, even with smaller sample sizes and shorter durations. However, for developers looking for a simpler and more cost-effective approach, pre/post studies offer a solid starting point that still provides valuable insights,” Carley said.
In light of the lack of guidelines and standardized best practices for wellness health app developers, the team also hopes to provide the beginnings of a blueprint for other companies to incorporate robust science into their product development.
