DENVER, Colo. – Over 40 members of the Colorado digital health ecosystem met on August 23^rd to assess the startups that had applied for this year’s Prime Health Challenge. Representatives from Denver Health, Medical Group Management Association, and the American Diabetes Association were among those chosen to evaluate the applicants.

“We need to set a high bar of acceptance,” Jeffrey Nathanson, the CEO of Prime Health, told the judges. “We’re looking for the best and the brightest.”

It was the first step in a three-month-long evaluation process that would culminate in a pitch competition on October 19^th. It was also an attempt to address a need shared by digital health startups and healthcare providers around the world.

A Need for Standards

The judges followed an evaluation process known as Prime Health Qualify.

Founded in 2012, Prime Health is responsible for convening the Colorado digital health ecosystem – the members of which include clinicians, executives, entrepreneurs, investors, administrators, academics, technologists, and policy-makers – at its monthly gatherings, quarterly summits, and annual challenges.

According to a 2015 Deloitte Business Trends report, “ecosystems are dynamic and co-evolving communities of diverse actors who create and capture new value through both collaboration and competition.” In the case of the Colorado digital health ecosystem, the startups, accelerators, advocacy organizations, and healthcare providers within it collaborate and compete to develop new digital health interventions like smartphone apps, wearable devices, and telemedicine platforms.

Though evidence is growing of the efficacy of digital health interventions, a common set of standards by which to assess them has yet to be developed – leading to dubious claims by the makers of some, and scathing critiques from leading figures like James Madara, CEO of the American Medical Association, who recently referred to such over-hyped interventions as “digital health snake oil.”

After launching in 2010, app-certification startup Happtique attempted to assess the thousands of health and wellness apps on the market, but had to shut down in 2014 after investigators discovered serious flaws in its certification program. Since then, others have attempted to fill the vacuum left by Happtique – including Evidation Health, which raised $11.5 million in June to assess digital health interventions – but as of yet none have achieved a scalable means of doing so.

Enlisting the Ecosystem

Startups were assessed using a rubric created in partnership with Valid Eval.

Under Nathanson’s guidance, Prime Health has spent the last three years developing a process for assessing digital health interventions that leverages the collective expertise of the Colorado digital health ecosystem. Known as Prime Health Qualify, this process employs a rubric created in partnership with Valid Eval, the maker of the online evaluation system at the heart of the Prime Health Qualify platform.

Each year, Prime Health invites digital health startups to apply for its annual challenge, a three-month-long program that culminates in a pitch competition where entrepreneurs attempt to convince major healthcare providers like Kaiser Permanente and HCA to pilot their products. Once applications have been collected, Prime Health matches the applicants with anywhere from five to 10 domain-area experts, who assess them using the Prime Health Qualify rubric.

After their assessments have been completed, the judges gather to review the composite scores generated for each startup by Prime Health Qualify. During these gatherings, they examine the areas of disagreement identified by the program and attempt to reach a consensus on the companies in question.

At Tuesday’s meeting, the judges decided which startups would go on to the semi-finals round of this year’s Prime Health Challenge. The gathering featured lively discussions about the companies’ value propositions, with the judges identifying areas in need of improvement and considering how each product could be integrated into the healthcare system.

“Our goal is to become the number one health innovation ecosystem in the country,” Nathanson explained after the meeting. “By engaging a large number of healthcare executives and ecosystem leaders in this review process, we’re giving the startups in Colorado a competitive advantage that doesn’t exist elsewhere.”

Like the coverage that CyberMed News provides? Follow us on Twitter, LinkedIn, and Facebook to make sure you keep up to date on the most recent developments in digital health.