This category of review seeks to answer two questions:
- Is the DHT safe?
- Is it clinically effective?
The criteria are largely based on the Evidence Standards Framework (ESF) published by NICE (UK based National Institute for Clinical Excellence). The Framework categorizes digital apps into tiers based on functionality. There is a difference in expectations of evidence/assurance quality, dependent upon the relevant functions and features of the DHT and its area of focus. The evidence/assurance expectations for each Tier gradually increase to reflect the increasing ‘risk’ profile of the products in each Tier. This approach is designed to ensure that technologies are held to a reasonable standard. This tier structure is detailed further in the table below.
The Framework looks for evidence of an appropriate professional being involved in the app’s design and development, or if the app has been externally accredited by an appropriate organization. A relevant professional is deduced in the context of an app. For example, for a simple yoga app the assessors would accept a qualified yoga instructor as a relevant professional, but for a complex clinical solution only a relevantly-qualified physician would be acceptable.
At higher levels of complexity and risk, the assessment includes an internet search for any evidence published about the app. If this exists, the assessor reviews the evidence against a series of questions to determine the quality of this evidence and the source publishing the evidence.
If the app includes behavioral change techniques, some points are awarded for referencing the evidence for the behavioral change techniques used within the app. However, higher scores are awarded if the app has provided direct evidence of its own effectiveness through a high-quality study.
If the DHT meets the definition of a medical device, the highest tier of complexity, the assessment looks for FDA clearance or pre-market approval.
The TIER structure
| Tier | Functionality or Purpose of DHT | Examples | Assurance of quality provided by the following |
| A | Provides no measurable patient impact, but do provide services to a health and care system. | ||
| Bi | Provides information, resources or guidance to the user about a specific condition or general health and lifestyle. Allows a healthcare professional to provide advice via the app. Allows two-way communication between patients or healthcare professionals. Simple monitoring, defined as recording data over time for the user to review, with no “intelligent” manipulation of the data by the app, and where the monitoring is not condition-specific. App does not interpret the data to produce a reminder, alert or notification. | Self-monitoring with a focus on well-being and general health. App-based journal to log symptoms or other markers. App may render the data in graphical views that illustrate trends and patterns. | Evidence that a suitably qualified professional or professional body is behind the DHT. |
| Bii | Provides condition-specific simple monitoring, or app which is not condition-specific but provides the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments, | Activity tracker with reminders. | Evidence that the DHT has been adopted and used (within a recognized healthcare setting if applicable) or evidence that the DHT is actively endorsed by a recognized healthcare system. |
| Ci | Involves the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments regarding the management of a specific condition. Delivers preventative behavior change. Has a recognized clinical calculator within the app (i.e. novel clinical calculators are not included here). | DHT which logs your manually entered blood sugar levels, and provides insights regarding whether you are controlling your diabetes better or worse than last month. | Safety can be demonstrated through suitable test data if applicable I.e. accuracy of clinical calculators etc. or through high quality observational studies/RCT’s that demonstrate safety. For clinical efficacy in Tier C, a high quality observational study is the minimum required. High quality study indicated by: Presence of a suitable comparator and p value of <0.05 (or non-overlapping confidence intervals). Suitable sample group/size. Alignment to claimed benefits/safety risks. |
| Tier Cii | Diagnoses a condition. Contains a novel clinical calculator. Automatically measures/records and transmits specific condition data to a 3rd party without input from the user. Provides treatment or guides the treatment of a condition. Alleviates symptoms of an existing condition. | Medical devices as classified by the FDA. | Safety can be demonstrated through suitable test data, or through high quality observational studies/RCT’s that demonstrate safety. Default expectation around clinical efficacy is an RCT – although there are scenarios where an RCT might not be practical, including diagnostics where it would not intuitively make sense to randomize to one diagnostic or another, but rather compare the same patients (and diagnoses) using the old (and new) diagnostics and measuring agreement between the two. High quality study indicated by: Presence of a suitable comparator and p value of <0.05 (or non-overlapping confidence intervals). Suitable sample group/size. Alignment to claimed benefits/safety risks. |