Before I dive into some of the detail about funnel plots, I wanted to start by highlighting a few key concepts, driven by risk-based monitoring (RBM) which are likely to come into play, whatever your specific approach to RBM is going to be:
- You want to start assessing site quality risk as early in the trial as possible, giving you the best possible chance of addressing and mitigating that risk before it becomes an issue and possibly effects patient safety or data quality.
- You want to be able to compare all sites with all other sites in the trial at any point in the trial in order to derive clear direction as to where to focus your monitoring resources.
- You want a simple way to assess risk at sites, whilst those may be at different stages in their recruitment or treatment cycles and may also vary considerably in the numbers of subjects they have recruited.
If you agree that any or all of these concepts are relevant to your RBM approach then funnel plots will be key to the way in which you visually and statistically assess site quality risk, and therefore please read on!
Many of the key risk indicators (KRI) used in RBM use rates of values per site in order to ascertain study normals and subsequently compare each site to those normals. If we take the reporting of adverse events as an example, a popular KRI plots the average number of adverse events reported per subject visit against the number of randomized subject visits at each site. An example from the OPRA RBM platform is shown below:
Each point on the chart represents a site, and the more randomized subject visits have occurred at a site the further along the x-axis the site is plotted. For this trial with this particular study design and patient population you can see from the plot that the study average AE reporting rate is just over 0.5, i.e. one adverse event reported for every two subject visits. Once we have this study normal, we can that look at sites which vary significantly from that normal – either because they are over reporting or under reporting AEs.
So far, so good, but what happens when we get a site with two subjects, both of whom report an AE on their first visit? Immediately that site’s AE reporting rate shoots up to 1 AE / subject visit and it would likely get flagged for reporting at twice the study average rate. It is quite possible that with such a small sample size of two subject visits that both subjects would have reported an AE and so we have created a false risk signal. Now, if that site continued to report at 1 AE / subject visit with a sample of 20+ data points, we most likely have a legitimate cause for concern. So how do we make sure that early on in a trial when we have a smaller data sample we don’t create false signals due to sampling error?
The answer is that we have to vary the thresholds in accordance with the sample size at each site. The result of this is that instead of a straight line threshold, we actually get a curved line. When you now look at an indicator like AE reporting rate, where you want to monitor both under reporting and over reporting you need two curves – one above the normal and one below. This creates what looks like a funnel, and hence the name ‘funnel plot’. If you look at the green lines in the picture above you will see this funnel shape very clearly. The way that OPRA works is that it actually uses multiple thresholds, so that degrees of risk can be plotted. Some companies like this approach as it allows an earlier warning of risk and allows us to look at how risk changes over time. If a site is yellow on our visualization today, but it was green last week, that indicates we are heading towards red – a more concerning scenario than if it was red last week, we intervened with the site, and now it is heading back towards yellow.
Using a funnel plot also means that we can start assessing risk earlier in a trial. By having greater tolerance to risk by using broader thresholds we can avoid false signals, but we do still get a good early warning if we are beyond those broader thresholds and may well be able to pick up an issue much earlier than we would if we were waiting for a large set of data to be accumulated before we can start to analyse it.
The third concept of being able to rank sites against other sites at any time in the trial is also supported by the funnel plot. When we look at any given site’s position on the graph relative to a threshold we can effectively see the relative level of that threshold breech, and hence compare two sites in a statistically fair manner. We can see a good example of this in the illustration below:
If I compare two sites, A and B. Here is some data about those sites:
Site A: 20 randomized visits, AE reporting rate 0.2 / subject visit
Site B: 71 randomized visits, AE reporting rate 0.23 / subject visit
By a traditional linear rating system, we would rate site A as the riskier site, as it is further from the study average reporting rate. But, what our statistical model is saying, is that given the possible influence of sampling error, that site A is actually closer to the green threshold line than site B, and so when assigning a risk score to the two sites, site B should be rated as the higher risk, as by this point in the trial, with the data volume it has, it should be trending closer to the study average than it actually is. So in this example, it is looking like site B may well be slightly under reporting AEs, whereas site A probably needs to be monitored remotely for a while to see if it pulls into trend or continues more along the path of site B. OPRA also makes the distinction between the relative risk levels of the two sites through the use of color. Site B, the higher risk site showing as orange or ‘Moderate’ risk, whereas site A is yellow and still classed as ‘Mild’ risk.
By taking this approach we have been able to focus our valuable monitoring resource on the riskiest site (site B), avoided the possibility of both wasting time and money monitoring site A unnecessarily, as well as avoiding the possibility of damaging our relationship with the investigator by implying there may be a study conduct issue at that site.
The development of a RBM platform which performs frequent calculation of risk and subsequent re-plotting of data points and thresholds in order to always ensure an immediate assessment of risk, has taken TRI many man years to develop and deliver, but the good news is that you don’t need to worry or go through the same process – we’ve done all the hard work for you! To get instant access to funnel plots for your KRIs through OPRA – the risk-based monitoring platform, contact us today at firstname.lastname@example.org