Better risk-based thinking will help produce better risk-based monitoring

TRI are truly excited to be publishing a guest blog from industry expert, Ben Locwin. This is the first part of a two part article on some of the external influences on risk-based monitoring and what we can do to improve our capability in this space.

How People Can Get Better At Risk-Based Thinking and Improve Their Organization’s RBM

Misinformation about risk based monitoring abounds. In fact, the level of misinformation on the topic long ago surpassed the level of correct information. I wanted to provide some practical insight as to why risk-based monitoring is so important to the work you do.

There is a litany of examples of where top-level understanding of risk assessment concepts allows people to make substantially better decisions. I’ll also share one example with you in particular which is not only challenging status quo paradigms, but also has gotten some uncomfortable resistance because of its very ability to allow for better decision-making.

‘But Ben, isn’t that exactly what we want: Better decision-making wherever possible?’

No. Based on the decisions regular people make all the time, they don’t tend to strive for better decision making in actual practice.

 

How Thinking Goes Wrong: Lotto, Pools, and Jabs

Though it makes sense in a business meeting to use quantified reasoning – why wouldn’t you want to use data to make decisions? Well, the reality is that even though everyone claims this out loud, very few individuals practice this behavior for their own decisions. Think about decisions with poor expected values that everyone engages in all the time: 30 million or more people buy lottery tickets for Powerball in the U.S. and 15 to 45 million people buy tickets for U.K.’s lotto, which have such a tremendously-low expected value (based on probability of winning vs. not winning) that they are sometimes referred to as ‘tax on the stupid.’

  • But surely, Ben, people use evidence and data to make decisions that really matter, such as for health and safety?

Not so fast: Parents over-watch their children to such an extent that the moniker ‘helicopter parent’ has pervaded child developmental literature. And yet with all the irrational concerns that parents act upon, they continuously overlook one of the most lethal impacts to children: swimming pools. It’s only after suffering a significant incident or fatality that families with pools decide to change their perspective on them.

pool

Wolf-in-sheep’s clothing: The hazard level of this is tremendous compared with other more ‘social’ concerns

And on the topic of health, immunization rates both for pediatric populations (MMR, DTaP, HiB, etc.) have been falling recently at the hand of the entirely non-evidence-based ‘anti-vaxxer’ movement. And it’s not just with children: Consider your own participation or the participation of your friends and family in seasonal influenza vaccines. The rate in the U.S. in 2014 was 46%; It’s the most effective preventive measure that exists for the flu, and even in those who later present as seropositive from contracting the flu virus of a different strain, significant adverse effects and hospitalizations are reduced in those who were previously immunized. But when faced with the data, individuals do not often take the best course of action.

 

Prognosticating Future Problems: A Criminal Justice Pursuit

I wanted to share with you an example that seems, on the face of it, to be vastly different from clinical trial RBM: Intelligent police-response systems. When I talk about decision science, what I refer to is taking all the available data and information, compiling it together, and looking at all of it holistically for better decision-making purposes. A new technology that’s available in the criminal justice system is something called the Beware system. The system is in use in Fresno, California and other police departments. It’s an electronic database which takes into account GPS coordinates, spatial distributions of localized criminal activity, as well as past track record of individuals involved in 9-1-1 calls. The system “searches, sorts and scores billions of commercial records in a matter of seconds-alerting responders to potentially deadly and dangerous situations while en route to, or at the location of a call.” Based on the software’s calculation of the factors, it assigns a ‘threat rating’ and a red, yellow, or green indicator for the officer.

dial

All of these data are critically important when appraising a police response situation. When using it as a guide, it’s important to understand the underlying risk assessment principles calculated by the Beware system: Not every address or location has an equivalent level of risk, and the system allows responding officers the opportunity to be prepared. Criminal recidivism refers to an individual’s preponderance to recommit crime after he or she has been involved in criminal acts in the past. So to say that past performance is an indicator of future behavior is an understatement.

Of course, detractors of the Beware system consider that it unfairly paints a picture of the situation before it can be properly appraised. Though individual citizens should be considered, in the eyes of the law, not guilty until proven otherwise – to turn a blind eye and dis-acknowledge available data is stupidity. Likewise, to not appraise your trial data using a risk-based approach with smart technologies is equivalent to not using information to make better decisions. Imagine if you have a particular study site that is continuously underperforming or producing more defects than other sites; Wouldn’t that be valid evidence to begin a course-correction for that site? Bayesian statistics would force you to acknowledge that the amassed evidence suggests a difference in performance at that site. It’s basic process improvement methodology. But you need to know what to fix first; and this usually goes wrong in the form of companies being unable to differentiate the signal from the noise. To not do it properly leads to a lot of organizations that I’ve seen expending a tremendous amount of resource and capital on trying to fix what actually isn’t the problem.

 

Given that ICH E6 has just undergone revision (r. 2) for the first time in 20 years, and the new revisions are deeply-steeped in risk-based monitoring, I’ll be connecting the dots between what you may be doing now and what you should be doing in my next article.

 

Ben Locwin, PhD, MBA, MS, is President at Healthcare Science Advisors and is an author of a wide variety of scientific articles for books and magazines. He is an expert contact for the American Association of Pharmaceutical Scientists (AAPS), a committee member for the American Statistical Association (ASA), and also a consultant for many industries including biological sciences, pharmaceutical, psychological, and academic. Follow him at @BenLocwin.