One neat trick for instantly improving medical trials (scientists hate this!)
One neat trick for instantly improving medical trials (scientists hate this!)
By justifying exclusion criteria and explaining the implications of any exclusion criteria you choose
First of all, I have to apologize for the title, but not enough to change it. Second, today’s essay is inspired by my own efforts to figure out my own drug trial both in animals and in people. When you start getting in the specifics of how a trial is designed, you end up with decisions that can only be justified to the FDA and God, so you realize how much of an art trial design really is. It all looks so clear from a 10,000 foot view…
I have a big problem with exclusion criteria in the methods sections of paper on medical interventions as they’re used today. While they have utility as tools to make running trials on interventions easier or more straightforward, they’re too often used to run different trials altogether than what’s promised in the abstract. In other words, they become “redefinition criteria”, rather than exclusion criteria.
Let me explain. Exclusion criteria, to me, are criteria you use to exclude anyone (or anything) that doesn’t fit in a pre-defined notion of what you’re attempting to select for. For example, if I was throwing an 80s party, I might have an exclusion criteria of no shorts below the knee. Short shorts are a defining characteristic of the 80s, so anyone with long shorts doesn’t get allowed in.
If, on the other hand, I was throwing an 80s party, and I made an exclusion criteria of “No ugly men”, that would not be a valid exclusion criteria. Ugly men were present at parties in the 80s, as the movie Revenge of the Nerds informs us. Any extrapolations that I or anyone made from this unreasonably attractive 80s party would not be generalizable, including the effect of any interventions on the outcome of that party. So, “no ugly men” would actually be a redefinition criteria, in which I’d redefine the 80s aesthetic to be as attractive as I want it to be.
These exclusion criteria are obviously valid and invalid, respectively. But one could imagine an exclusion criteria that’s justifiable in some circumstances but not others. For instance, if I wanted my party to reflect the actual fabric quality of 80s fashion, I might have an exclusion criteria of actual vintage clothing only. That wouldn’t make sense if I simply wanted my party to reflect the aesthetics of the 80s, so I shouldn’t then apply that exclusion criteria.
However, if I do apply that exclusion criteria, I would then need to, first of all, justify this exclusion criteria explicitly. I would make sure the partygoers knew why I wanted to only include actual vintage clothing based on what conclusions I wanted to draw from the party. Second, I’d make sure that any extrapolations people draw from that party are made with this exclusion criteria in mind. For example, I would want to make sure that nobody thinks that all 80s clothing smelled musty or had patches on it. Instead, I’d want to make sure that they know this is simply an artifact of the fact that all the clothing at this party was 40 years old.
I may have beaten this metaphor to death. So let me explain using an actual example from a paper I saw floating around Twitter. This paper allegedly showed that intermittent fasting was as effective as dieting in young overweight women. They did using a normal dieting study, where the women were randomized to an intermittent fasting diet vs. a calorie-restricted diet, and then they compared weight loss. Simple enough. People on Twitter were using this to argue in favor of the effectiveness intermittent fasting.
However, if we go into the exclusion criteria section, we get, “Participants were non-smokers, not currently dieting or losing weight, with regular menstrual cycles and no evidence of hyperandrogenism or polycystic ovary syndrome, and no oral contraceptive use during the previous 6 months. They did not have high intakes of alcohol (428 units per week) or phytooestrogens, and were not suffering from diagnosed diabetes, cardiovascular disease, major psychiatric morbidity or cancer.”
Some of these exclusion criteria are pretty straightforward. I’d assume you wouldn’t want any participants who were dieting or losing weight because that makes it difficult to tell if your intervention worked. Also, you’d want to avoid any participants with conditions that make them predisposed to gain or lose weight, as that would make an apples-to-apples comparison across groups more difficult.
However, there are other exclusion criteria that I don’t understand at all. Why no oral contraceptive use? Why no smoking? Why no diabetes, given that this is an overweight subject group?
There might well be good reasons for these exclusions that I’m not aware of. Perhaps if I ran my own trial on dieting, I’d find that it’s impossible to have women on oral contraceptives in my groups. However, those reasons are not outlined here. And, perhaps more importantly, the implications of those restrictions are never discussed throughout the rest of the paper. Is it possible that a given woman on oral contraceptives would find it more or less difficult to lose weight with intermittent fasting? Who knows!
These unjustified exclusions can have real clinical implications, too. For example, most asthma studies exclude morbidly obese people, as morbidly obese asthma is notoriously resistant to treatment and there aren’t good explanations as to why. However, once asthma drugs are approved, they’re approved for all asthmatics equally. As a result, morbidly obese people get prescribed asthma drugs that were never tested on people like them [1].
If asthma drug studies had to justify why they were excluding morbidly obese people and had to discuss this fact in their studies and their marketing, morbidly obese patients and their doctors would be able to make more informed decisions about their health. This would also open up the market for asthma drugs that did include the morbidly obese in their trials and could be more relevant for their specific needs.
The opportunity is there to improve medical trials. We just need to justify exclusion criteria.
[1] This is literally going on today by the way. The FDA approved Tezpire as a breakthrough drug for asthma in December 2021. Tezspire excluded morbidly obese people from their efficacy trials. This fact is not mentioned anywhere in Tezspire’s labeling.