Patient sub-population effects: The regulator’s dilemma

At the DIA/FDA statistics workshop last week, amongst the many and varied interesting presentations, the one that most caught my eye was Dr Armin Koch’s talk on examples of patient sub-populations. (Dr Koch, of the Hannover Medical School, is a member of the Scientific Advice Working Party and Biostatistics Working Party of the European Medicines Agency).

Molecular MedicineUnderstandably, regulators are very reluctant to accept claims that a drug is effective in a subgroup if that subgroup was only identified after the trial was complete. The European regulatory guidance states:

“A specific claim of a beneficial effect in a particular subgroup requires pre-specification of the corresponding null hypothesis and an appropriate confirmatory analysis strategy. It is highly unlikely that claims based on subgroup analyses would be accepted in the absence of a significant effect for the overall study population.”

The reason of course is very simple – if we study enough possible subgroups (a.k.a. “torturing the data”) we are bound to find a subgroup that has, by chance, shown an effect. Many times such subgroup effects evaporate in light of subsequent trials – as Stephen Senn says, “enjoy your subgroup while you have it – you may never encounter it again”. Koch cited the PRIEST 1 trial that suggested efficacy of Amlodipine in a subgroup of  a non-ischemic patient, followed by the PRIEST 2 trial that failed to replicate the benefit, as an example of this.

Of course, statistical adjustments for the multiplicity of analyses can be made, but how do we know how many analyses were actually made, particularly if the data has been made public and different teams perform analyses? If the teams that don’t find anything don’t report their analyses, how can one adjust results from the teams that do find a subgroup effect and do publish?

But… Koch then related the results of the CAPRIE trial in clopidogrel ( – the overall results were that clopidogrel, compared to aspirin, was both effective in reducing the risk of ischemic stroke, myocardial infarction or vascular death in a patient with diabetic nephropathy, and safe. But subsequent subgroup analysis showed that not only was the effect limited to men (there was absolutely no effect in women) but also the safety was limited to men and there were increased cardiovascular risks for women.

And… the subsequent PLATO trial that compared ticagrelor to clopidogrel showed superiority but had an unanticipated subgroup effect. When the results were analysed by region, ticagrelor was (highly) superior in Europe but inferior in North America. This posed a considerable quandary to regulators (only approve ticagrelor in Europe? What if a patient then flies to North America – should they then stop taking it?) until it was deduced that it was due to an interaction with a high dose of aspirin that was commonly prescribed in the US, but hardly ever in Europe.

So if the regulators were prepared to act on the basis of these “after the event” analyses, why not others? It’s well accepted that many drugs don’t work for everybody, and the proportion of ‘responders’ varies between 25% and 60% depending on disease area (Spear at al. TRENDS in Molecular Medicine Vol 7 No. 5 May 2001). So shouldn’t we be looking to identify these subgroups regardless, even though there is ‘multiplicity’ risk? Analytical approaches to mining large data sets using machine learning techniques make this much more practicable, and they include methods for estimating the ‘false discovery rate’. Perhaps it’s time statisticians became more comfortable with them, because we are going to find subgroup effects that are just as compelling as those in Koch’s examples.

Koch concludes that there is too much worry about type I error and that we must look into subgroups as there may be important data there. I agree, the risk of over analysis needs to be taken into account, but there are risks with not looking too and a means of balancing the two needs to be developed.

Tom Parke

Tom Parke

Tom Parke has been working at Tessella for over ten years. For a large part of that time he has ...

© Copyright 2017 Tessella
All rights reserved