Email updates

Keep up to date with the latest news and content from Trials and BioMed Central.

Open Access Commentary

Against pragmatism: on efficacy, effectiveness and the real world

David M Kent* and Georgios Kitsios

Author Affiliations

Institute for Clinical Research and Health Policy Studies and Department of Medicine, Tufts Medical Center, Tufts University School of Medicine, Boston, USA

For all author emails, please log on.

Trials 2009, 10:48  doi:10.1186/1745-6215-10-48

The electronic version of this article is the complete one and can be found online at: http://www.trialsjournal.com/content/10/1/48


Received:1 June 2009
Accepted:6 July 2009
Published:6 July 2009

© 2009 Kent and Kitsios; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Explanatory and pragmatic trials represent ends of a continuum of attitudes about clinical trial design. Recent literature argues that pragmatic trials are more informative about clinical care in the real world. Although there is place for more pragmatic studies to inform clinical practice and health policy decision-making, we are concerned that it is generally under-appreciated that extrapolating the results of broadly inclusive pragmatic trials to the care of real patients may often be as problematic as extrapolating the results of narrowly focused explanatory or efficacy trials. Simplistic interpretation of pragmatic trials runs the risk of driving harmful policies.

Background

Determining the 'true' treatment effect of a given therapy is a bit like determining the 'true' weight of a liter of water. Those who answer that a liter of water weighs a kilogram are either assuming an implicit 'on planet earth, at sea level, at four degrees Celsius' or confusing the intrinsic property of mass with the extrinsic property of weight. Like weight, treatment effect is an extrinsic property, emerging only through an interaction between the intervention, the patient, and the circumstances in which it is being measured [1]. Adjust the context and a different effect emerges – just as a liter of water weighs a little over a third of a kilogram on Mars.

Treweek and Zwarenstein present a narrative review on pragmatic trials highlighting the importance of considering the context of a trial when interpreting its result [2]. Efficacy (or 'explanatory') trials, they argue, with their emphasis on ideal patients in ideal settings, may not generalize to routine care and are therefore not especially informative regarding practical clinical and policy decisions. What is needed, they assert, is a more pragmatic attitude toward trial design, such that trials include all patients who might be considered for a therapy in the real world, so that the trial result can be applicable to practical decisions. Framed in this way, who can argue that trials should not be more pragmatic? Indeed, the argument has been gaining momentum, and we see a welcome shift to a more pragmatic attitude [3,4]. Yet, on the cusp of the era of comparative effectiveness, it is essential that we examine critically the ability of pragmatic trials to make up for this well-appreciated limitation of efficacy trials and provide information directly applicable to the real world.

Discussion

First, let us recognize that if the purpose of a trial is to examine whether a treatment for a given condition works at all or not (perhaps the most common reason for a trial), then designing a trial toward the explanatory/efficacy end of the continuum is probably a wise decision. The US Food and Drug Administration guidance on this gets to the heart of the issue – a trial may yield null results for a myriad of reasons [5]: the therapy may just not work, or the therapy may work in more carefully diagnosed patients, or those better selected in terms of disease severity, or in those with less co-morbidities or better compliance, or with closer monitoring, or in settings with a higher commitment to and/or more experience with the intervention. The therapy may even work in the enrolled patients in the trial setting, but measurement error obscured the treatment signal or inadequate sample size drowned it in statistical noise. Thus, a null pragmatic trial provides little information about whether our treatment has some potential value. On the other hand, a null efficacy trial, performed under the most favorable possible circumstances, can give us definitive information that a therapy is not of value.

Treweek and Zwarenstein concede that proof of efficacy may be an important pre-requisite for an effectiveness trial [2]. The question then becomes how does one interpret the results of a pragmatic trial in the context of a previously favorable efficacy trial? If only pragmatic trials have implications for 'real world' settings, as the authors suggest, then the results of these trials presumably trump or nullify the results of efficacy trials. Thus, if a broadly inclusive effectiveness trial yields null results, policy decisions should reflect the fact that the treatment does not work in the real world, even when efficacy was demonstrated in a more 'explanatory' trial. Another view, the view that we hold, is that when effectiveness and efficacy trials yield discordant results, the therapy works in some situations and not in others. Physician judgment is then necessary to make treatment decisions based on the degree to which specific patients and settings in the real world correspond to those in the efficacy trial. Indeed, this is not so terribly different from what we might have recommended before the pragmatic trial; the negative effectiveness trial provides important information about how cautious physicians need to be in generalizing the efficacy results.

So, if null efficacy trials are more informative than null effectiveness trials, surely positive pragmatic effectiveness trials must be more informative than positive efficacy trials? However, consider also the potential of positive pragmatic trials to elevate low-value or even harmful care to the status of evidence-based medicine. A broadly inclusive trial is likely to include many patient groups where there is little likelihood of any benefit. For example, we would not be at all persuaded to add a new oral hypoglycemic to the standard regimen in an 85-year-old woman with recent onset diabetes, Class II heart failure and a glycated hemoglobin of 8, who is on five other medications, even if the patient meets the inclusion criteria of a well-conducted effectiveness trial which showed overall benefit – simply because the potential for benefit is too small to warrant the risks of polypharmacy. Similarly, for any treatment with a narrow therapeutic index, we would withhold therapy from patients highly likely to be poorly adherent, even if such patients were included in a positive pragmatic trial.

The issues of extrapolating trial results to patients are still more complex than in these relatively straightforward examples, since heterogeneity of baseline risk can result in situations in which the average result of a trial may be misleading even about the typical patient in the trial itself [6,7]. Unless accompanied by more careful analytic approaches [7-10], broadly inclusive pragmatic trials (which increase the baseline heterogeneity of outcome risk, competing risk, and risk of treatment-related harm) have the potential to exacerbate, rather than reduce, the difficulty of applying clinical trial results to individual patients [11].

The last point we wish to make concerns the increased attention with which care is delivered in the context of a trial. Finding a signal of efficacy can be difficult and additional resources are often used to ensure that performance of therapeutic procedures or adherence to a medical regimen is especially meticulous. Unfortunately, helping patients in the real world is also often difficult and frequently requires meticulous care, and some new therapies may require substantial organizational or structural changes to be effective. While ultra-meticulous care in trials can give misleading results (that are potentially harmful to patients if generalized to inappropriate settings), 'pragmatically' testing such new therapies in resource-challenged environments may often be a recipe – or an excuse – for inertia; it sets so-called 'usual care', rather than the 'best attainable and sustainable' care [12], as the aspirational standard.

Conclusion

This is a very partial account of the limitations of pragmatic trials. More issues will undoubtedly be revealed as the pendulum swings in the pragmatic direction and we gain more experience with this type of trial. The main point we wish to emphasize is that while both types of trials yield useful information, pragmatic trials do not provide a more accurate measure of the 'true' treatment effect, since the concept of a true effect is fundamentally illusory. While extrapolating the results of efficacy trials to the care of individual patients in the real world can be problematic, and requires careful physician judgment and decision-making, the same is unfortunately true for the results of effectiveness trials. Unless more attention is paid to these under-appreciated limitations, pragmatic trials run the risk of driving harmful policies.

Competing interests

DMK was partially supported by a grant from the NIH (NIH/NCRR 1UL1 RR025752). GK is Pfizer-Tufts Medical Center Research Fellow in Clinical Research.

Authors' contributions

DMK and GK discussed the concepts in the manuscript. DMK wrote the first draft. Both authors revised the manuscript for important content and approved the final manuscript.

References

  1. Avorn J: Powerful Medicines: The Benefits, Risks, and Costs of Prescription Drugs. NY: Knopf; 2004.

  2. Treweek S, Zwarenstein M: Making trials matter: pragmatic and explanatory trials and the problem of applicability.

    Trials 2009, 10:37.

    (3 June 2009)

    PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  3. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K: A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.

    CMAJ 2009, 180:E47-E57. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  4. Tunis SR, Stryer DB, Clancy CM: Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.

    JAMA 2003, 290:1624-1632. PubMed Abstract | Publisher Full Text OpenURL

  5. US Food and Drug Administration: Guidance for Institutional Review Boards and Clinical Investigators 1998 Update.

    2009.

    7-1-2009

  6. Rothwell PM: Can overall results of clinical trials be applied to all patients?

    Lancet 1995, 345:1616-1619. PubMed Abstract | Publisher Full Text OpenURL

  7. Kent DM, Hayward RA: Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification.

    JAMA 2007, 298:1209-1212. PubMed Abstract | Publisher Full Text OpenURL

  8. Hayward RA, Kent DM, Vijan S, Hofer TP: Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis.

    BMC Med Res Methodol 2006, 6:18. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  9. Rothwell PM, Mehta Z, Howard SC, Gutnikov SA, Warlow CP: Treating individuals 3: from subgroups to individuals: general principles and the example of carotid endarterectomy.

    Lancet 2005, 365:256-265. PubMed Abstract | Publisher Full Text OpenURL

  10. Rothwell PM, Warlow CP: Prediction of benefit from carotid endarterectomy in individual patients: a risk-modelling study. European Carotid Surgery Trialists' Collaborative Group.

    Lancet 1999, 353:2105-2110. PubMed Abstract | Publisher Full Text OpenURL

  11. Kent DM, Alsheikh-Ali A, Hayward RA: Competing risk and heterogeneity of treatment effect in clinical trials.

    Trials 2008, 9:30. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  12. Bloom BR: The highest attainable standard: ethical issues in AIDS vaccines.

    Science 1998, 279:186-188. PubMed Abstract | Publisher Full Text OpenURL