Open Access Open Badges Research

"Flogging dead horses": evaluating when have clinical trials achieved sufficiency and stability? A case study in cardiac rehabilitation

Louise Dent1*, Rod Taylor2, Kate Jolly3 and James Raftery4

Author Affiliations

1 University of Southampton Clinical Trials Unit MP131, University of Southampton, SO16 6YD, UK

2 PenTAG, Peninsula Medical School, Exeter, Devon, UK

3 Department of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham, Warwickshire, UK

4 NIHR Evaluation, Trials and Studies Coordinating Centre, University of Southampton, Southampton, SO16 7NS, UK

For all author emails, please log on.

Trials 2011, 12:83  doi:10.1186/1745-6215-12-83

Published: 21 March 2011



Most systematic reviews conclude that another clinical trial is needed. Measures of sufficiency and stability may indicate whether this is true.

Objectives: To show how evidence accumulated on centre-based versus home-based cardiac rehabilitation, including estimates of sufficiency and stability


Systematic reviews of clinical trials of home versus centre-based cardiac rehabilitation were used to develop a cumulative meta-analysis over time. We calculated the standardised mean difference (SMD) in effect, confidence intervals and indicators of sufficiency and stability. Sufficiency refers to whether the meta-analytic database adequately demonstrates that an intervention works - is statistically superior to another. It does this by assessing the number of studies with null results that would be required to make the meta-analytic effect non-statistically significant. Stability refers to whether the direction and size of the effect is stable as new studies are added to the meta-analysis.


The standardised mean effect difference reduced over fourteen comparisons from a non-significant difference favouring home-based cardiac rehabilitation to a very small difference favouring hospital (SMD -0.10, 95% CI -0.32 to 0.13). This difference did not reach the sufficiency threshold (failsafe ratio 0.039 < 1) but did achieve the criteria for stability (cumulative slope 0.003 < 0.005).


The evidence points to a relatively small effect difference which was stable but not sufficient in terms of the suggested thresholds. Sufficiency should arguably be based on substantive significance and decided by patients. Research on patient preferences should be the priority. Sufficiency and stability measures are useful tools that need to be tested in further case studies.