Skip to main content

Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane review groups

Abstract

Background

Missing outcome data or the inconsistent reporting of outcome data in clinical research can affect the quality of evidence within a systematic review. A potential solution is an agreed standardized set of outcomes known as a core outcome set (COS) to be measured in all studies for a specific condition. We investigated the amount of missing patient data for primary outcomes in Cochrane systematic reviews, and surveyed the Co-ordinating Editors of Cochrane Review Groups (CRGs) on issues related to the standardization of outcomes in their CRG’s reviews. These groups are responsible for the more than 7,000 protocols and full versions of Cochrane Reviews that are currently available, and the several hundred new reviews published each year, presenting the world’s largest collection of standardized systematic reviews in health care.

Methods

Using an unselected cohort of Cochrane Reviews, we calculated and presented the percentage of missing patient data for the primary outcome measure chosen for each review published by each CRG. We also surveyed the CRG Co-ordinating Editors to see what their policies are with regards to outcome selection and outcomes to include in the Summary of Finding (SoF) tables in their Cochrane Reviews. They were also asked to list the main advantages and challenges of standardizing outcomes across all reviews within their CRG.

Results

In one fifth of the 283 reviews in the sample, more than 50% of the patient data for the primary outcome was missing. Responses to the survey were received from 90% of Co-ordinating Editors. Thirty-six percent of CRGs have a centralized policy regarding which outcomes to include in the SoF table and 73% of Co-ordinating Editors thought that a COS for effectiveness trials should be used routinely for a SoF table.

Conclusions

The reliability of systematic reviews, in particular meta-analyses they contain, can be improved if more attention is paid to missing outcome data. The availability of COSs for specific health conditions might help with this and the concept has support from the majority of Co-ordinating Editors in CRGs.

Peer Review reports

Background

Cochrane Reviews are systematic reviews of research in human health care and health policy, and are internationally recognized as a high quality source of evidence for decision making. They bring together research evidence on the effects of healthcare interventions or the accuracy of diagnostic tests into the world’s largest, standardized collection of systematic reviews, with more than 5,000 full reviews and protocols for another 2,000 published to date. Outcome selection in systematic reviews needs to be relevant to patients, clinicians and policy-makers if the findings of a review are to influence practice and future research. Inconsistent choice of outcome measures in clinical trials means that many meta-analyses are unable to include data from all the relevant studies. For example, the five most accessed Cochrane Reviews in 2009, together with the top cited review in that year, all described inconsistencies in the outcomes reported in eligible trials, which hampered the ability of the reviewers to resolve the uncertainties about health care that they set out to tackle [1]. An additional problem that can affect the quality of evidence within a systematic review is that of missing outcome data from all eligible trials. Missing outcome data in trials can occur for a number of reasons. Participant data can be missing as a result of attrition or as a result of the selective non-reporting of an outcome in a study. Recent research has shown that outcome reporting bias (ORB), that is, results-based selection for publication of a subset of the original outcome variables, is also a major problem in randomized trials [2]. The Outcome Reporting Bias in Trials (ORBIT) study of the impact of ORB in randomized trials on the results of Cochrane Reviews found that ORB is an ‘under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews’ [3]. This study found that more than half (157/283 (55%)) of the reviews did not include full data for the review’s primary outcome of interest from all eligible trials. The median amount of review primary outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. In this same study, an assessment of the impact of ORB was undertaken for reviews with a single meta-analysis of the primary review outcome where there was missing outcome data for a whole study that was known or suspected of being measured but not reported. Of the 25 reviews that met these criteria, a sensitivity analysis revealed that the conclusions to eight of these reviews were not robust to ORB—that is, the treatment effect estimate changed from a significant result favoring treatment to a non-significant result. For all the 25 reviews assessed, the median percentage change in the treatment effect estimates after the adjustment based on the sensitivity analysis was 39% (interquartile range 18% to 67%) [3].

One way to reduce the amount of missing outcome data by ensuring that eligible trials contribute usable information is the definition and implementation of an agreed minimum set of standardized outcomes, to be measured and reported in all trials for a particular disease or condition, referred to as a ‘Core Outcome Set’ (COS) [4, 5]. Furthermore, some Cochrane Review Groups (CRGs) now include a Summary of Findings (SoF) table in their reviews [6]. The SoF table presents the results of the review for up to seven outcomes that are important to patients and aims to improve the understanding and retrieval of key findings.

In this paper, we present results for missing data in single review primary outcomes from the Cochrane Reviews included in the ORBIT study. We also report the results of a survey of the Co-ordinating Editors of CRGs, undertaken to obtain their views on the benefits and challenges of core outcome sets and how they might improve the quality of the reviews in their CRG. We were also interested in establishing whether the amount of missing data from a systematic review was influenced by whether or not a CRG had a centralized policy on outcome selection. The Co-ordinating Editors have responsibility for overseeing the work of their CRG and deciding on the publication of each of the Cochrane Reviews from within their CRG.

Methods

Missing data

The missing study data reported here was part of the larger ORBIT project to estimate the prevalence and effect of outcome reporting bias in clinical trials on the primary outcomes of systematic reviews from three issues of The Cochrane Library: issue 4, 2006; issue 1, 2007, and issue 2, 2007. For the 283 reviews in this study, the number of participants in each of the eligible studies was extracted and the percentage of missing participant data was calculated as the number of participants missing from the meta-analyses of the single review primary outcome (as a result of the non-reporting of outcome data in a study), as defined by the lead reviewer. Data from reviews containing at least five eligible studies are presented.

Survey of Co-ordinating Editors of Cochrane Review Groups

With the exception of the Cochrane Methodology Review Group (which focuses on topics relating to methodology, rather than healthcare interventions and tests), the Co-ordinating Editors for the 50 CRGs (as of 2011) were contacted and asked to participate in a short online survey with regards to how a COS might improve the quality of the reviews in their CRG. The survey was constructed using the online survey software, SurveyGizmo, and was open for responses between 1 November 2011 and 31 July 2012. Non-responders were contacted periodically if a response to the survey was not received. As part of the survey, Co-ordinating Editors were provided with a brief synopsis of how COSs could make it easier for their review authors to identify appropriate outcomes for Cochrane Reviews, and present their findings clearly and succinctly. They were also presented with the missing data plot for all CRGs and made aware of the data relating to their CRG. Co-ordinating Editors were asked whether or not they had any centralized CRG policies relating to the outcomes to include in their reviews (including the distinction between primary and secondary outcomes) and outcomes to include in the SoF tables (providing one is included in the review), or whether these decisions are left to the authors of the review. Patterns in the amount of missing participant data between CRGs with and without a centralized policy on outcome selection were examined. Finally, Co-ordinating Editors were asked if they thought that a COS for effectiveness trials should be used routinely for a SoF table and what they thought were the main advantages and challenges of standardizing outcomes across all reviews in a particular condition covered by their CRG. All advantages and challenges were independently reviewed by two authors (JJK and PRW) and categorized into common topics. Any discrepancies were resolved through discussion.

Results

Of the 283 reviews included in the ORBIT study, 143, from 40 CRGs, were included here, that is, reviews that contained at least 5 eligible studies (Figure 1). Forty-one (29%) reviews contained no missing patient data, while 26 (18%) had more than 50% of the patient participant data on the single primary outcome missing from the review. Accountable reasons for missing data included the difficulty to measure certain outcomes, the diverse range of possible outcomes to measure and disputes around the importance and best outcomes to measure which are both robust and reliable for use in systematic reviews. A summary of the relevant responses with regards to the reasons for missing outcome data that was provided by the Co-ordinating Editors is provided in Table 1.

Figure 1
figure 1

The percentage of missing patient data for the primary outcome for each review that had at least five included studies (reviews first published in Issue 4, 2006 to Issue 2, 2007).

Table 1 Responses from Cochrane review group co-editors who provided reasons for missing data in their reviews

Responses to the survey were received from Co-ordinating Editors for 45 (90%) of the 50 CRGs. Fourteen Co-ordinating Editors (31%) had previously been involved in the development of a COS and 16 (36%) were aware of other work that has been done or is ongoing to develop a COS for conditions relevant to their CRG. Overall, 16 (36%) CRGs have a centralized policy for outcome selection while the remaining 29 leave outcome selection and the relative importance of outcomes chosen to the discretion of the authors of the relevant review. From Figure 1, there did not appear to be any association between the existence of a CRG policy for outcome selection and the amount of missing data in their reviews. Similarly, 16 (36%) CRGs have a centralized policy regarding which outcomes to include in the SoF table (14 of which had a centralized policy for outcome selection), while 25 (56%) left this decision to the review authors and a further four (9%) are not yet including the SoF table in their reviews. All four CRGs where the SoF has not yet been adopted had no CRG policy for outcome selection but two of the four Co-ordinating Editors were aware of ongoing COS development work for conditions covered by their CRG.

A total of 33 (73%) Co-ordinating Editors thought that a COS for effectiveness trials should be used routinely for a SoF table. Reasons for this were provided by 23 Co-ordinating Editors and included relevant outcomes measured and reported (10), comparability of outcomes (6), improved interpretability of outcomes (4), standardization of outcomes (2) and reduces risk of bias (1). Ten of the 12 Co-ordinating Editors who did not think a COS should be used in the SoF table thought that the scope and diversity of their reviews would not allow a COS to be applied across the reviews for their CRG.

The Co-ordinating Editors listed 100 advantages and 82 challenges associated with standardizing outcomes across reviews in their CRG (Table 2). The most common advantage for having a COS was that systematic reviews or meta-analyses would benefit from the standardization of outcomes. The main challenges related to the process needed to develop a COS, decisions around when a COS should be applied and the need to persuade potential authors of Cochrane Reviews to use the COS.

Table 2 Advantages and challenges of standardizing outcomes across all reviews for a particular condition

Discussion

There is considerable variability within and between CRGs in the amount of data that are missing for the primary outcomes chosen for a review. However, there did not appear to be any association between the existence of a CRG policy for outcome selection and the amount of missing data in their reviews (Figure 1). Furthermore, the survey of Co-ordinating Editors revealed a clear lack of clarity and consensus on the outcomes to measure within the specialties covered by CRGs.

The set of reviews that are reported in this study are slightly dated. However, the reviews used came from the earlier ORBIT study which started in 2006. The missing participant data from the primary outcome from each review was extracted during the ORBIT study, although this remained unpublished data. Presenting the data here accompanied the Cochrane Co-Editor survey data well and demonstrated the importance of COSs from a reviewer perspective. A recent inspection of a random sample of 20 of the 283 reviews used in ORBIT revealed that only 25% (5/20) of these reviews identified new studies for inclusion in the most updated version of the review – this meant on average, in at least 75% of cases, the percent missing data from the reviews used in 2006/2007 still remains unchanged.

The suggestion that a COS should be routinely used as part of a SoF table was well received, with most Co-ordinating Editors believing that this would be beneficial. The scope and diversity of reviews within some CRGs is likely to be a barrier to the application of a COS across the reviews for the particular group. It was noted by some of the Co-ordinating Editors in these CRGs that outcome specification for particular conditions would be difficult without further refinement to specific populations and interventions (for example, pharmacological versus non-pharmacological treatments).

The main advantage of a COS as suggested by the majority of Co-ordinating Editors was that the concept of a COS would benefit the systematic review process by increasing the amount of usable information for use in a meta-analysis. The challenges that were identified appear to reflect a lack of experience with COS, which could be overcome with training or support materials (for example on how to develop a COS and how to encourage others to use it). To help with this, the Core Outcome Measures in Effectiveness Trials (COMET) Initiative brings together researchers interested in the development and application of COS (http://www.comet-initiative.org). COMET aims to collate and stimulate relevant resources (both applied and methodological), facilitate the exchange of ideas and information, and foster methodological research. The Initiative has three strategic goals: to increase the number of COSs developed using evidence-based methods; to increase their impact on the quality of research by raising awareness and increasing their use; and to establish methods for the development of COS [5]. This will include work with Co-ordinating Editors and others in The Cochrane Collaboration to facilitate activities within CRGs to develop COS and to implement these within Cochrane Reviews, including the provision of training workshops and the preparation of training materials. This work should also have benefits for systematic reviewers outside of The Cochrane Collaboration, who wish to maximize the impact of COS on the quality of reviews and the research studies that they bring together.

Conclusions

The reliability of systematic reviews, in particular meta-analyses they contain, can be improved if more attention is paid to missing outcome data. The availability of COSs for specific health conditions might help with this and the concept has support from the majority of Co-ordinating Editors in CRGs.

Abbreviations

COS:

core outcome set

CRG:

Cochrane Review Group

ORB:

outcome reporting bias

ORBIT:

Outcome Reporting Bias in Trials

SoF:

summary of findings.

References

  1. Tovey D: Impact of Cochrane Reviews [editorial], The Cochrane Library. 2010, http://www.thecochranelibrary.com/details/editorial/756937/The-Impact-of-Cochrane-Reviews-by-Dr-David-Tovey.html (accessed 21 August 2012)

    Google Scholar 

  2. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008, 3 (8): e3081-10.1371/journal.pone.0003081.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR: The impact of outcome reporting bias on systematic reviews. BMJ. 2010, 340: c365-10.1136/bmj.c365.

    Article  PubMed  Google Scholar 

  4. Clarke M: Standardising outcomes for clinical trials and systematic reviews. Trials. 2007, 8: 39-10.1186/1745-6215-8-39.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Williamson PR, Altman D, Blazeby J, Clarke M, Gargon E: Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012, 17: 1-2.

    Article  PubMed  Google Scholar 

  6. Rosenbaum SE, Glenton C, Oxman AD: Summary of findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010, 63: 620-626. 10.1016/j.jclinepi.2009.12.014.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We thank all the Cochrane Review Group Co-Editors who provided responses to our survey.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jamie J Kirkham.

Additional information

Competing interests

PRW chairs the Management Group of the COMET Initiative. MC is also a member of the COMET Management Group and EG is the COMET Project Co-ordinator. JJK has no conflict of interest to declare.

Authors’ contributions

JJK, EG, MC and PRW designed the survey of co-editors. The electronic version of the survey was set up by JJK. The amount of missing data from each review was extracted by JJK, who produced the missing data plots for each CRG. JJK, EG and MC sent invitations and reminders to Co-ordinating Editors to complete the survey. JJK prepared the initial manuscript. PRW and MC were involved in the revision of this manuscript. PRW conceived the idea for the study. All authors commented on and approved the final manuscript before submission.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Kirkham, J.J., Gargon, E., Clarke, M. et al. Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane review groups. Trials 14, 21 (2013). https://doi.org/10.1186/1745-6215-14-21

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1745-6215-14-21

Keywords