Reporting quality of abstracts in phase III clinical trials of systemic therapy in metastatic solid malignancies

Abstract

Background

Manuscript abstracts represent a critical source of information for oncology practitioners. Practitioners may utilize the information contained in abstracts as a basis for treatment decisions particularly when full-text articles are not accessible. In 2007, the Consolidated Standards of Reporting Trials (CONSORT) extension statement for abstracts provided a minimum list of elements that should be included in abstracts. In this study we evaluate the degree of adherence to these recommendations and accessibility of full text publications in oncology publications.

Methods

A systematic review of abstracts of randomized, controlled, phase III trials in metastatic solid malignancies published between January 2009 and December 2011 in PubMed, Medline, and Embase was completed. Abstracts were assigned a completeness score of 0–18 based on the number of CONSORT-recommended elements. Accessibility through open access was recorded.

Results

174 abstracts with data for 95,956 patients were reviewed. The median completeness score was 9 (range, 3–17). Open access to full text articles was available for 80 % of abstracts. The remaining 20 % (35 out of 174) had a median cost of 38 USD (range: $22–49.95). The least frequently reported elements were: trial design description (20 %), participant allocation method (13 %), blinding (24 %), trial enrollment status (22 %), registration and name of trial (26 %) and funding source (18 %). The most frequently reported elements were eligibility criteria (98 %), study interventions (100 %), and primary endpoint (87 %).

Conclusion

There is poor adherence to the CONSORT recommendations for abstract reporting in publications of randomized cancer clinical trials which could negatively impact clinical decision-making. Full-text articles are frequently available through open access.

Background

Clinicians utilize peer-reviewed publications to guide the management of patients with cancer. Clinical trial publications provide clinicians with an understanding of the specific patient population that was studied, the details of the experimental intervention, and the potential harms and benefits. Given the importance placed on clinical trials, guidelines, such as those generated by the Consolidated Standards of Reporting Trials (CONSORT) group, have been adopted by the International Committee of Medical Journal Editors to promote comprehensive and transparent reporting [1, 2].

An abstract is a concise summary of a clinical trial that is frequently utilized by clinicians to decide whether or not a study is of sufficient interest to read the full text. For many clinicians, such as those practicing in community-based settings or low and middle income countries, financial and/or information technology barriers limit access to full-text publications and the publically available abstract may be the only source of clinical trial information. However, several analyses have demonstrated inconsistencies in the data presented in abstracts, as compared with full-text manuscripts, that could potentially impact interpretation of the study results [3–9]. Similar inconsistencies have been reported when comparing abstracts presented at scientific meetings with the final study results published in peer-reviewed journals [10–14]. To address these discrepancies, the CONSORT group published an extension statement in 2008 concerning the quality of reporting in journal and conference abstracts. A 17 item checklist with explanations and examples was generated to establish uniform standards of abstract reporting [15]. In this analysis we assess the degree to which oncology clinical trial abstracts adhere to these reporting recommendations.

Methods

Literature search

Citations were reviewed from Medline, PubMed, and Embase published between 1 January 2009 and 31 December 2011 to identify eligible publications for analysis. ‘Metastatic” or ‘advanced” were the key words included in the search and were further refined using the following filters: ‘publication dates = 1/1/2009-12/31/2011’; ‘subjects = cancer’; ‘article type = clinical trial, phase III’; ‘language = English’; and ‘species = human.’ Publications that included patients with metastatic solid tumors receiving a pharmacologic intervention in a phase III clinical trial, as indicated by study authors, were included in the analysis. All other studies, including phase I and II studies, observational studies, letters, case reports, editorials, and studies exploring a device or behavioral intervention, were excluded. In the setting where multiple publications were found from the same trial, only the initial publication was used for the analysis. If the initial publication was published prior to 2009, neither that publication nor subsequent related publications were included in the analysis.

Data extraction

The CONSORT extension for abstract reporting outlines 17 general recommendations (Table 1) [15]. Additional recommendations are found in an accompanying paper entitled ‘CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.’ A multidisciplinary panel of five providers reviewed these recommendations and assembled a consensus list of 18 key reporting elements that were used in data collection and outlined in Table 1. The reporting element ‘Eligibility criteria for participants and the settings where the data were collected’ was divided into two separate reporting elements - ‘Eligibility criteria for participants’ and ‘The settings where the data was collected.’ Allocation concealment was evaluated based on specific criteria provided by the CONSORT committee's explanations and elaborations [15]. One CONSORT recommendation not included was ‘contact details for the corresponding author’ as this is specific to conference abstracts and no conference abstracts were reviewed in our analysis.

Table 1 Elements of abstract reporting

Eligible publications were evaluated for each of the 18 reporting elements in the abstract only. Additional data were extracted from either the abstract or full-text manuscript including the sample size, intervention type, use of placebo control, publication year, funding source, journal name and impact factor, and whether the primary study endpoint was met and specifically commented on in the discussion portion of the abstract. We also investigated whether the manuscript was available in full-text form though open access. If the full text was not available through open access, we recorded the cost (in US dollars) to access the full-text manuscript. Finally, we recorded whether the article was published in a journal that endorses the CONSORT statement.

The intervention types were classified as targeted therapy, chemotherapy, immunotherapy, or hormonal therapy. The primary endpoint was obtained from either the abstract or full manuscript. The results section was reviewed to determine if the primary endpoint was met. We compared regulatory approval dates derived from the Drugs@FDA Website with the manuscript publication date to assess whether the intervention was approved for another indication. Data extraction was done by two independent reviewers (SS and KN) and discrepancies resolved by a third reviewer (AA). Inter-rater agreement x kappa values ranged from 0.66 to 1.00 with a mean ± standard deviation of 0.88 ± 0.10.

Statistical analysis

The primary objective of this systematic review was to describe the quality of abstract reporting in phase III randomized controlled trials of systemic therapy for metastatic solid tumors in the context of the CONSORT abstract extension statement. Quality of abstract reporting was defined by the ‘completeness score’, calculated as the total number of CONSORT elements (Table 1) reported in each abstract. Each abstract received a score of 0–18. The secondary objective was to determine which clinical trial characteristics were associated with the ‘completeness score.’ We constructed linear models to assess the unadjusted association between the completeness score and covariates in the abstracts. This facilitated the assessment of categorical and continuous covariates in the same model structure. An exploratory objective was to describe the accessibility of full-text manuscripts through open access. All analyses were performed using Stata version 13.1 (College Station, TX, USA) using two-sided p values and p <0.05 considered as significant.

Results and discussion

Literature search

Of the 589 potentially relevant publications identified in our search, 174 publications met the eligibility criteria for inclusion in the analysis. The reasons for study inclusion and exclusion are detailed in Fig. 1.

Fig. 1

Flowchart of screening of publications included in the analysis

The characteristics of the publications included in the analysis are detailed in Table 2. A total of 95,956 patients were involved in the trials included in this review with THE median sample size being 421 (range 19–4,312). Breast cancer was the most common cancer subtype explored (21 %) and chemotherapy was the most common intervention (48 %). The majority of studies were supported by industry.

Table 2 Characteristics of studies included in analysis (n = 174)

Abstract reporting

The proportion of abstracts reporting each of the 18 reporting elements is shown in Table 3. The majority of the abstracts did not provide a specific description of the trial design (20 %) or the setting in which the trial took place (14 %). The method by which participants were assigned to interventions to ensure concealment was only reported in 13 % of abstracts and masking or blinding of the group allocation was only reported in 24 % of abstracts. Most abstracts did not report the time period over which the primary outcome was assessed (17 %) or how many patients were analyzed in each group (26 %).

Table 3 Proportion of articles addressing each of the 18 abstract reporting elements (n = 174)

Some reporting elements were applied more consistently across abstracts. All of the abstracts analyzed reported the intervention intended for each study group (100 %). The objective or hypothesis was reported in 95 % of abstracts and the primary outcome was reported in 87 % of abstracts. Summary statistics for the outcome for each group (87 %) and conclusions based on the results reported in the abstract (95 %) were reported consistently. Important or unexpected adverse events were reported in 74 % of abstracts.

Abstract reporting completeness scores

The mean ± standard deviation completeness score was 9.2 ± 2.7 (range 3–17; Fig. 2). Among 174 abstracts analyzed, 144 (83 %) were published in journals that endorse the CONSORT recommendations. There was no significant difference in completeness scores between abstracts published in journals that do (9.3 ± 2.8), and do not (8.7 ± 2.1) endorse the CONSORT recommendations (p = 0.241). There were also no significant differences in the mean completeness scores for placebo-controlled trials, treatment type or funding source (see Table 4). We did find differences (p = 0.014) in the mean completeness score for studies that had interventions approved for another indication (8.9 ± 2.6) having lower completeness scores than for studies with interventions not approved for another indication (10.1 ± 2.9). If the trial met the efficacy endpoint, the mean completeness score was higher (9.8 ± 3.0) than if the trial did not meet the efficacy endpoint (8.8 ± 2.4) (p = 0.018). We also found a linear trend with increasing completeness scores based on the year of publication across 2009, 2010 and 2011 (p = 0.039).

Fig. 2

Distribution of abstract reporting completeness scores (n = 174). The completeness score was derived by adding each of the 18 abstract reporting elements addressed in each abstract

Table 4 Publication characteristics associated with abstract reporting completeness score

Accessibility of full-text manuscripts through open access

Of the 174 abstracts reviewed for our analysis, the accompanying full text of the manuscript was available through open access in 80 % (139/174). There was no significant difference in the mean completeness scores for publications that were available free of charge versus those that required purchase (9.4 ± 2.8 versus 8.7 ± 2.3 respectively; p = 0.210). The median cost to purchase an article not available through open access was 38 US dollars (range: $22–$49.95).

Conclusion

An abstract is a snapshot of the overall design and results of a clinical trial and in many circumstances may be the only portion of a manuscript read by a clinician due to time constraints and barriers to accessing full-text publications. Therefore, sufficiently detailed and transparent reporting of oncology clinical trial abstracts is critical. Several guidelines have been developed, including those from the CONSORT group, in an effort to standardize the quality of abstract reporting [1, 2, 4, 15]. However, despite the 2008 CONSORT extension statement for reporting randomized controlled trials in journal and conference abstracts, there is significant heterogeneity in reporting of individual elements as demonstrated by our analysis and that of others [16–23].

Variability in reporting randomized controlled trials in journal abstracts may be secondary to several factors. Authors may be unaware of the CONSORT extension recommendations and journals may not require or enforce compliance to these guidelines. The CONSORT extension guidelines are not oncology specific and emphasis on certain reporting elements may be different for various disciplines. Limitations on word count for abstracts may also impact the ability of authors to comply with the CONSORT recommendations.

Notably, our unadjusted analyses demonstrated that most analyzed variables, including journal endorsement of CONSORT recommendations, placebo controlled trials, treatment type, or funding source, were not associated with completeness of abstract reporting. This suggests that suboptimal reporting is not limited to subsets of journals but rather a generalized issue. Interestingly, we reported that completeness scores have been linearly increasing over time as publications from 2011 had higher completeness scores. Our inclusion of publications spanning only a 3 year time interval (2009–2011), however, limits the interpretation of this finding.

There are several prior analyses addressing the quality of abstract reporting in randomized clinical trials but the majority of these have not evaluated oncology trials. However, during the preparation of this manuscript, Ghimire et al. published an analysis of oncology randomized controlled trials over a three-year period both prior to and after adoption of the CONSORT extension statement for abstract reporting. Substantial heterogeneity and selectivity in the quality of abstract reporting was demonstrated with only half of the CONSORT recommendations consistently being reported after the adoption of the CONSORT extension statement [24]. Our analysis extends these findings in several ways. We investigated the potential association between several study characteristics and abstract reporting completeness. For example, we demonstrated that journal endorsement of the CONSORT statement was not associated with abstract reporting completely, nor was public availability of the full-text manuscript. These findings suggest that other factors, including authors’ lack of awareness of the guidelines or journals lack of enforcement of guidelines, may potentially be contributing to the poor quality of abstract reporting. Another study done prior to adoption of the CONSORT extension statement utilized a checklist derived from the original CONSORT statement and applied to American Society of Clinical Oncology Annual Meeting abstracts and also demonstrated the generally poor quality of abstract reporting.

Prior studies have investigated the use of a more stringent application of the CONSORT guidelines during the peer review process in improving the quality of abstracts. Hopewell et al. conducted an interrupted time series analysis of the quality of abstracts describing medical randomized controlled trials both before and after the implementation of the CONSORT extension statement for abstracts in five leading medical journals. Journals were divided into three categories - those that did not endorse the CONSORT guidelines, those that endorsed the CONSORT guidelines but did not have an active policy to enforce them, and those that endorsed the CONSORT guidelines and had an active editorial process to enforce them. Interestingly, there was only a significant change in the mean number of items reported pre and post implementation of the CONSORT extension statement in the journals that had an editorial process to enforce the guidelines during the peer review process [25]. This highlights the importance of not only awareness of the guidelines by authors but the need for implementation and enforcement of these guidelines during the editorial review. Cobo et al. investigated the effect of adding an additional review based on reporting guidelines to the conventional peer review process for all original research manuscripts submitted to one journal. The overall quality of the manuscripts was improved with the addition of a second guideline-based review. Interestingly, during the editing process, authors were more likely to make manuscript revisions based on the conventional reviews rather than the guideline-based reviews. The authors postulated several potential reasons for this including a lack of awareness of these guidelines when designing a research study, making it more difficult to adhere to reporting certain elements in the publication process [26].

Accurate and transparent reporting of clinical trial information in abstracts may be particularly important in situations in which there is no access to full-text articles. Community oncologists, who provide 85 % of the cancer care in the USA, as estimated by the National Cancer Institute, are less likely to have comprehensive journal access and may be less likely to read multiple full-text articles when payment is required. The cost of purchasing these articles may be prohibitive to oncology physicians in the developing world where cost and access to information technology is potentially prohibitive, and who may rely only on the abstract content to make clinical decisions. Fortunately, we did find that open access to full-text articles was available in 80 % of our dataset. It is arguably more imperative for conference abstracts, in which a full text of a research study is not yet available, to stringently follow CONSORT guidelines as this is the only reflection of the details of a study.

There are some limitations to our study. This analysis was conducted specifically in randomized phase III metastatic solid tumor trials and further standardization of reporting may be necessary for other types of trials and disease states. The analysis included only English language studies. The importance of certain reporting elements may not be agreed upon universally by all authors or sub-specialities. Some may argue that several of the most poorly reported elements such as the setting in which the trial took place, reporting of allocation concealment, recruitment status, registration number, and funding source are important elements to include in the full text but are not necessarily informative in the abstract. Removal of these may raise completeness scores as well as allow researchers more word space to address other key trial details in the abstract. Future studies are needed to address this.

Importantly, CONSORT guidelines and extension statements represent a minimum standard in reporting. Adoption and expansion of these guidelines by the oncology community, and implementation and enforcement of these guidelines by journal editors, will potentially lead to greater adherence and improved quality of publications and improved clinical decision making.