Email updates

Keep up to date with the latest news and content from Trials and BioMed Central.

Open Access Methodology

Quality assessment of reporting of randomization, allocation concealment, and blinding in traditional chinese medicine RCTs: A review of 3159 RCTs identified from 260 systematic reviews

Jia He1, Liang Du1*, Guanjian Liu1*, Jin Fu2, Xiangyu He2, Jiayun Yu2 and Lili Shang2

Author Affiliations

1 Chinese Cochrane Centre, Chinese Evidence-Based Medicine Centre, West China Hospital, Sichuan University, Chengdu, 610041, China

2 West China School of Clinical Medicine, Sichuan University, Chengdu, 610041, China

For all author emails, please log on.

Trials 2011, 12:122  doi:10.1186/1745-6215-12-122

The electronic version of this article is the complete one and can be found online at: http://www.trialsjournal.com/content/12/1/122


Received:15 October 2010
Accepted:13 May 2011
Published:13 May 2011

© 2011 He et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Randomized controlled trials (RCTs) which are of poor quality tend to exaggerate the effect estimate and lead to wrong or misleading conclusions. The aim of this study is to assess the quality of randomization methods, allocation concealment and blinding within traditional Chinese medicine (TCM) RCTs, discuss issues identified for current TCM RCTs, and provide suggestions for quality improvement.

Methods

We searched Chinese Biomedical Database (CBM, 1978 to July 31, 2009) and the Cochrane Library (Issue 2, 2009) to collect TCM systematic reviews and meta-analyses according to inclusion/exclusion criteria, from which RCTs could be identified. The quality assessment involved whether the randomization methods, allocation concealment and blinding were adequate or not based the study reported. Stratified analyses were conducted of different types of diseases published in different journals (both Chinese and foreign) using different interventions. SPSS 15.0 software was used for statistic analyses.

Results

A total of 3159 RCTs were included, of which 2580 were published in Chinese journals and 579 in foreign journals. There were 381 (12%) RCTs which used adequate randomization methods; 207 (7%) RCTs which used adequate allocation concealment and 601 (19%) which used adequate blinding; there were 130 (4%) RCTs which both used adequate randomization methods and allocation concealment; and there were only 100 (3%) RCTs which used adequate randomization methods, allocation concealment, as well as blinding. In the RCTs published in foreign journals, the adequate randomization methods, allocation concealment and blinding accounted for a relatively large proportion (25%, 26%, and 60%, respectively) and increased with years, while in the RCTs published in Chinese journals, only the adequate randomization methods improved over time. The quality of non-drug intervention (chiefly acupuncture) RCTs was higher than that of drug intervention RCTs. In drug intervention, the quality of listed drugs is higher than the others. The quality of all included RCTs of all types of diseases was generally poor and no studies that were large in size and of high quality were found.

Conclusion

The quality of the current TCM RCTs as judged by their publications is generally poor, especially those published in Chinese journals. In future, researchers of TCM RCTs should attach more importance to experimental design and methodological quality, receive relevant training, and improve reporting quality using the Consolidated Standards of Reporting Trials (CONSORT) statement, so as to improve the quality of TCM clinical research and ensure truth and reliability of conclusions.

Background

Traditional Chinese medicine (TCM) is centuries old and has developed a unique system of diagnosis and treatment of disease in the practice [1]. TCM is not only an important part of Chinese health but also is accepted in many parts of the world thus offering a complementary or alternative form of health care. International medicine has been increasingly interested in TCM [2-4]. Randomized controlled trials (RCTs) and systematic reviews are commonly regarded first-class evidence in judging the treatment effect of interventions [5,6]. Systematic reviews are based on RCTs, whose deficiency may yield unreliability of the conclusions [7,8], therefore, the quality of included RCTs is of obvious relevance to the interpretability and reliability of the conclusions. Research shows that the RCTs which are of poor quality tend to exaggerate the effect estimates and lead to misleading conclusions [9-16]. The aim of this study is to assess the quality of randomization methods, allocation concealment and blinding of TCM RCTs based on study reports, discuss issues identified for current TCM RCTs, and provide suggestions for quality improvement.

Methods

Literature selection

The following databases were searched: Chinese Biomedical Database (CBM, 1978 to July 31, 2009) and the Cochrane Library 2009, Issue 2.

The following search strategies were used:

CBM:

#1 title: systematic review (in Chinese)

#2 keywords: systematic review (in Chinese)

#3 title: systematic overview (in Chinese)

#4 keywords: systematic overview (in Chinese)

#5 title: meta analysis (in Chinese)

#6 keywords: meta analysis (in hinese)

#7 title: meta-analysis (in Chinese)

#8 keywords: meta-analysis (in Chinese)

#9 title: systematic review (in English)

#10 keywords: systematic review (in English)

#11 title: systematic reviews (in English)

#12 keywords: systematic reviews (in English)

#13 title: meta analysis (in English)

#14keywords: meta analysis (in English)

#15 title: meta-analysis (in English)

#16 keywords: meta-analysis (in English)

#17 title: meta analyses (in English)

#18 keywords: meta analyses (in English)

#19 title: meta-analyses (in English)

#20 keywords: meta-analyses (in English)

#21 #1-20/or

The Cochrane Library 2009, Issue 2

#1 herb in Title, Abstract or Keywords

#2 herbal in Title, Abstract or Keywords

#3 herbs in Title, Abstract or Keywords

#4 Chinese in Title, Abstract or Keywords

#5 China in Title, Abstract or Keywords

#6 acupuncture in Title, Abstract or Keywords

#7 alternative complementary medicine in Title, Abstract or Keywords

#8 plant in Title, Abstract or Keywords

#9 moxibustion in Title, Abstract or Keywords

#10 #1~#9/or

All identified RCTs, published in Chinese journals and The Cochrane Library, included in systematic reviews and meta-analyses of TCM (including Chinese herbal medicine, Chinese medicine, Chinese standardized remedies, Chinese medicine preparations and Chinese medicine extracts), integrated TCM and Biomedicine, as well as acupuncture and massage (including acupuncture and electro-acupuncture), were included. Repeatedly-published articles, translated articles, non-full text articles and articles without mentioning sources of RCTs were excluded.

Quality assessment

The quality assessment in this study was performed according to three major procedures in methodological part of the Cochrane Handbook: ① Whether the randomization method was adequate (such as referring to a randomized number table, using a computer random number generator, coin tossing and throwing dice, etc.) or inadequate (such as sequence generated by odd or even date of birth, sequence generated by some rule based on date of admission or hospital record number, etc.). ② Whether allocation concealment was adequate (such as central allocation, sequentially numbered drug containers of identical appearance, sequentially numbered, opaque, sealed envelopes, etc) or inadequate (such as using an open random allocation schedule, alternation or rotation, date of birth, case record number, etc.). ③ Whether blinding was adequate (such as no blinding, but the review authors judge that the outcome and the outcome measurement are not likely to be influenced by lack of blinding; blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken; either participants or some key study personnel were not blinded, but outcome assessment was blinded and the non-blinding of others unlikely to introduce bias) or inadequate (such as no blinding or incomplete blinding, and the outcome or outcome measurement is likely to be influenced by lack of blinding; blinding of key study participants and personnel attempted, but likely that the blinding could have been broken, either participants or some key study personnel were not blinded, and the non-blinding of others likely to introduce bias) [17]. Assessment was based on trial publication only. We did not contact trial authors for clarification.

Stratified analysis

Included RCTs were published in Chinese journals or foreign journals.

Medicine and non-medicine interventions were divided with the former including listed medicines (Chinese standardized remedies, Chinese medicine extracts and single herbs), individualized prescription, mixture (listed medicines and individualized prescription) and unclear (the intervention was not reported).

Types of Diseases: classified according to the 10th version of the international disease classification (ICD-10) [18].

Data extraction

The researchers used EndNote software to manage titles and abstracts of citations obtained through database search. They(J He, and L Du) read titles and abstracts for primary screening based on the inclusion criteria, and then acquired the full text to read for further screening, before finally deciding whether they would be included or not. Four researchers extracted data, using a pro-forma, and then input the extracted materials into Epidata software. Two other researchers checked the data. Any differences were settled through discussion.

Data analysis

SPSS 15.0 was used for descriptive analyses and χ2 test.

Results

A total of 2041 articles were retrieved (1790 from CBM and 251 from The Cochrane Library). After the primary screening, there were 1781 articles were excluded and 260 articles (173 from CBM and 87 from The Cochrane Library) were included. A total of 3159 RCTs were identified from the 260 systematic reviews/meta-analyses, of which 2580 were published in Chinese journals and 579 in foreign journals (Figure 1).

thumbnailFigure 1. Flow chart of literature selection.

Assessment of reported randomization methods, allocation concealment and blinding

A total of 3159 RCTs were included, among which 381(12%) used adequate randomization methods; 207 (7%) used adequate allocation concealment; 601 (19%) used adequate blinding; 130 (4%) both used adequate randomization methods and allocation concealment; 100 (3%) were adequate for all three methods.

RCTs published in foreign journals were better than those published in Chinese journals, and the difference was significant (P = 0.000) (Table 1).

Table 1. Characters of randomization methods, allocation concealment and blinding of included RCTs

Changes of RCTs reported quality with time

All 3159 RCTs were published between 1965 and 2008 (Figure 2). The proportion of adequate randomization methods of RCTs increased year by year, but there was no clear increase in allocation concealment and blinding, and the allocation concealment accounted for a very small proportion (Figure 3). In terms of Chinese journals, the proportion of adequate randomization methods of RCTs between 1993 and 2007 increased sharply, but there was no clear increase in the allocation concealment and blinding, and the allocation concealment just occupied a very small proportion (Figure 4). By comparison, there was a large rise in the proportion of adequate randomization methods, allocation concealment and blinding of the RCTs published in foreign journals from 1993 through 2007 (Figure 5). In order to avoid bias caused by chance, RCTs published between 1965 and 1992, as well as in 2008, which number of the included RCTs was very small, were excluded from this analysis.

thumbnailFigure 2. Numbers of 3159 RCTs with years.

thumbnailFigure 3. Reported quality of RCTs between 1993 and 2007 (n = 2909).

thumbnailFigure 4. Reported quality of RCTs published in Chinese journals between 1993 and 2007 (n = 2535).

thumbnailFigure 5. Reported quality of RCTs published in foreign journals between 1993 and 2007 (n = 374).

Reported quality of RCTs of different interventions

Among 3159 included RCTs, 2546 (81%) were drug interventions and 613 (19%) were non-drug interventions. In the non-drug group, the proportion of adequate randomization methods, allocation concealment and blinding were 21%, 19%, and 38%, respectively, which were larger than those in drug intervention, and the difference were significant (P = 0.000) (Table 2).

Table 2. Characters of RCTs of different interventions (Chinese journals VS foreign journals)

In different drug intervention, the proportion of adequate randomization of Chinese standardized remedies RCTs were the largest (13%), the proportion of adequate allocation concealment of Chinese medicine extracts RCTs were the largest (9%), and the proportion of adequate blinding of single herbs RCTs were the largest (40%).

In different non-drug intervention, the proportion of adequate randomization and blinding (22% and 41%, respectively) of acupuncture RCTs were larger, and the proportion of adequate allocation concealment (19%) of acupuncture RCTs was the same with massage RCTs.

In terms of drug intervention RCTs published in foreign journals, except that the proportion of adequate randomization methods of mixed RCTs (including listed medicine and individualized prescription) and unclear RCTs were smaller than those of RCTs published in Chinese journals, other proportion of adequate randomization methods, allocation concealment, and blinding were all larger than those of RCTs published in Chinese journals.

In terms of non-drug RCTs published in foreign journals, the proportion of adequate randomization methods, allocation concealment and blinding were all larger than those of RCTs published in Chinese journals.

Reported quality of RCTs of different types of diseases

The included 3159 RCTs were classified according to the 10th version of the international disease classification (ICD-10), involving 16 types of diseases and lacking 5 types (Table 3). Quantities and qualities of all types of diseases had imbalanced distribution. Circulatory system RCTs had the largest number but were of low quality.

Table 3. Characters of RCTs of different types of diseases (Chinese journals VS foreign journals)

Discussion

Many studies have showed that RCTs not using randomization, allocation concealment or blinding exaggerate estimates of effect to various extents. Compared with the RCTs using blinding, the RCTs not using blinding yield 17% larger estimates of treatment effects and in trials with subjective outcomes, effect estimates are exaggerated by 25%. Compared with the RCTs using adequate allocation concealment, RCTs using unclear or inadequate concealment of allocation exaggerate estimates of effect by 30%-41% [10-14]. These showed that compared with other "flaws", unclear or inadequate allocation concealment will cause a larger bias, which highlights the importance of allocation concealment. This study indicates that the adequate allocation concealment takes up the smallest proportion (7%) of the three assessed aspects. Although the adequate randomization methods accounted for a larger proportion (12%) than allocation concealment, there are also some investigations which showed that only 6.8% of the RCTs published in Chinese journals were deemed authentic randomized trials [19]. So the quality of the TCM RCTs in this study may be overstated.

Compared with those published in Chinese journals, the TCM RCTs published in foreign journals are of higher quality. In terms of changes over time, the proportion of reports of adequate randomization has increased year on year, suggesting that more attention has been paid to the randomization methods of Chinese TCM RCTs. However, the proportion of adequate allocation concealment and blinding are small and has remained at approximately the same level. By contrast, the proportion of adequate randomization methods, allocation concealment and blinding of the TCM RCTs published in foreign journals has increased overtime. This suggests that Chinese journals are failing to improve the standards of published RCTs.

Acupuncture, thousands of years of history in China, is a traditional medicine recognized by the World Health Organisation and has become a component of health care in many countries [20]. Since late 1990s, numbers of acupuncture clinical trials have grown dramatically with the increase of research funds in western countries [21]. The results of this study shows that in the light of interventions of TCM RCTs, the quality of non-drug intervention is evidently higher than that of drug intervention, and acupuncture is the largest group. In drug intervention, the quality of RCTs of Chinese standardized remedies, Chinese medicine extracts and single herbs is comparatively high while that of individualized prescription, mixtures (including listed medicine and individualized prescription) and those where the intervention is unclear is low. Individualized prescription which is common in TCM is influenced by a doctor's experience and training. There are no agreed formulae for drug ingredients, no standard sources of drugs, or preparation methods. In addition, doses of Chinese herbs are not standard. Listed medicine and single herbs can overcome the above shortcomings of individualized prescription, so the quality of RCTs of these kinds is higher and clinical research develops in a more extensive way.

With the development of TCM, current TCM RCTs involve wide study fields and many types of diseases. This study was aimed at determining which medical specialties produce with high-quality RCTs, but these were not identified.

The quality assessment in this study was based on how the original study was reported, so it was affected by the quality of reporting. There also may be some other factors not in the power of the reseachers such as journal word length etc. Some studies have showed that use of the Consolidated Standards of Reporting Trials (CONSORT) statement is associated with improvements in the reporting quality of RCTs, but few researchers and journals adopt it [22-30]. The CONSORT extensions for acupuncture (the Standards for Reporting Interventions in Clinical Trials of Acupuncuture, STRICTA) and herbal medicine were designed to improve the completeness and transparency of reporting of interventions in controlled trials of acupunture and herbal medicine [31,32].

Conclusion

The quality of the current TCM RCTs as judged by their publications is generally poor, especially those published in Chinese journals. In future, researchers should attach more importance to design and methodological quality, especially to allocation concealment. The quality of RCTs can also be improved through promoting participation of methodologists and statisticians, enhancing international cooperation, and adopting guidelines such as the CONSORT statement and its extensions for acupuncture and herbal medicine. Meanwhile, medical journals should also be more discerning in what they accept for publication, so as to improve the quality of TCM clinical research and ensure truth and reliability of conclusions.

Abbreviations

RCT: randomized control trial; TCM: traditional Chineses medicine; CBM: Chinese Biomedical Database; CONSORT: Consolidated Standards of Reporting Trials; STRICTA: Standards for Reporting Interventions in Clinical Trials of Acupuncuture.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JH and LD participated in the design of the study and drafted the article. GL conceived the study, interpreted the data and revised the manuscript. JF, XH, JY and LS extracted the data and performed the statistical anlysis. All authors read and approved the manuscript.

References

  1. Wang G, Mao Bing, Xiong ZY, et al.: The quality of reporting of randomized controlled trails of Traditional Chinese Medicine: A survey of 13 randomly selected journals from mainland China.

    Clinical Therapeutics 2007, 29:1457. OpenURL

  2. Lai SL, Hu JQ, Guo XF: Evidence-Based Medicine and Clinical Studies of Traditional Chinese Medicine.

    Journal of Guangdong University of Traditional Chinese Medicine 2000, 17(3):1-8. OpenURL

  3. Liu JP, Xia Y: Quality Appraisal of Systematic Reviews or Meta-analysis on Traditional Chinese Medicine Published in Chinese Journals.

    Chinese Journal of Integrative Medicine 2007, 27(4):306-308. OpenURL

  4. Chen WH, Liu DB, Zhou W, et al.: Symptomatology in Evidence-based Traditional Chinese Medicine.

    Chinese Archives of Traditional Chinese Medicine 2005, 23(3):456-447. OpenURL

  5. Sackett DL, Richardson WS, Roseberg W, Haynes RB: Evidence-based Medicine: How to Practice and Teach.

    New York: Churchill Livingstone 1997. OpenURL

  6. Cook DJ, Mulrow CD, Haynes RB: Systematic reviews: Synthesis of best evidence for Clinical decisions.

    Annals Internal Medicine 1997, 126:376-380. OpenURL

  7. Bian ZX, Li YP, David Moher, et al.: Improving the quality of randomized controlled trials in Chinese herbal medicine, part I: clinical trial design and methodology.

    Journal of Chinese Integrative Medicine 2006, 4:121. OpenURL

  8. Peter Juni, Altman DouglasG, Matthias Egger: Systematic reviews in health care: assessing the quality of controlled clinical trials.

    BMJ 2001, 323:42. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  9. Gluud LL: Bias in clinical intervention research.

    American Journal of Epidemiology 2006, 163:493-501. PubMed Abstract | Publisher Full Text OpenURL

  10. Schulz KF, Chalmers I, Hayes RJ, et al.: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.

    JAMA 1995, 273(5):408-412. PubMed Abstract | Publisher Full Text OpenURL

  11. Moher D, Pham B, Jones A, et al.: Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta- analyses?

    Lancet 1998, 352(9128):609-613. PubMed Abstract | Publisher Full Text OpenURL

  12. Wood L, Egger M, Gluud LL, et al.: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study.

    BMJ 2008, 336(7644):601-605. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  13. Kunz R, Vist G, Oxman AD: Randomisation to protect against selection bias in healthcare trials.

    Cochrane Database of Systematic Reviews 2007., (2) OpenURL

  14. Kunz R, Oxman AD: The unpredictability paradox: rovicw of empirical comparisons of randomized and non-randomized clinical trials.

    BMJ 1998, 317:1185-1190. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  15. Li J: Allocation Concealment: Why and How?

    Chinese Journal of Evidence-based Medicine 2004, 4(10):714-715. OpenURL

  16. Schulz KF, Chalmers I, Grimes DA, et al.: Assessing the quality of randomization from reports of controlled trials published in obstertrics and gynecology in journals.

    JAMA 1994, 272:125-128. PubMed Abstract | Publisher Full Text OpenURL

  17. Higgins JPT, Altman DG: Chapter 8: Assessing risk of bias in included studies. [http://www.cochrane-handbook.org] webcite

    In Cochrane Handbook for Systematic Reviews of Interventions.Version 5.0.1 [updated September 2008]. The Cochrane Collaboration Edited by Higgins JPT, Green S. 2008. OpenURL

  18. WHO: the Tenth Revision of the International Classification of Diseases (ICD-10).

  19. Wu TX, Li YP, Bian ZX, et al.: Randomized trials published in some Chinese journals: how many are randomized?

    Trials 2009, 10:46. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  20. Guan ZH: Acupuncture Medicine Research Progress and Prospects at Home and Abroad.

    Yunnan Journal of Traditional Chinese Medicine and Materia Medica 2000, 21(1):7-8. OpenURL

  21. Lao LX: Evidence-based medicine in acupuncture.

    World Journal of Acupuncture-Moxibustion 2007, 17(4):1. OpenURL

  22. Begg CB, Cho MK, Eastwood S, et al.: Improving the quality of reporting of randomized controlled trials: the CONSORT statement.

    JAMA 1996, 276:637-639. PubMed Abstract | Publisher Full Text OpenURL

  23. Altman DG, Schulz KF, Moher D, et al.: The revised CONSORT statement for reporting randomized trials: explanation and elaboration.

    Annals of Internal Medicine 2001, 134:663-694. PubMed Abstract | Publisher Full Text OpenURL

  24. Moher D, Hopewell S, Schulz KF, et al.: CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials.

    BMJ 2010, 340:c869. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  25. Plint AC, Moher D, Morrison A, et al.: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review.

    The Medical journal of Australia 2006, 185(5):263-267. PubMed Abstract | Publisher Full Text OpenURL

  26. Altman DG: Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors.

    BMJ 2005, 330:1056-1057. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  27. Moher D, Jones A, Lepage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before and after evaluation?

    JAMA 2001, 285:1992-1995. PubMed Abstract | Publisher Full Text OpenURL

  28. Hopewell S, Altman DG, Moher D, et al.: Endorsement of the CONSORT Statement by high impact medical journals-survey of journal editors and journal 'Instructions to Authors'.

    Trials 2008, 9:20. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  29. Xu L, Li J, Zhang MM, et al.: Chinese authors do need CONSORT: Reporting quality assessment for five leading Chinese medical journals.

    Contemporary Clinical Trials 2008, 29:727-731. PubMed Abstract | Publisher Full Text OpenURL

  30. Wang L, Li YL, Li J: Quality of reporting of trial abstracts needs to be improved: using the CONSORT for abstracts to assess the four leading Chinese medical journals of traditional Chinese medicine.

    Trials 2010, 11:75. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  31. MacPherson H, Altman DG, Hammerschlag R, Youping L, Taixiang W, et al.: Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): Extending the CONSORT Statement.

    PLoS Medicine 2010, 7(6):e1000261. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  32. Gagnier JJ, Boon H, Rochon P, et al.: Reporting randomized, controlled trials of herbal interventions: an elaborated CONSORT statement.

    Annals of Internal Medicine 2006, 144(5):364-367. PubMed Abstract OpenURL