Log in | Register

Quality of reporting of randomized controlled trials published in Intensive Care Medicine from 2001 to 2010

Nicola Latronico| Marta Metelli| Maddalena Turin| Simone Piva| Frank A. Rasulo| Cosetta Minelli
Original
Volume 39, Issue 8 / August , 2013

Pages 1386 - 1395

Abstract

Purpose

To evaluate the methodological quality of randomized controlled trials (RCTs) published in Intensive Care Medicine from 2001 to 2010, and to compare it with a previous review of RCTs published from 1975 to 2000.

Methods

We assessed the quality of reporting of randomization, blinding and participant flow, both individually and combined within the Jadad scale, and compared them with findings from our previous review. For RCTs published from 2001 to 2010, we also evaluated the frequency of distorted finding presentation (spin) and inflated predicted treatment effect (delta inflation).

Results

In the 221 RCTs from 2001 to 2010, the sample size was significantly larger than in the older series, and there was a higher proportion of studies with negative findings. Reporting of the rationale for sample size estimation and allocation concealment increased significantly, but reporting of other important individual methodological components did not change substantially compared with the previous period and remained low. Among RCTs from 2001 to 2010, a spin strategy was used in 69 of 111 RCTs with statistically negative results, while delta inflation was present in 7 of 11 RCTs evaluating survival as a primary outcome. Papers with higher Jadad scores were cited more often than the others.

Conclusions

Quality of reporting of RCTs published in Intensive Care Medicine has only partly improved over time, and spin and delta bias are of frequent occurrence. There is a need for stronger adherence to CONSORT recommendations, with special emphasis on accurate description of randomization and blindness, and correct reporting of statistically non-significant results.

Keywords

References

  1. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S (1995) Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 16:62–73
    • View reference on PubMed
    • View reference on publisher's website
  2. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273:408–412
    • View reference on PubMed
    • View reference on publisher's website
  3. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609–613
    • View reference on PubMed
    • View reference on publisher's website
  4. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T (2001) The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 134:663–694
    • View reference on PubMed
    • View reference on publisher's website
  5. Kjaergard LL, Villumsen J, Gluud C (2001) Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 135:982–989
    • View reference on PubMed
    • View reference on publisher's website
  6. Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S (2012) Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 344:e1119
    • View reference on PubMed
    • View reference on publisher's website
  7. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, JP AI, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA (2012) Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med
  8. Jüni P, Altman DG, Egger M (2008) Assessing the quality of randomised controlled trialssystematic reviews in health care. BMJ, London, pp 87–108
  9. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17:1–12
    • View reference on PubMed
    • View reference on publisher's website
  10. Altman DG (2001) Systematic reviews of evaluations of prognostic variables. BMJ 323:224–228
    • View reference on PubMed
    • View reference on publisher's website
  11. Huwiler-Muntener K, Juni P, Junker C, Egger M (2002) Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287:2801–2804
    • View reference on PubMed
    • View reference on publisher's website
  12. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336:601–605
    • View reference on PubMed
    • View reference on publisher's website
  13. Aberegg SK, Richards DR, O’Brien JM (2010) Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care 14:R77
    • View reference on PubMed
    • View reference on publisher's website
  14. Boutron I, Dutton S, Ravaud P, Altman DG (2010) Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 303:2058–2064
    • View reference on PubMed
    • View reference on publisher's website
  15. Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG (2012) Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA 308:2594–2604
    • View reference on PubMed
    • View reference on publisher's website
  16. Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A (2002) Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med 28:1316–1323
  17. Azoulay E, Citerio G, Timsit JF (2013) The identity of Intensive Care Medicine. Intensive Care Med 39:343–344
  18. Sud S, Sud M, Friedrich JO, Meade MO, Ferguson ND, Wunsch H, Adhikari NK (2010) High frequency oscillation in patients with acute lung injury and acute respiratory distress syndrome (ARDS): systematic review and meta-analysis. BMJ 340:c2327
    • View reference on PubMed
    • View reference on publisher's website
  19. Felder TM, Palmer NR, Lal LS, Mullen PD (2011) What is the evidence for pharmaceutical patient assistance programs? A systematic review. J Health Care Poor Underserved 22:24–49
    • View reference on PubMed
  20. Poolman RW, Struijs PA, Krips R, Sierevelt IN, Lutz KH, Bhandari M (2006) Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Med Res Methodol 6:44
    • View reference on PubMed
    • View reference on publisher's website
  21. Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM (2007) Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 14:390–398
    • View reference on PubMed
    • View reference on publisher's website
  22. Sut N, Senocak M, Uysal O, Koksalan H (2008) Assessing the quality of randomized controlled trials from two leading cancer journals using the CONSORT statement. Hematol Oncol Stem Cell Ther 1:38–43
    • View reference on PubMed
  23. Bai Y, Gao J, Zou DW, Li ZS (2009) Methodological reporting of randomized clinical trials in major gastroenterology and hepatology journals in 2006. Hepatology 49:2108–2112
    • View reference on PubMed
    • View reference on publisher's website
  24. Danilla S, Wasiak J, Searle S, Arriagada C, Pedreros C, Cleland H, Spinks A (2009) Methodological quality of randomised controlled trials in burns care. A systematic review. Burns 35:956–961
    • View reference on PubMed
    • View reference on publisher's website
  25. Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG (2010) The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340:c723
    • View reference on PubMed
    • View reference on publisher's website
  26. Strech D, Soltmann B, Weikert B, Bauer M, Pfennig A (2011) Quality of reporting of randomized controlled trials of pharmacologic treatment of bipolar disorders: a systematic review. J Clin Psychiatr 72:1214–1221
    • View reference on publisher's website
  27. Agha RA, Camm CF, Edison E, Orgill DP (2012) The methodological quality of randomized controlled trials in plastic surgery needs improvement: a systematic review. J Plast Reconstr Aesthet Surg
  28. Mills EJ, Wu P, Gagnier J, Devereaux PJ (2005) The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials 26:480–487
    • View reference on PubMed
    • View reference on publisher's website
  29. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D (2012) Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 11:MR000030
  30. Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 152:726–732
    • View reference on PubMed
    • View reference on publisher's website
  31. Dechartres A, Charles P, Hopewell S, Ravaud P, Altman DG (2011) Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. J Clin Epidemiol 64:136–144
    • View reference on PubMed
    • View reference on publisher's website
  32. Hirst A, Altman DG (2012) Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS ONE 7:e35621
    • View reference on PubMed
    • View reference on publisher's website
  33. Karanicolas PJ, Farrokhyar F, Bhandari M (2010) Practical tips for surgical research: blinding: who, what, when, why, how? Can J Surg 53:345–348
    • View reference on PubMed
  34. Schulz KF, Altman DG, Moher D, Fergusson D (2010) CONSORT 2010 changes and testing blindness in RCTs. Lancet 375:1144–1146
    • View reference on PubMed
    • View reference on publisher's website
  35. Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P (2009) Reporting of sample size calculation in randomised controlled trials: review. BMJ 338:b1732
    • View reference on PubMed
    • View reference on publisher's website
  36. Moher D, Fortin P, Jadad AR, Juni P, Klassen T, Le Lorier J, Liberati A, Linde K, Penna A (1996) Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 347:363–366
    • View reference on PubMed
    • View reference on publisher's website
  37. Schulz KF, Grimes DA (2002) Allocation concealment in randomised trials: defending against deciphering. Lancet 359:614–618
    • View reference on PubMed
    • View reference on publisher's website
  38. Guyatt GH, Mills EJ, Elbourne D (2008) In the era of systematic reviews, does the size of an individual trial still matter. PLoS Med 5:e4
    • View reference on PubMed
    • View reference on publisher's website
  39. Schulz KF, Grimes DA (2005) Multiplicity in randomised trials II: subgroup and interim analyses. Lancet 365:1657–1661
    • View reference on PubMed
    • View reference on publisher's website
  40. Schulz KF, Grimes DA (2005) Multiplicity in randomised trials I: endpoints and treatments. Lancet 365:1591–1595
    • View reference on PubMed
    • View reference on publisher's website
  41. Schulz KF, Grimes DA (2005) Sample size calculations in randomised trials: mandatory and mystical. Lancet 365:1348–1353
    • View reference on PubMed
    • View reference on publisher's website
  42. Lundh A, Gotzsche PC (2008) Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol 8:22
    • View reference on PubMed
    • View reference on publisher's website
  43. Hyman M (2010) Science for sale: protect yourself from medical research deception. http://www.huffingtonpost.com/dr-mark-hyman/dangerous-spin-doctors-7-_b_747325.html. Accessed 28 June 2012
  44. International Committee of Medical Journal Editors (2013) Uniform requirements for manuscripts submitted to biomedical journals: publishing and editorial issues related to publication in biomedical journals: obligation to publish negative studies. http://www.icmje.org/publishing_1negative.html. Accessed 2 April 2013
  45. Altman DG, Bland JM (1995) Absence of evidence is not evidence of absence. BMJ 311:485
    • View reference on PubMed
    • View reference on publisher's website
  46. Powers JH (2008) Noninferiority and equivalence trials: deciphering ‘similarity’ of medical interventions. Stat Med 27:343–352
    • View reference on PubMed
    • View reference on publisher's website
  47. Drazen JM (2012) Believe the data. N Engl J Med 367:1152–1153
    • View reference on PubMed
    • View reference on publisher's website
  48. Kesselheim AS, Robertson CT, Myers JA, Rose SL, Gillet VBA, Ross KM, Glynn RJ, Joffe S, Avorn J (2012) A randomized study of how physicians interpret research funding disclosures. N Engl J Med 367:1119–1127
  49. (2005) In praise of soft science. Nature 435:1003
  50. Bonati MR, Drusini AG (1996) Morgagni and the impact factor. Nature 381:271–271
    • View reference on PubMed
    • View reference on publisher's website
  51. Smith R (2003) Medical journals and pharmaceutical companies: uneasy bedfellows. BMJ 326:1202–1205
    • View reference on PubMed
    • View reference on publisher's website
  52. Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med 2:e138
    • View reference on PubMed
    • View reference on publisher's website
  53. Lundh A, Barbateskovic M, Hrobjartsson A, Gotzsche PC (2010) Conflicts of interest at medical journals: the influence of industry-supported randomised trials on journal impact factors and revenue—cohort study. PLoS Med 7:e1000354
    • View reference on PubMed
    • View reference on publisher's website
  54. McVeigh ME, Mann SJ (2009) The journal impact factor denominator: defining citable (counted) items. JAMA 302:1107–1109
    • View reference on PubMed
    • View reference on publisher's website
  55. Callaham M, Wears RL, Weber E (2002) Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 287:2847–2850
    • View reference on PubMed
    • View reference on publisher's website
  56. Nieminen P, Carpenter J, Rucker G, Schumacher M (2006) The relationship between quality of research and citation frequency. BMC Med Res Methodol 6:42
    • View reference on PubMed
    • View reference on publisher's website
  57. Etter JF, Stapleton J (2009) Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. J Clin Epidemiol 62:831–837
    • View reference on PubMed
    • View reference on publisher's website
  58. Filion KB, Pless IB (2008) Factors related to the frequency of citation of epidemiologic publications. Epidemiol Perspect Innov EP+I 5:3

Sign In

Connect with ICM

Top 5 Articles Editors Picks Supplement