Log in | Register

How to derive and validate clinical prediction models for use in intensive care medicine

José Labarère| Renaud Bertrand| Michael J. Fine
Statistics for Intensivists
Volume 40, Issue 4 / April , 2014

Pages 513 - 527

Abstract

Background

Clinical prediction models are formal combinations of historical, physical examination and laboratory or radiographic test data elements designed to accurately estimate the probability that a specific illness is present (diagnostic model), will respond to a form of treatment (therapeutic model) or will have a well-defined outcome (prognostic model) in an individual patient. They are derived and validated using empirical data and used to assist physicians in their clinical decision-making that requires a quantitative assessment of diagnostic, therapeutic or prognostic probabilities at the bedside.

Purpose

To provide intensivists with a comprehensive overview of the empirical development and testing phases that a clinical prediction model must satisfy before its implementation into clinical practice.

Results

The development of a clinical prediction model encompasses three consecutive phases, namely derivation, (external) validation and impact analysis. The derivation phase consists of building a multivariable model, estimating its apparent predictive performance in terms of both calibration and discrimination, and assessing the potential for statistical over-fitting using internal validation techniques (i.e. split-sampling, cross-validation or bootstrapping). External validation consists of testing the predictive performance of a model by assessing its calibration and discrimination in different but plausibly related patients. Impact analysis involves comparative research [i.e. (cluster) randomized trials] to determine whether clinical use of a prediction model affects physician practices, patient outcomes or the cost of healthcare delivery.

Conclusions

This narrative review introduces a checklist of 19 items designed to help intensivists develop and transparently report valid clinical prediction models.

Keywords

References

  1. Laupacis A, Sekar N, Stiell IG (1997) Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 277:488–494
    • View reference on PubMed
    • View reference on publisher's website
  2. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG (2009) Prognosis and prognostic research: what, why, and how? BMJ 338:375
    • View reference on publisher's website
  3. Steyerberg EW, Moons KG, van der Windt DA et al (2013) Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 10:e1001381
    • View reference on PubMed
    • View reference on publisher's website
  4. Vincent JL, Moreno R (2010) Clinical review: scoring systems in the critically ill. Crit Care 14:207
    • View reference on PubMed
    • View reference on publisher's website
  5. Reilly BM, Evans AT (2006) Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144:201–209
    • View reference on PubMed
    • View reference on publisher's website
  6. Moons KG, Altman DG, Vergouwe Y, Royston P (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338:b606
    • View reference on PubMed
    • View reference on publisher's website
  7. Steyerberg EW (2009) Clinical prediction models: a practical approach to development, validation, and updating. Springer, New York
  8. McGinn TG, Guyatt GH, Wyer PC et al (2000) Users’ guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 284:79–84
    • View reference on PubMed
    • View reference on publisher's website
  9. Royston P, Moons KG, Altman DG, Vergouwe Y (2009) Prognosis and prognostic research: developing a prognostic model. BMJ 338:b604
    • View reference on PubMed
    • View reference on publisher's website
  10. Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ 338:b605
    • View reference on PubMed
    • View reference on publisher's website
  11. Wasson JH, Sox HC, Neff RK, Goldman L (1985) Clinical prediction rules. Applications and methodological standards. N Engl J Med 313:793–799
    • View reference on PubMed
    • View reference on publisher's website
  12. Moons KG, Kengne AP, Grobbee DE et al (2012) Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98:691–698
    • View reference on PubMed
    • View reference on publisher's website
  13. Moons KG, Kengne AP, Woodward M et al (2012) Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98:683–690
    • View reference on PubMed
    • View reference on publisher's website
  14. Stiell IG, Wells GA (1999) Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 33:437–447
    • View reference on PubMed
    • View reference on publisher's website
  15. Labarere J, Schuetz P, Renaud B et al (2012) Validation of a clinical prediction model for early admission to the intensive care unit of patients with pneumonia. Acad Emerg Med 19:993–1003
    • View reference on PubMed
    • View reference on publisher's website
  16. Renaud B, Labarere J, Coma E et al (2009) Risk stratification of early admission to the intensive care unit of patients with no major criteria of severe community-acquired pneumonia: development of an international prediction rule. Crit Care 13:R54
    • View reference on PubMed
    • View reference on publisher's website
  17. Guyatt GH (2006) Determining prognosis and creating clinical decision rules. In: Haynes RB, Sackett DL, Guyatt GH, Tugwell P (eds) Clinical epidemiology: how to do clinical practice research. Lippincott Williams & Wilkins, New York
  18. Altman DG (2009) Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 27:235–243
    • View reference on PubMed
    • View reference on publisher's website
  19. Randolph AG, Guyatt GH, Calvin JE, Doig G, Richardson WS (1998) Understanding articles describing clinical prediction tools. Evidence Based Medicine in Critical Care Group. Crit Care Med 26:1603–1612
    • View reference on PubMed
    • View reference on publisher's website
  20. Altman DG (1991) Practical statistics for medical research. Chapman & Hall/CRC, London
  21. Aujesky D, Obrosky DS, Stone RA et al (2005) Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med 172:1041–1046
    • View reference on PubMed
    • View reference on publisher's website
  22. Fine MJ, Auble TE, Yealy DM et al (1997) A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 336:243–250
    • View reference on PubMed
    • View reference on publisher's website
  23. Royston P, Altman DG, Sauerbrei W (2006) Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25:127–141
    • View reference on PubMed
    • View reference on publisher's website
  24. Altman DG, Lausen B, Sauerbrei W, Schumacher M (1994) Dangers of using “optimal” cutpoints in the evaluation of prognostic factors. J Natl Cancer Inst 86:829–835
    • View reference on PubMed
    • View reference on publisher's website
  25. Steyerberg EW, Schemper M, Harrell FE (2011) Logistic regression modeling and the number of events per variable: selection bias dominates. J Clin Epidemiol 64:1464–1465 (author reply 1463–1464.)
    • View reference on PubMed
    • View reference on publisher's website
  26. Sauerbrei W, Royston P, Binder H (2007) Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med 26:5512–5528
    • View reference on PubMed
    • View reference on publisher's website
  27. Harrell FE Jr (2001) Regression modelling strategies with applications to linear models, logistic regression, and survival analysis. Springer, New York
  28. Vergouwe Y, Royston P, Moons KG, Altman DG (2010) Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 63:205–214
    • View reference on PubMed
    • View reference on publisher's website
  29. Altman DG, Bland JM (2007) Missing data. BMJ 334:424
    • View reference on PubMed
    • View reference on publisher's website
  30. Groenwold RH, White IR, Donders AR et al (2012) Missing covariate data in clinical research: when and when not to use the missing-indicator method for analysis. CMAJ 184:1265–1269
    • View reference on PubMed
    • View reference on publisher's website
  31. Groenwold RH, Donders AR, Roes KC, Harrell FE Jr, Moons KG (2012) Dealing with missing outcome data in randomized trials and observational studies. Am J Epidemiol 175:210–217
    • View reference on PubMed
    • View reference on publisher's website
  32. Liublinska V, Rubin DB (2012) Re: “dealing with missing outcome data in randomized trials and observational studies”. Am J Epidemiol 176:357–358
    • View reference on PubMed
    • View reference on publisher's website
  33. Janssen KJ, Donders AR, Harrell FE Jr et al (2010) Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 63:721–727
    • View reference on PubMed
    • View reference on publisher's website
  34. Concato J, Feinstein AR, Holford TR (1993) The risk of determining risk with multivariable models. Ann Intern Med 118:201–210
    • View reference on PubMed
    • View reference on publisher's website
  35. Harrell FE Jr, Lee KL, Mark DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–387
    • View reference on PubMed
    • View reference on publisher's website
  36. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996) A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49:1373–1379
    • View reference on PubMed
    • View reference on publisher's website
  37. Courvoisier DS, Combescure C, Agoritsas T, Gayet-Ageron A, Perneger TV (2011) Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol 64:993–1000
    • View reference on PubMed
    • View reference on publisher's website
  38. Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Chapman & Hall/CRC, New York
  39. Marshall RJ (2001) The use of classification and regression trees in clinical epidemiology. J Clin Epidemiol 54:603–609
    • View reference on PubMed
    • View reference on publisher's website
  40. Sun GW, Shook TL, Kay GL (1996) Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49:907–916
    • View reference on PubMed
    • View reference on publisher's website
  41. Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley, New York
  42. Sullivan LM, Massaro JM, D’Agostino RB Sr (2004) Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 23:1631–1660
    • View reference on PubMed
    • View reference on publisher's website
  43. Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21:128–138
    • View reference on PubMed
    • View reference on publisher's website
  44. Rufibach K (2010) Use of Brier score to assess binary predictions. J Clin Epidemiol 63:938–939
    • View reference on PubMed
    • View reference on publisher's website
  45. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD (2005) Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58:475–483
    • View reference on PubMed
    • View reference on publisher's website
  46. Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130:515–524
    • View reference on PubMed
    • View reference on publisher's website
  47. Altman DG, Royston P (2000) What do we mean by validating a prognostic model? Stat Med 19:453–473
    • View reference on PubMed
    • View reference on publisher's website
  48. Steyerberg EW, Harrell FE Jr, Borsboom GJ et al (2001) Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54:774–781
    • View reference on PubMed
    • View reference on publisher's website
  49. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall/CRC, New York
  50. Bleeker SE, Moll HA, Steyerberg EW et al (2003) External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56:826–832
    • View reference on PubMed
    • View reference on publisher's website
  51. Toll DB, Janssen KJ, Vergouwe Y, Moons KG (2008) Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 61:1085–1094
    • View reference on PubMed
    • View reference on publisher's website
  52. Peek N, Arts DG, Bosman RJ, van der Voort PH, de Keizer NF (2007) External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol 60:491–501
    • View reference on PubMed
    • View reference on publisher's website
  53. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61:76–86
    • View reference on PubMed
    • View reference on publisher's website
  54. Collins GS, Moons KG (2012) Comparing risk prediction models. BMJ 344:e3186
    • View reference on PubMed
    • View reference on publisher's website
  55. Campbell MK, Piaggio G, Elbourne DR, Altman DG (2012) Consort 2010 statement: extension to cluster randomised trials. BMJ 345:e5661
    • View reference on PubMed
    • View reference on publisher's website
  56. Donner A, Klar N (2000) Design and analysis of cluster randomized trials in health research. Arnold, London
  57. Hussey MA, Hughes JP (2007) Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 28:182–191
    • View reference on PubMed
    • View reference on publisher's website
  58. Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE (2003) Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. Int J Technol Assess Health Care 19:613–623
    • View reference on PubMed
    • View reference on publisher's website
  59. von Elm E, Altman DG, Egger M et al (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 147:573–577
    • View reference on publisher's website
  60. Renaud B, Santin A, Coma E et al (2009) Association between timing of intensive care unit admission and outcomes for emergency department patients with community-acquired pneumonia. Crit Care Med 37:2867–2874
    • View reference on PubMed
    • View reference on publisher's website
  61. Chalmers JD, Mandal P, Singanayagam A et al (2011) Severity assessment tools to guide ICU admission in community-acquired pneumonia: systematic review and meta-analysis. Intensive Care Med 37:1409–1420
  62. Marti C, Garin N, Grosgurin O et al (2012) Prediction of severe community-acquired pneumonia: a systematic review and meta-analysis. Crit Care 16:R141
    • View reference on PubMed
    • View reference on publisher's website
  63. Ewig S, Woodhead M, Torres A (2011) Towards a sensible comprehension of severe community-acquired pneumonia. Intensive Care Med 37:214–223
  64. Yealy DM, Auble TE, Stone RA et al (2005) Effect of increasing the intensity of implementing pneumonia guidelines: a randomized, controlled trial. Ann Intern Med 143:881–894
    • View reference on PubMed
    • View reference on publisher's website

Sign In

Connect with ICM

Top 5 Articles Editors Picks Supplement