Log in | Register

Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessmentOpen access

Lilian Minne| Saeid Eslami| Nicolette de Keizer| Evert de Jonge| Sophia E. de Rooij| Ameen Abu-Hanna
Original
Volume 38, Issue 1 / January , 2012

Pages 40 - 46

Abstract

Purpose

The aim of our study was to explore, using an innovative method, the effect of temporal changes in the mortality prediction performance of an existing model on the quality of care assessment. The prognostic model (rSAPS-II) was a recalibrated Simplified Acute Physiology Score-II model developed for very elderly Intensive Care Unit (ICU) patients.

Methods

The study population comprised all 12,143 consecutive patients aged 80 years and older admitted between January 2004 and July 2009 to one of the ICUs of 21 Dutch hospitals. The prospective dataset was split into 30 equally sized consecutive subsets. Per subset, we measured the model’s discrimination [area under the curve (AUC)], accuracy (Brier score), and standardized mortality ratio (SMR), both without and after repeated recalibration. All performance measures were considered to be stable if <2 consecutive points fell outside the green zone [mean ± 2 standard deviation (SD)] and none fell outside the yellow zone (mean ± 4SD) of pre-control charts. We compared proportions of hospitals with SMR>1 without and after repeated recalibration for the year 2009.

Results

For all subsets, the AUCs were stable, but the Brier scores and SMRs were not. The SMR was downtrending, achieving levels significantly below 1. Repeated recalibration rendered it stable again. The proportions of hospitals with SMR>1 and SMR<1 changed from 15 versus 85% to 35 versus 65%.

Conclusions

Variability over time may markedly vary among different performance measures, and infrequent model recalibration can result in improper assessment of the quality of care in many hospitals. We stress the importance of the timely recalibration and repeated validation of prognostic models over time.

Keywords

References

  1. Lucas PJ, Abu-Hanna A (2009) Prognostic methods in medicine. Artif Intell Med 15:105–119
    • View reference on publisher's website
  2. Abu-Hanna A, Lucas PJ (2001) Prognostic models in medicine. AI and statistical approaches. Methods Inf Med 40:1–5
    • View reference on PubMed
  3. Zimmerman D (1999) Benchmarking: measuring yourself against the best. Trustee 52:22–23
    • View reference on PubMed
  4. Zimmerman JE, Alzola C, Von Rueden KT (2003) The use of benchmarking to identify top performing critical care units: a preliminary assessment of their policies and practices. J Crit Care 18:76–86
    • View reference on PubMed
    • View reference on publisher's website
  5. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG (2009) Prognosis and prognostic research: what, why, and how? Br Med J 338:b375
    • View reference on publisher's website
  6. Wyatt JC (1995) Prognostic models: clinically useful or quickly forgotten? Br Med J 311:1539–1541
    • View reference on publisher's website
  7. Mallet S, Royston P, Waters R, Dutton S, Altman DG (2010) Reporting performance of prognostic models in cancer: a review. BMC Med 8:21
    • View reference on publisher's website
  8. de Rooij SE, Abu-Hanna A, Levi M, de Jonge E (2007) Identification of high-risk subgroups in very elderly intensive care unit patients. Crit Care 11:R33
    • View reference on PubMed
    • View reference on publisher's website
  9. Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963
    • View reference on PubMed
    • View reference on publisher's website
  10. de Jonge E, Bosman RJ, van der Voort PH, Korsten HH, Scheffer GJ, de Keizer NF (2003) Intensive care medicine in the Netherlands, 1997–2001. I. Patient population and treatment outcome. Ned Tijdschr Geneeskd 147:1013–1017
    • View reference on PubMed
  11. Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. Br Med J 338:b605
    • View reference on publisher's website
  12. Altman DG (1990) Practical statistics for medical research. Chapman and Hall, London
  13. Fleis J, Levin B, Paik M (2003) Statistical methods for rates and proportions. J Wiley, New York
  14. Wheeler JW (2004) Advanced topics in statistical process control. SPC Press, Knoxville
  15. Kramer AA (2005) Predictive mortality models are not like fine wine. Crit Care 9:636–637
    • View reference on PubMed
    • View reference on publisher's website
  16. Le Gall JR, Neumann A, Hemery F, Bleriot JP, Fulgencio JP, Garrigues B et al (2005) Mortality prediction using SAPS II: an update for French intensive care units. Crit Care 9:R645–R652
    • View reference on PubMed
    • View reference on publisher's website
  17. Harrison DA, Brady AR, Parry GJ, Carpenter JR, Rowan K (2006) Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom. Crit Care Med 34:1378–1388
    • View reference on PubMed
    • View reference on publisher's website
  18. Steyerberg EW (2009) Clinical Prediction Models. A practical approach to development, validation, and updating. Springer, New York
  19. Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD (2004) Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23:2567–2586
    • View reference on PubMed
    • View reference on publisher's website
  20. Bakhshi-Raiez F, Peek N, Bosman RJ, de Jonge E, de Keizer NF (2007) The impact of different prognostic models and their customization on institutional comparison of intensive care units. Crit Care Med 35:2553–2560
    • View reference on PubMed
    • View reference on publisher's website

Sign In

Connect with ICM

Top 5 Articles Editors Picks Supplement