When key fusion study indices are based on certain national databases, and then the underlying data is changed, what happens to these indices which are used by researchers to conduct studies?
Yes, things change over time. But when it comes to national databases like the National Surgical Quality Improvement Program (NSQIP), this is a big issue.
According to the authors of a study to look into this issue, “The aim of this study was to investigate the influence of changes in the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) database over the years on the calculation of the modified Frailty Index (mFI) and the modified Charlson Comorbidity Index (mCCI) for posterior lumbar fusion studies.”
The researchers showcased this in the recent study, “Systematic Changes in the National Surgical Quality Improvement Program Database Over the Years Can Affect Comorbidity Indices Such as the Modified Frailty Index and Modified Charlson Comorbidity Index for Lumbar Fusion Studies.”
This study appears in the June 1,2018 edition of Spine.
“Multiple studies have utilized the mFI and/or mCCI and showed them to be predictors of adverse postoperative outcomes.”
“However, changes in the NSQIP database have resulted in definition changes and/or missing data for many of the variables included in these indices. No studies have assessed the influence of different methods of treating missing values when calculating these indices on such studies.”
Blake Shultz, primary author for the manuscript told OTW, “We chose to investigate this topic after noticing that a number of variables included in the modified Charlson Comorbidity Index and modified Frailty Index are no longer being collected in recent years of the NSQIP database. This makes both carrying out and interpreting studies using these indices difficult in 2011 and in more recent years.”
“We found that the mean values of the mCCI and mFI changed for lumbar fusion patients with three different methods of handling the missing data.
“For example, if missing data was treated as null (i.e., the condition was not present in a patient), the mean mFI increased by 33.3% between 2005 and 2014. If patients with missing values were dropped, it was impossible to calculate the mFI in 2014 because all patients had missing data.”
“Finally, if the mFI was normalized by dividing the raw score by the number of variables actually collected in a given year, the mean mFI increase 183.3% between 2005-2014.”
“Practically, this shows that unless researchers report the methods they use to handle missing data when using these indices, it is very difficult to draw meaningful conclusions from their use in longitudinal studies, and studies at different time points. Researchers should take care to describe their treatment of missing data. Furthermore, a method of calculating these indices across years that takes into account changes in variable collection should be designed. Our normalization method for the mFI could serve as an example of this.”
“Most importantly, researchers should take away from our study that changes in data collection in national databases across the years can dramatically influence study results. It is important for authors to discuss how these changes could have influenced their results, and to delineate their treatment of missing data so that readers can interpret these studies appropriately.”
