On 2014 Apr 07, Gary Collins commented:

Thanks for the comments.

The systematic review only included articles published in 2010 in the Core Clinical Journals [Abridged Index Medicus], so articles published in 2007 will not have been eligible, nor will articles published in 2011 onwards. The search string identified a large 11826 potential articles, mainly because of the inclusive search string. Prediction models are not easily identifiable due to inconsistent terminology for such models (e.g. Prognostic models, prediction models/rules, risk scores etc...and many don't even provide this in the title or abstract), no MeSH terms that are consistently used and tagged to such articles etc...

Also, whilst there are clearly some good examples of external validation studies, the majority are not so well done and are poorly reported, yet validation is all what we are interested in - i.e. does a model actually work.

It is very unlikely that including more up-to-date articles will have changed our conclusions. However, a reporting guideline is currently under peer-review to improve the reporting of studies developing or validating multivariable prediction models...

http://blogs.bmj.com/bmj/2011/08/03/gary-collins-opening-up-multivariable-prediction-models/

They will hopefully been out by the summer.

On 2014 Apr 06, Shervin Assari commented:

I wish a file was available to show the 11,826 articles. I am sure I am author of 1 or 2. I wish I could look at the evaluation results related to my work.

Khedmat H, Karami GR, Pourfarziani V, Assari S, Rezailashkajani M, Naghizadeh MM. A logistic regression model for predicting health-related quality of life in kidney transplant recipients.Transplant Proc. 2007 May;39(4):917-22.

On 2014 Apr 06, GRAHAM COLDITZ commented:

A pity this thorough review searched the literature back in February 2011 before our external validation of the Rosner-Colditz log-incidence breast cancer model, using an independent cohort (California Teachers Study) and comparable follow-up time period (calendar years) and endpoints (invasive breast cancer) was reported. see http://link.springer.com/article/10.1007/s10549-013-2719-3

Overall, this rigorous review reminds us to move beyond just publishing risk models but rather to carefully plan and implement validation studies. This paper should move the field forward and exemplifies the value of systematic review of methods to advance the conduct and reporting of studies.

On 2014 Apr 06, GRAHAM COLDITZ commented:

A pity this thorough review searched the literature back in February 2011 before our external validation of the Rosner-Colditz log-incidence breast cancer model, using an independent cohort (California Teachers Study) and comparable follow-up time period (calendar years) and endpoints (invasive breast cancer) was reported. see http://link.springer.com/article/10.1007/s10549-013-2719-3

Overall, this rigorous review reminds us to move beyond just publishing risk models but rather to carefully plan and implement validation studies. This paper should move the field forward and exemplifies the value of systematic review of methods to advance the conduct and reporting of studies.

On 2014 Apr 06, Shervin Assari commented:

I wish a file was available to show the 11,826 articles. I am sure I am author of 1 or 2. I wish I could look at the evaluation results related to my work.

Khedmat H, Karami GR, Pourfarziani V, Assari S, Rezailashkajani M, Naghizadeh MM. A logistic regression model for predicting health-related quality of life in kidney transplant recipients.Transplant Proc. 2007 May;39(4):917-22.

On 2014 Apr 07, Gary Collins commented:

Thanks for the comments.

The systematic review only included articles published in 2010 in the Core Clinical Journals [Abridged Index Medicus], so articles published in 2007 will not have been eligible, nor will articles published in 2011 onwards. The search string identified a large 11826 potential articles, mainly because of the inclusive search string. Prediction models are not easily identifiable due to inconsistent terminology for such models (e.g. Prognostic models, prediction models/rules, risk scores etc...and many don't even provide this in the title or abstract), no MeSH terms that are consistently used and tagged to such articles etc...

Also, whilst there are clearly some good examples of external validation studies, the majority are not so well done and are poorly reported, yet validation is all what we are interested in - i.e. does a model actually work.

It is very unlikely that including more up-to-date articles will have changed our conclusions. However, a reporting guideline is currently under peer-review to improve the reporting of studies developing or validating multivariable prediction models...

http://blogs.bmj.com/bmj/2011/08/03/gary-collins-opening-up-multivariable-prediction-models/

They will hopefully been out by the summer.