## Statistical Software

### Evaluating risk prediction markers for binary outcomes via relative utility curves

Stuart G. Baker, 2014

#### Introduction

A risk prediction marker is a baseline variable for predicting an event in a control group that is used to make treatment decisions for persons at high risk for the event. This program evaluates a risk prediction model and the addition of an additional risk prediction marker or an additional set of risk prediction markers. The user must have already fit two risk prediction models in a training sample, a baseline Model 1 and an expanded Model 2, and applied the risk prediction model to a test sample. The input is either a pair of cross-classified tables of risks or a list of predicted risks and binary outcomes by model.

This program computes ROC curves, relative utility curves, maximum acceptable testing harms, and test tradeoffs for evaluating an additional marker for risk prediction.

#### Reference

Baker SG, Schuit S, Steyerberg EW, Pencina MJ, Vickers A, Moons KGM, Mol BWJ, Lindeman KS. How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through. Under submission.

#### Requirement

Mathematica Version 8 or later.

#### To run example in manuscript

*copy* | all files into some folder called "FOLDER" |

*start* | a new Mathematica session |

*type* | SetDirectory["FOLDER"] |

*type* | << rufit.m |

*type* | RUFit[dataCS,NewFitQ->True,MaxBoot->10000] |

To try on your own data,

*type* | TrialFit[**dataset**, options] where **dataset** can either be in matrix or list form |

Options

| Default | Explanation |

NewFitQ | TRUE | Fit data or used stored results from previous fit |

P | Automatic | Event rate; Automatic is based on sample under Model 2 |

ShowDataQ | TRUE | Show table of counts or not |

ShowEstimateQ | TRUE | Show details of estimation |

FracRUatMaxT | 0.05 | fraction of relative utility under Model 1 for maximum value of risk threshold range |

MaxBoot | 100 | Number of bootstrap iterations |

dataset={matx, maty, riskscore, riskscorename, model1name, model2name, datasetname, "table"},

matx | matrix for cross-classification of risk among persons with the event, |

maty | matrix for cross-classification of risk among persons without the event |

riskscore | list of risks corresponding to each category |

riskscorename | list of names of risk intervals corresponding to each category |

model1name | name of model 1 |

model2name | name of model 2 |

datasetname | name of data set |

dataset={y, model1risk, model2risk, model1name, model2name, datasetname, "list"},

y | list of binary outcomes (0, 1) by indvidual |

model1risk | list of predicted risks for model 1 by individual |

model2risk | list of predicted risks for model 2 by individual |

model1name | name of model 1 |

model2name | name of model 2 |

datasetname | name of data set |

datasetname | name of data set |

Downloads

Download All (zip 37kb)

File name / size | Description |

rufit.m (M File, 8kb) | main packages with input checks |

rufitkey.m (M File, 10kb) | main processing function |

rufitcore.m (M File, 9kb) | computes concave ROC curve and corresponding RU curve |

rufitdru.m (M File, 2kb) | interpolates for computing difference in RU at risk thresholds |

rufitboot.m (M File, 6kb) | bootstrap computation of standard errrors |

rufitplot.m (M File, 12kb) | plots of ROC and RU curves and calibration |

rufitreport.m (M File, 7kb) | calibration plot and test tradeoff |

rufitdata.m (M File, 7kb) | generates data examples from the literature |

rufitnull.m (M File, 7kb) | simulation under null hypothesis RUFITNull[] |

rufitnullstat.m (M File, 7kb) | compute statistics under null hypothesis |

rufitnullgen.m (M File, 6kb) | generate simulated data under null hypothesis |

Disclaimer

This code is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and non-infringement. In no event shall the NCI or the individual developers be liable for any claim, damages or other liability of any kind. Use of this code by recipient is at recipient's own risk. NCI makes no representations that the use of the code will not infringe any patent or proprietary rights of third parties.

Last updated: March 18, 2014

Back to top