Skip to content
National Cancer Institute National Cancer Institute U.S. National Institutes of Health
Division of Cancer Prevention logo
Home Site Map Contact DCP
Programs & Resources

Biometry Research Group

Statistical Software

Evaluating risk prediction markers for binary outcomes via relative utility curves

Stuart G. Baker, 2014


A risk prediction marker is a baseline variable for predicting an event in a control group that is used to make treatment decisions for persons at high risk for the event. This program evaluates a risk prediction model and the addition of an additional risk prediction marker or an additional set of risk prediction markers. The user must have already fit two risk prediction models in a training sample, a baseline Model 1 and an expanded Model 2, and applied the risk prediction model to a test sample. The input is either a pair of cross-classified tables of risks or a list of predicted risks and binary outcomes by model.

This program computes ROC curves, relative utility curves, maximum acceptable testing harms, and test tradeoffs for evaluating an additional marker for risk prediction.


Baker SG, Schuit S, Steyerberg EW, Pencina MJ, Vickers A, Moons KGM, Mol BWJ, Lindeman KS. How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through. Under submission.


Mathematica Version 8 Exit Disclaimer or later.

To run example in manuscript

copy all files into some folder called "FOLDER"
start a new Mathematica session
type   SetDirectory["FOLDER"]
type << rufit.m
type RUFit[dataCS,NewFitQ->True,MaxBoot->10000]

To try on your own data,

typeTrialFit[dataset, options] where dataset can either be in matrix or list form


NewFitQTRUEFit data or used stored results from previous fit
PAutomaticEvent rate; Automatic is based on sample under Model 2
ShowDataQTRUEShow table of counts or not
ShowEstimateQTRUEShow details of estimation
FracRUatMaxT0.05fraction of relative utility under Model 1 for maximum value of risk threshold range
MaxBoot100Number of bootstrap iterations

dataset={matx, maty, riskscore, riskscorename, model1name, model2name, datasetname, "table"},

matxmatrix for cross-classification of risk among persons with the event,
matymatrix for cross-classification of risk among persons without the event
riskscorelist of risks corresponding to each category
riskscorenamelist of names of risk intervals corresponding to each category
model1namename of model 1
model2namename of model 2
datasetnamename of data set

dataset={y, model1risk, model2risk, model1name, model2name, datasetname, "list"},

ylist of binary outcomes (0, 1) by indvidual
model1risklist of predicted risks for model 1 by individual
model2risklist of predicted risks for model 2 by individual
model1namename of model 1
model2namename of model 2
datasetnamename of data set
datasetnamename of data set


Download All (zip 37kb)

File name / sizeDescription
rufit.m (M File, 8kb)main packages with input checks
rufitkey.m (M File, 10kb)main processing function
rufitcore.m (M File, 9kb)computes concave ROC curve and corresponding RU curve
rufitdru.m (M File, 2kb)interpolates for computing difference in RU at risk thresholds
rufitboot.m (M File, 6kb)bootstrap computation of standard errrors
rufitplot.m (M File, 12kb)plots of ROC and RU curves and calibration
rufitreport.m (M File, 7kb)calibration plot and test tradeoff
rufitdata.m (M File, 7kb)generates data examples from the literature
rufitnull.m (M File, 7kb)simulation under null hypothesis RUFITNull[]
rufitnullstat.m (M File, 7kb)compute statistics under null hypothesis
rufitnullgen.m (M File, 6kb)generate simulated data under null hypothesis


This code is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and non-infringement. In no event shall the NCI or the individual developers be liable for any claim, damages or other liability of any kind. Use of this code by recipient is at recipient's own risk. NCI makes no representations that the use of the code will not infringe any patent or proprietary rights of third parties.

Last updated: March 18, 2014

Back to top