Evaluating Risk Prediction Markers via Relative Utility Curves

By Stuart G. Baker, 2017

Introduction

A risk prediction marker is a baseline variable for predicting an event that is used to make treatment decisions for persons at high risk for the event. This program compares two risk prediction models: Model 1, which is the baseline model and Model 2, which includes additional markers.  The input is either a pair of cross-classified tables of risks (for event and no event) or a list of binary outcomes and predicted risks for two models. This program outputs ROC curves, relative utility curves (with summary test tradeoff), a risk mapping plot, and a decision rule plot.

References

Baker SG. Putting risk prediction in perspective: relative utility curves. Journal of the National Cancer Institute 2009 101; 1538-1542.

Baker SG, Van Calster B, Steyerberg EW. Evaluating a new marker for risk prediction using the test tradeoff: An update.  International Journal of Biostatistics 2012; 8:5.

Baker SG. The summary test tradeoff: a new measure of the value of an additional risk prediction marker. Stat Med. 2017 To appear.

Requirement

Mathematica Version 11 or later.

To run the program

copy all files into some folder called "FOLDER"
start a new Mathematica session
type SetDirectory["FOLDER"]
type << rufit.m
type RUFit[data, options]

Key Function: RUFit[dataset, options] where

Option Default Explanation
P Automatic Probability of event. Automatic computes from data; otherwise a number between 0 and 1
ShowPlot “Standard” “Talk” , “Basic”, “Standard” or  “All”

Table data for two models

dataset = {matx, maty, riskscore, riskscorename, model1name, model2name, datasetname, "table"},

matx matrix for cross-classification of risk among persons with the event,
maty matrix for cross-classification of risk among persons without the event
riskscore list of risks corresponding to each category
riskscorename list of names of risk intervals corresponding to each category
model1name name of model 1
model2name name of model 2
datasetname name of data set

List data for two models

dataset = {y, model1risk, model2risk, model1name, model2name, datasetname, "list"},

Y list of binary outcomes (0, 1) by individual
model1risk list of predicted risks for model 1 by individual
model2risk list of predicted risks for model 2 by individual
model1name name of model 1
model2name name of model 2
datasetname name of data set

File Contents

Download All (ZIP, 46 KB)

File name Description
rufit.m main packages
rufitkey.m main processing function
rufitcore.m computes concave ROC curve and corresponding RU curve
rufitplot.m plots of ROC and RU curves
rufitreport.m reports on intermediate computations (not for users)
rufitdata.m generates data examples from the literature
rufitrange.m generates range of test tradeoff
rufitapprox.m separate functions for approximations
rufittangentcondition.m separate functions for tangent condition

Disclaimer

This code is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and non-infringement. In no event shall the NCI or the individual developers be liable for any claim, damages or other liability of any kind. Use of this code by recipient is at recipient's own risk. NCI makes no representations that the use of the code will not infringe any patent or proprietary rights of third parties.

Last updated: June 15, 2017