Model Evaluation Audit for Classification Problem

In this vignette we present plots for classification models evaluation.

library(auditor)
library(mlbench)

Dataset

We work on PimaIndianDiabetes dataset.

data(PimaIndiansDiabetes)
head(PimaIndiansDiabetes)

##   pregnant glucose pressure triceps insulin mass pedigree age diabetes
## 1        6     148       72      35       0 33.6    0.627  50      pos
## 2        1      85       66      29       0 26.6    0.351  31      neg
## 3        8     183       64       0       0 23.3    0.672  32      pos
## 4        1      89       66      23      94 28.1    0.167  21      neg
## 5        0     137       40      35     168 43.1    2.288  33      pos
## 6        5     116       74       0       0 25.6    0.201  30      neg

We transform dependent variable into binary vector.

pima <- PimaIndiansDiabetes
pima$diabetes <- ifelse(pima$diabetes == "pos", 1, 0)

Models

We fit 2 models: glm and svm.

model_glm <- glm(diabetes~., data = pima, family = binomial)

library(e1071)
model_svm <- svm(diabetes~., data = pima)

Model Audit

First step of auditing is creating modelAudit object. It’s an object that can be used to audit a model. It wraps up a model with meta-data.

au_glm <- audit(model_glm, data = pima, y = pima$diabetes)
au_svm <- audit(model_svm, data = pima, y = pima$diabetes, label = "svm")

Receiver Operating Characteristic (ROC)

modelAudit object can be used for plotting charts.

plotROC(au_glm, au_svm)

LIFT Chart

plotLIFT(au_glm, au_svm)

Other methods

Other methods and plots are described in vignettes:

• Model Residuals audit

• Model Fit audit

• Model Performance audit

• Observation Influence audit