How to plot a ROC curve for a knn model

Question

I am using ROCR package and i was wondering how can one plot a ROC curve for knn model in R? Is there any way to plot it all with this package?

I don't know how to use the prediction function of ROCR for knn. Here's my example, i am using isolet dataset from UCI repository where i renamed the class attribute as y:

cl<-factor(isolet_training$y)
knn_isolet<-knn(isolet_training, isolet_testing, cl, k=2, prob=TRUE)

Now my question is, what are the arguments to pass to the prediction function of ROC. I tried the 2 below alternatives which are not working:

library(ROCR)
pred_knn<-prediction(knn_isolet$y, cl)
pred_knn<-prediction(knn_isolet$y, isolet_testing$y)

Solution

There's several steps to solve in order to get you a ROC curve here. I am just going to make up some data since you did not provide an easy way of getting the data you are using. Note that the ROCR package wants the class labels to be positive/negative, not factors, so let's make them like that.

# Generate fake data
isolet_training <- sweep(matrix(rnorm(400), 40, 10), 1, rep(0:1, each=20))
isolet_testing <- sweep(matrix(rnorm(400), 40, 10), 1, rep(0:1, each=20))
# Generate class labels
cl <- cl_testing <- rep(c(-1, 1), each=20)

You can now train your knn and obtain its class probabilities from the "prob" attribute.

knn_isolet <- class::knn(isolet_training, isolet_testing, cl, k=2, prob=TRUE)
prob <- attr(knn_isolet, "prob")
# you can probably use just `knn` instead of `class::knn`,
# but for some reason it did not work for me.

However, they come on a form that ROCR does not accept so we need to invert them for the -1 class and rescale them.

prob <- 2*ifelse(knn_isolet == "-1", 1-prob, prob) - 1

Now you can feed the "probabilities" into the ROCR package's functions and obtain a ROC curve.

pred_knn <- prediction(prob, cl_testing)
pred_knn <- performance(pred_knn, "tpr", "fpr")
plot(pred_knn, avg= "threshold", colorize=T, lwd=3, main="Voilà, a ROC curve!")

Comments

[Backlin]

I guess it can be done since the ROCR package is all about visualizing various aspects of classifiers. It would be great if you could provide a toy example where you show how you fit your kNN classifier.

[spektra]

@Backlin I just added an example.

[ForEverNewbie]

If the vector to be provided in the predictions argument must contain the positive category probabilities of each case, why ifelse(knn_isolet == "-1", 1-prob, prob) alone is not enough to obtain such probabilities?