How to build multiclass SVM in R?

Question

I am working on the project handwritten pattern recognition(Alphabets) using Support Vector Machines. I have 26 classes in total but I am not able to classify using SVM in R. I can classify the images only if it is a binary class. How to use SVM for Multiclass SVM in R?

I am using "e1071" package.

Thanks in advance.

Solution 1

There is no direct equivalent of Multiclass SVM in e1071. Besides, all approaches to use SVM for multiclass classification use techniques like 'one vs rest' or encoding, amongst others. Here is a reference detailing most common approaches... http://arxiv.org/ftp/arxiv/papers/0802/0802.2411.pdf

If you want to use e1071 for multiclass SVM, you best can create 26 svm models, one for each class, and use the probability score to predict. This approach should be good enough for handwritten pattern recognition.

Solution 2

For a multi class classifier, you can get probabilities for each class. You can set 'probability = TRUE' while training the model & in 'predict' api. This will give you the probabilities of each class. Below is the sample code for iris data set:

data(iris)
attach(iris)
x <- subset(iris, select = -Species) 
y <- Species
model <- svm(x, y, probability = TRUE)
pred_prob <- predict(model, x, decision.values = TRUE, probability = TRUE)

With above code, 'pred_prob' will have probabilities among other data. You can access only probabilities in the object with below statement:

attr(pred_prob, "probabilities")

         setosa  versicolor   virginica
1   0.979989881 0.011347796 0.008662323
2   0.972567961 0.018145783 0.009286256
3   0.978668604 0.011973933 0.009357463
...

Hope this helps.

NOTE: I believe when you give 'probability' internally svm performs one vs rest classifier because, it takes lot more time with 'probability' param set against the model with 'probability' param not being set.

Solution 3

I guess the approved answer is outdated. The libsvm which is used in package e1071 supports also multi-class classification in a "1-vs.-1" model. I.e., it creates (L-choose-2) number of separation planes.

Here's an example code:

# Create 2d data
set.seed(1)
x1 = matrix(c(rnorm(20, 0), rnorm(20, 0)), ncol=2)
x2 = matrix(c(rnorm(20, 0), rnorm(20, 4)), ncol=2)
x3 = matrix(c(rnorm(20, 4), rnorm(20, 0)), ncol=2)
x4 = matrix(c(rnorm(20, 4), rnorm(20, 4)), ncol=2)
x = rbind(x1,x2,x3,x4)
y = factor(c(rep(1,20), rep(2,20), rep(3,20), rep(4,20)))

# Multiclass Classification (1 vs. 1)
fit = svm(y~x, kernel = "linear", cost = 10, scale=F)
plot(x, col=y, xlim=c(-3,6), ylim=c(-2.5,6.5))
table(y, predict(fit))

Comments

[Admin]

Thanks for the reply :) I have tried creating 26 SVM models using e1071 package The problem that i faced is, when i train class 1 images with class 1 image in test, i arise an error like "Model is empty". How to create models for each class and predict using probability score ?

[Gaurav]

@AbiramiM You have to input all your images in a model and mark the modelled class as 1 or 0, which will be your dependent variable. You have to do this for all classes separately and input all images each time.

[Aureliano Buendia]

Thanks, @Pavan! I had a six-classes classification problem and adapting your answer I was apt to crack it. Awesome! Something that I was not aware is that the class attribute MUST be categorical for this to work properly. Initially my dataset had the classes represented as integers. No error was raised, but the line attr(pred_prob, "probabilities") returned a NULL object. I had to change my dataset to convert the class attribute to some sort of categorical representation. Then I had the probabilities to each class returned, similar to your example. Thanks once again!

[Maverick Meerkat]

This answer is probably outdated. svm supports multiclass in a "one-vs.-one" approach.