SKLearn how to get decision probabilities for LinearSVC classifier

Question

I am using scikit-learn's linearSVC classifier for text mining. I have the y value as a label 0/1 and the X value as the TfidfVectorizer of the text document.

I use a pipeline like below

 pipeline = Pipeline([
    ('count_vectorizer',   TfidfVectorizer(ngram_range=(1, 2))),
    ('classifier',         LinearSVC())
  ])

For a prediction, I would like to get the confidence score or probability of a data point being classified as 1 in the range (0,1)

I currently use the decision function feature

pipeline.decision_function(test_X)

However it returns positive and negative values that seem to indicate confidence. I am not too sure about what they mean either.

However, is there a way to get the values in range 0-1?

For example here is the output of the decision function for some of the data points

    -0.40671879072078421, 
    -0.40671879072078421, 
    -0.64549376401063352, 
    -0.40610652684648957, 
    -0.40610652684648957, 
    -0.64549376401063352, 
    -0.64549376401063352, 
    -0.5468745098794594, 
    -0.33976011539714374, 
    0.36781572474117097, 
    -0.094943829974515004, 
    0.37728641897721765, 
    0.2856211778200019, 
    0.11775493140003235, 
    0.19387473663623439, 
    -0.062620918785563556, 
    -0.17080866610522819, 
    0.61791016307670399, 
    0.33631340372946961, 
    0.87081276844501176, 
    1.026991628346146, 
    0.092097790098391641, 
    -0.3266704728249083, 
    0.050368652422013376, 
    -0.046834129250376291, 

Solution 1

You can't. However you can use sklearn.svm.SVC with kernel='linear' and probability=True

It may run longer, but you can get probabilities from this classifier by using predict_proba method.

clf=sklearn.svm.SVC(kernel='linear',probability=True)
clf.fit(X,y)
clf.predict_proba(X_test)

Solution 2

If you insist on using the LinearSVC class, you can wrap it in a sklearn.calibration.CalibratedClassifierCV object and fit the calibrated classifier which will give you a probabilistic classifier.

from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn import datasets

#Load iris dataset
iris = datasets.load_iris()
X = iris.data[:, :2] # Using only two features
y = iris.target      #3 classes: 0, 1, 2

linear_svc = LinearSVC()     #The base estimator

# This is the calibrated classifier which can give probabilistic classifier
calibrated_svc = CalibratedClassifierCV(linear_svc,
                                        method='sigmoid',  #sigmoid will use Platt's scaling. Refer to documentation for other methods.
                                        cv=3) 
calibrated_svc.fit(X, y)


# predict
prediction_data = [[2.3, 5],
                   [4, 7]]
predicted_probs = calibrated_svc.predict_proba(prediction_data)  #important to use predict_proba
print predicted_probs

Here is the output:

[[  9.98626760e-01   1.27594869e-03   9.72912751e-05]
 [  9.99578199e-01   1.79053170e-05   4.03895759e-04]]

which shows probabilities for each class for each data point.

Comments

[Sakib]

Is SVC different from LinearSVC? Will it lead to different results? and also is the probability between 0 and 1 in SVCs predict probabilties?

[Farseer]

Yes. predict_proba returns probabilities. SVC using representation theorem for training(krenel trick), so it may run longer, but result should be very similar(it may be a little different, depends on implementation). @Sakib

[Sakib]

Great! I will try this out but it looks like this is what I needed.

[Sakib]

I have one more question though. What is the difference between a decision function and prediction probability?

[Farseer]

decision function is your hypothesis(model). prediction probability is just a probability that your hypothesis returns. But is very broad definitions. @Sakib

[Antoine]

unfortunately, results are very different between SVC and linearSVC because there are some fundamental differences: stackoverflow.com/questions/33843981/… stackoverflow.com/questions/35076586/…