How to increase the model accuracy of logistic regression in Scikit python?

Question

I am trying to predict the admit variable with predictors such as gre,gpa and ranks.But the prediction accuracy is very less(0.66).The dataset is given below. https://gist.github.com/abyalias/3de80ab7fb93dcecc565cee21bd9501a

Please find the codes below:

 In[73]: data.head(20)
 Out[73]: 

   admit  gre   gpa  rank_2  rank_3  rank_4
0      0  380  3.61     0.0     1.0     0.0
1      1  660  3.67     0.0     1.0     0.0
2      1  800  4.00     0.0     0.0     0.0
3      1  640  3.19     0.0     0.0     1.0
4      0  520  2.93     0.0     0.0     1.0
5      1  760  3.00     1.0     0.0     0.0
6      1  560  2.98     0.0     0.0     0.0

y = data['admit']
x = data[data.columns[1:]]

from sklearn.cross_validation import  train_test_split
xtrain,xtest,ytrain,ytest  = train_test_split(x,y,random_state=2)

ytrain=np.ravel(ytrain)

#modelling 
clf = LogisticRegression(penalty='l2')
clf.fit(xtrain,ytrain)
ypred_train = clf.predict(xtrain)
ypred_test = clf.predict(xtest)

In[38]: #checking the classification accuracy
accuracy_score(ytrain,ypred_train)
Out[38]: 0.70333333333333337
In[39]: accuracy_score(ytest,ypred_test)
Out[39]: 0.66000000000000003

In[78]: #confusion metrix...
from sklearn.metrics import confusion_matrix
confusion_matrix(ytest,ypred)

Out[78]: 
array([[62,  1],
       [33,  4]])

The ones are wrongly predicting.How to increase the model accuracy?

Solution

Since machine learning is more about experimenting with the features and the models, there is no correct answer to your question. Some of my suggestions to you would be:

1. Feature Scaling and/or Normalization - Check the scales of your gre and gpa features. They differ on 2 orders of magnitude. Therefore, your gre feature will end up dominating the others in a classifier like Logistic Regression. You can normalize all your features to the same scale before putting them in a machine learning model.This is a good guide on the various feature scaling and normalization classes available in scikit-learn.

2. Class Imbalance - Look for class imbalance in your data. Since you are working with admit/reject data, then the number of rejects would be significantly higher than the admits. Most classifiers in SkLearn including LogisticRegression have a class_weight parameter. Setting that to balanced might also work well in case of a class imbalance.

3. Optimize other scores - You can optimize on other metrics also such as Log Loss and F1-Score. The F1-Score could be useful, in case of class imbalance. This is a good guide that talks more about scoring.

4. Hyperparameter Tuning - Grid Search - You can improve your accuracy by performing a Grid Search to tune the hyperparameters of your model. For example in case of LogisticRegression, the parameter C is a hyperparameter. Also, you should avoid using the test data during grid search. Instead perform cross validation. Use your test data only to report the final numbers for your final model. Please note that GridSearch should be done for all models that you try because then only you will be able to tell what is the best you can get from each model. Scikit-Learn provides the GridSearchCV class for this. This article is also a good starting point.

5. Explore more classifiers - Logistic Regression learns a linear decision surface that separates your classes. It could be possible that your 2 classes may not be linearly separable. In such a case you might need to look at other classifiers such Support Vector Machines which are able to learn more complex decision boundaries. You can also start looking at Tree-Based classifiers such as Decision Trees which can learn rules from your data. Think of them as a series of If-Else rules which the algorithm automatically learns from the data. Often, it is difficult to get the right Bias-Variance Tradeoff with Decision Trees, so I would recommend you to look at Random Forests if you have a considerable amount of data.

6. Error Analysis - For each of your models, go back and look at the cases where they are failing. You might end up finding that some of your models work well on one part of the parameter space while others work better on other parts. If this is the case, then Ensemble Techniques such as VotingClassifier techniques often give the best results. Models that win Kaggle competitions are many times ensemble models.

7. More Features _ If all of this fails, then that means that you should start looking for more features.

Hope that helps!

Comments

[geompalik]

You could start by tuning the C parameter of logistic regression. You could also try different classification methods like SVMs and trees.

[piman314]

You should not try to be optimising the accuracy on your test set. You should optimise on the training set and use the test set as an object evaluation of the method. Can you edit your answer to show the accuracy score based on the training set?

[Aby Mathew]

Hi,accuracy based on training set is added.

[Aby Mathew]

@geompalik,I tried with putting C=0.01,100.when 100,the accuracy on training set is increased to 72.66% and accuracy on test set is 68.99%.But still no remarkable difference

[geompalik]

Two points: (i) Evaluating a model on the training set as indicated by ncfirth above, is a bad practice in general since a model fits the training data and such a score would not say anything about its generalizing ability. You should opt for cross-validation. (ii) I agree with the points of Abhinav below. I would suggest to try normalizing your gre and gpa, because their values dominate your feature vectors. Try for example: scikit-learn.org/stable/modules/generated/…

[renakre]

Nice answer. Can you please elaborate on You can optimize on other metrics also such as Log Loss and F1-Score. How do we do this? I appreciate any help!

[tagoma]

Regarding 4. Hyperparameters tuning, bayesian optimization gets people exciting these days. It shall offer the right balance between model performance versus number of hyperparameters combinations tested.