Compute correlation between features and target variable

24,117

Solution 1

You could use pandas corr on each column:

df.drop("Target", axis=1).apply(lambda x: x.corr(df.Target))

Solution 2

Since Pandas 0.24 released in January 2019, you can simply use DataFrame.corrwith():

df.corrwith(df["Target"])

Solution 3

You can use scipy.stats.pearsonr on each of the feature columns like so:

import pandas as pd
import numpy as np
from scipy.stats import pearsonr

# example data
df = pd.DataFrame([[1, 2, 4 ,6], [1, 3, 4, 7], [4, 6, 8, 12], [5, 3, 2 ,10]],
                  columns=['Feature1', 'Feature2','Feature3','Target'])

# Only compute pearson prod-moment correlations between feature
# columns and target column
target_col_name = 'Target'
feature_target_corr = {}
for col in df:
    if target_col_name != col:
        feature_target_corr[col + '_' + target_col_name] = \
            pearsonr(df[col], df[target_col_name])[0]
print("Feature-Target Correlations")
print(feature_target_corr)
Share:
24,117
Cox Tox
Author by

Cox Tox

Updated on July 18, 2022

Comments

  • Cox Tox
    Cox Tox over 1 year

    What is the best solution to compute correlation between my features and target variable ?? My dataframe have 1000 rows and 40 000 columns...

    Exemple :

    df = pd.DataFrame([[1, 2, 4 ,6], [1, 3, 4, 7], [4, 6, 8, 12], [5, 3, 2 ,10]], columns=['Feature1', 'Feature2','Feature3','Target'])
    

    This code works fine but this is too long on my dataframe ... I need only the last column of correlation matrix : correlation with target (not pairwise feature corelation).

    corr_matrix=df.corr()
    corr_matrix["Target"].sort_values(ascending=False)
    

    The np.corcoeff() function works with array but can we exclude the pairwise feature correlation ?

  • Cox Tox
    Cox Tox about 5 years
    Exactly what I need !
  • Juan-Kabbali
    Juan-Kabbali almost 3 years
    This will compute unnecessarily the correlation between all columns. Instead you should use df.corrwith(df['target'])
  • Mujeebur Rahman
    Mujeebur Rahman over 2 years
    How can we plot this correlation array as a heatmap?